Introduction to Programming, Aug-Dec 2008 Lecture 8, Monday 08 Sep 2008 Measuring efficiency -------------------- Quicksort --------- In merge sort, we split the list into two parts directly. Since the second part could have had elements smaller than those in the first part and vice versa, we have to spend some time merging the two sorted lists. What if we could locate the median (middle) value in the list? We could then collect all the elements less than the median in one half and those bigger than the median in the other half. If we sort these two halves inductively, we can directly combine them using ++ rather than a merge. Unfortunately, finding the median value is not easier than sorting the list. However, we can use a variation of this idea by picking up an arbitrary element of the list and using it to "split" the list into a lower half and upper half. This algorithm is called quicksort. quicksort [] = [] quicksort (x:xs) = (quicksort lower) ++ [splitter] ++ (quicksort upper) where splitter = x lower = [ y | y <- xs, y <= x ] upper = [ y | y <- xs, y > x ] Notice that we use the first element of the list as the splitter, for simplicity. When computing lower, we include other elements that have the same value as x, but we make sure that x itself is left out by writing "y <- xs". We could, symmetrically, have added the equal values in upper instead of lower. The problem with the worst case behaviour of quicksort is that we cannot guarantee that lower and upper are half the size of the original list. In the worst case, the first element is either the largest or the smallest element and we have to inductively sort a list of length n-1. The worst case recurrence becomes T(n) = T(n-1) + O(n) which we know yields T(n) = O(n^2). The O(n) factor captures the cost of computing lower and upper and combining the answer using ++. In practice, however, quicksort works very fast and is often the algorithm used for implementing built-in sort functions in many applications. In fact, one can formallyprove that in the "average case", quicksort is an O(n log n) algorithm, like mergesort. The details of this analysis are beyond the scope of this course. Defining complexity for arithmetic functions -------------------------------------------- Consider the problem of deciding whether a number n is a prime. A naive algorithm is to check whether any number between 2 and n-1 divides n. This requires about O(n) divisions. One can be slightly more sophisticated and observe that it is sufficient to check for divisors from 2 to square root of n. This would suggest that checking primality can be done in time polynomial in n. Why, then, was it considered a sensational result when Agrawal, Kayal and Saxena announced in 2002 that testing primality can be done in polynomial time? The answer is that the input size for an arithmetic function is measured in terms of the number of digits of the input, not the number itself. Consider any basic arithmetic operation we learn in school --- for instance multiplication. When we multiply two numbers, the time it takes depends on the length of the two numbers. It does not take 100 times as much effort to multiply a 5 digit number by a 4 digit number as compared to the time it takes to multiply a 5 digit number by a 2 digit number. (Multiplying two numbers in time proportional to the value of the numbers amounts to doing repeated addition. For instance, 345*12 is 345+345+...+345 12 times while 345*1234 is 345+345+...+345 1234 times and hence takes about 100 times the effort! But, clearly this is not how we normally multiply multidigit numbers.) In base 10, the number of digits to write down n is log_10 n. Since computers typically represent values in base 2, the input size for an arithmetic function is usually assumed to be log_2 n, which, as we have agreed, we always write as just log n. Notice that our earlier naive procedure for primality now takes time O(2^k), where k = log n is the size of the input. Even the slightly cleverer trick of going only upto square root of n takes time O(2^{k/2}), which is still exponential in k. The complexity of "Divide and conquer" -------------------------------------- Recall our divide and conquer approach to multiplication. We divide each number into two k/2 digit parts, by regarding the leftmost k/2 digits and rightmost k/2 digits as separate blocks. Thus, we have a_k a_{k-1} ... a_{k/2+1} a_{k/2} ... a_2 a_1 <---------A_L-----------> <------A_R--------> b_k b_{k-1} ... b_{k/2+1} b_{k/2} ... b_2 b_1 <---------B_L-----------> <------B_R--------> The products A_L*B_R, A_R*B_L etc are k/2 bit multiplications. It is easy to verify that A*B can be expressed as 10^k(A_L*B_L) + 10^{k/2}(A_L*B_R + A_R*B_L) + A_R*B_R Multiplying a number by 10^j just requires adding j 0's to the right, so the original problem reduces to performing 4 multiplications of size k/2. We can thus write T(k) = 4 T(k/2) + O(k) T(1) = 1 If we expand this out, we get T(k) = 4 T(k/2) + O(k) = 4 (4 T(k/4) + O(k/2)) + O(k) = 4^2 T(k/2^2) + 2O(k) + O(k) = 4^2 (4 T(k/2^3) + O(k/4)) + 2O(k) + O(k) = 4^3 T(k/2^3) + 4O(k) + 2O(k) + O(k) = ... = 4^m T(1) + sum_{i = 1 to m} 2^i O(k), where m = log k = k^2 + .... Thus, our divide and conquer solution has the same complexity, O(k^2), as the direct algorithm we learn in school! Notice that the only difference between the recurrence for this multiplication procedure and the one for mergesort is in the coefficient of the T(n/2) term: Mergesort : T(n) = 2T(n/2) + O(n) Multiplication : T(n) = 4T(n/2) + O(n) Clearly, the problem arises from the fact that we need to consider 4 multiplications of size T(n/2) and not just 2. In fact, the following can be shown: Let T(k) = a T(k/b) + O(k). Then, T(k) = O(k log n), if (a = b) T(k) = O(k^(log_b a)), if (a > b) T(k) = O(k), if (a < b) We will not prove this here, but notice that this tells us that we can modify mergesort to merge 3 sorted lists of size n/3 or 4 sorted lists of size n/4 etc and the complexity will remain the same. minout ------ To see an example of the last case described above T(k) = O(k), if (a < b) we look at the function "minout. Let l be a list of distinct natural numbers (i.e. integers from the set {0,1,2,...}). The function "minout l" returns the smallest natural number not present in l (i.e., if 0 is not in l, then "minout l" returns 0, else if 0 is in l but 1 is not in l, them "minout l" returns 1, etc). Note that l is NOT assumed to be sorted. The function should take time O(n). Observe that if we sort l in ascending order, we can scan the sorted list from left to right and look for the first gap. However, sorting takes time O(n log n). Observe that the output of minout l ranges from 0 to (length l). Either l contains all numbers [0..(length l)-1], in which case minout l is length l, or there is a gap in this sequence, in which case minout l is in the range 0..(length l) - 1. Our strategy will be to narrow down the range for the output of minout to half the original in each step. The only values in l that contribute to minout are those between 0 and (length l) - 1. Values that are less than 0 or greater than or equal to (length l) are irrelevant. Suppose, in one pass, we calculate the following sublists of l: firsthalf = [ x | x <- l, x >= 0, x < (length l) `div` 2 ] secondhalf = [ x | x <- l, x >= (length l) `div` 2, x < (length l)] We have two cases to consider: a. Not all values in the range 0 to ((length l) `div` 2) - 1 appear in firsthalf. To check this, we just have to verify that (length firsthalf) == (length l) `div` 2. Then, we know that the smallest missing number lies in this range. b. Otherwise, the missing number lies in the range [(length l) `div` 2 .. (length l) - 1]. In this way, we get the following recurrence for T(n): T(n) = T(n/2) + O(n) We need O(n) time to construct firsthalf and secondhalf. Having done this, we can focus on one of the two sublists and ignore the other. From the result quoted above, this yields T(n) = O(n), as required. (Note: When writing the definition of minout, the case when we focus on the second half requires us to shift the values. For instance, if l = [0,1,2,3,5,6], then firsthalf = [0,1,2] and secondhalf is [3,5,6]. Since there is no gap in firsthalf, we inductively check "minout [3,5,6]". From the definition of minout, the answer to this is 0, which is clearly incorrect for the original list! In fact, we need to reduce all the values in secondhalf by 3 and check "minout [0,2,3]". Alternatively, we can write an auxiliary function that computes minout with respect to a lower bound. In this case, the original function would be "auxminout 0 [0,1,2,3,5,6]" while the inductive call would be "auxminout 3 [3,5,6]".) ======================================================================