Introduction to Programming, Aug-Dec 2008 Lecture 19, Monday 03 Nov 2008 Listing out a heap tree in sorted order --------------------------------------- For balanced search trees, inorder traversal produces a sorted list of values (in linear time). We can also list out the values of a heap tree in sorted order. We know that the largest value is at the root, so we put out the root value first. Now, both the left and right subtrees are heap trees. If we list them out using the same process, they will independently yield sorted lists. We can then merge these lists to get a single sorted list. horder :: (HTree a) -> [a] horder HNil = [] horder (HTree x h1 h2) = x:(merge (horder h1) (horder h2)) where merge :: (Ord a) => [a] -> [a] -> [a] merge l1 [] = l1 merge [] l2 = l2 merge (x:xs) (y:ys) | x <= y = x:(merge xs (y:ys)) | otherwise = y:(merge (x:xs) ys) Note that the output of horder is in descending order. Applying reverse to the output will produce a list in ascending order in linear time. What is the complexity of horder? Assuming the heap tree is size balanced, we have to inductively horder both subtrees of size N/2 and merge them, so we have: T(N) = 2T(N/2) + O(N) or T(N) = O(N log N) Can we do better? Observe that we can now sort an arbitrary list by constructing a heap tree and listing it out using horder. Sorting takes at least O(N log N) time. We can construct a heap in time O(N). Thus, if we could improve the complexity of horder below O(N log N), we would have a sorting algorithm that is below O(N log N)! To complete this discussion, we formally define heapsort: heapsort l = horder (heapify (mkbtree l)) or, using the builtin operator "." for function composition: heapsort = horder . heapify . mkbtree Leftist heaps ------------- We have seen how to construct a size balanced heap (we shall henceforth write just "heap" for "heap tree") from a list in linear time. Recall that to implement a priority queue using heap, we need to implement the following operations: insert :: (PriorityQueue a) -> a -> (PriorityQueue a) delmax :: (PriorityQueue a) -> (a,(PriorityQueue a)) Thus, the one-shot heap construction procedure we have described is not enough. We need an efficient way to update heaps incrementally. We shall describe a technique to combine two heaps of size M and N in time O(log(M + N)). This will solve both the problems above. 1. To insert a value x in heap h, we construct a trivial heap of one element containing x and use the union algorithm to combine this with h. 2. To delete the maximum value in a heap, we remove the root and then combine the left and right subtrees using the union algorithm. Since union takes time O(log(M+N)), we can implement both the required operations in logarithmic time. To define our union operation, we need "leftist heaps". A leftist heap is one in which, at every node, the left subtree is has at least as many nodes as the right subtree. Recall that the definition of a heap does not require any specific order between the subtrees. Thus, if we exchange the left and right subtrees of a heap, we still have a heap. We can use this fact to write a procedure to convert an arbitrary heap into a leftist one, bottom up. We first write a function that realigns the left and right subtrees, according to size: realign :: (HTree a) -> (HTree a) realign HNil = HNil realign (HNode x h1 h2) | (size h1) < (size h2) = HNode x h2 h1 | otherwise = HNode x h1 h2 where size :: (HTree a) -> Int size HNil = 0 size (HTree x h1 h2) = 1 + (size h1) + (size h2) Thus, realign just reorders the left and right subtrees if the leftist property is violated at a node. As usual, we can convert size into a constant time function by storing the size of a heap as one of the values under HNode. In other words, we redefine HTree to be (Eq a) => data HTree a = HNil | HNode Int a (HTree a) (HTree a) However, for simplicity, we shall stick to the original definition in the rest of this exposition. We can now make an entire heap leftist using realign. mkleftist :: (HTree a) -> (HTree a) mkleftist HNil = HNil mkleftist (HNode x h1 h2) = realign (HNode x lh1 lh2) where lh1 = mkleftist h1 lh2 = mkleftist h2 Let us call the rightmost path in a heap the "right spine". The main property of a leftist heap of size n is that the length of the right spine is less than log n. This can be proved easily, by induction on the size of the heap. Let lrs(h) denote the length of the right spine of heap h. n = 0 : The heap is empty and the result is trivial n > 0 : Consider a heap h with root x and left and right subtrees h1 and h2 of size p and q, respectively. Then lrs(h) = 1 + lrs(h2) -- By definition of right spine < 1 + (log q) -- By induction hypothesis on h2 = log 2 + log q -- Arithmetic ... = log 2q -- Arithmetic ... <= log (p + q) -- h is leftist, so p >= q < log (1 + p + q) -- Arithmetic ... = log (size h) Union of leftist heaps ---------------------- Let us look at the problem of combining two leftist heaps: Suppose that h = x and h' = y / \ / \ h1 h2 h3 h4 Clearly, the bigger of x and y should become the root of the combined heap. Suppose x is bigger. We then merge h' with the right subheap h2, using mkleftist to preserve the leftist nature of the heap. Symmetrically, if y is bigger, we inductively merge h with h4. Here is the definition of union. union :: (HTree a) -> (HTree a) -> (HTree a) union h HNil = h union HNil h = h union (HTree x h1 h2) (HTree y h3 h4) | x < y = realign (HTree y h3 (union (HTree x h1 h2) h4)) | otherwise = realign (HTree x h1 (union h2 (HTree y h3 h4))) Each step of union makes one move down the right spine of either h or h'. Since we have already seen that lrs(h) < log (size h) for leftist heaps, it follows that each evaluation of union takes at most log(size h) + log(size h') steps. For any integers m and k, log m^k = k log m, so log m^k = O(log m). From this, it follows that O(log m + log n) = O(log mn) = O(log max(m,n)) = O(log (m+n)) Hence, O(log(size h)) + O(log(size h')) = O(log(size h + size h')), which is the bound we want for union. ====================================================================== Memoization ----------- Most of the functions we have written have natural inductive definitions. However, naive evaluating an inductive definition is often computationally wasteful because we repeatedly compute the same value. A typical example of this is the function to compute the nth Fibonacci number. fib :: Int -> Int fib 0 = 1 fib 1 = 1 fib n = (fib (n-1)) + (fib (n-2)) Let us look at the computations involved in, say, evaluating fib 4. fib 4 / \ fib 3 fib 2 / \ / \ fib 2 fib 1 fib 1 fib 0 / \ | | | fib 1 fib 0 1 1 1 | | 1 1 Observe that fib 2 is evaluated twice, independently. Now, suppose we compute fib 6: fib 6 / \ fib 5 fib 4 / \ / \ fib 4 ... ... .... Here, fib 6 produces two independent computations of the entire tree we saw above for fib 4, each of which duplicates the computation for fib 2. In this way, as we compute fib n for larger and larger, more and more evaluations are duplicated. The result is that computing fib n becomes exponential in n. However, we know that we can naively enumerate the nth Fibonacci number in n steps by counting upwards from the first two as follows : 1,1,2,3,5,8,13,21,.... One way to get around the wasteful recomputation we saw above is to remember the values we have computed and not recompute any value that is already known. We maintain a table, or a "memo", where we note down all known values. Before computing a fresh value, we look up this memo. If the value already exists in the memo, we use it. Otherwise, we use the inductive definition to calculate the value and write it into the memo for future reference. This process is called "memoization". How do we maintain the memo? One simple way is to just maintain an association list of (value,function) pairs. Here is a how a simple memoized version of fib would look: fib :: Int -> Int fib n = memofib n [] The function memofib takes an argument for which the function is to be computed, together with the current memo, and returns the function value with an updated memo. memofib :: Int -> [(Int,Int)] -> (Int,[(Int,Int)]) memofib n l | elem n [x | (x,y) <- l] -- value found in memo, return it with the memo unchanged = (head [y | (x,y) <- l, x == n],l) | n = 0 -- base case, add it to memo = (1,(0,1):l) | n = 1 -- base case, add it to memo = (1,(1,1):l) | otherwise -- compute the new value and update the memo = (valc,memoc) where -- need to sequentially compute the inductive values so -- updates to the memo are preserved (vala,memoa) = memofib (n-1) l (valb,memob) = memofib (n-2) memoa valc = vala + valb memoc = (n,valc):memob Note that each value fib n is computed exactly once. Computing a value involves looking up two values in the memo and updating one value into the memo. To count the total number of memo lookups, we note that each value fib n is looked up twice in the memo after it is computed, when computing fib (n+1) and fib (n+2). Thus, the overall complexity is 2n*(memo lookup cost) + n*(memo update cost. In the naive list based memo, the look up cost for the memo is O(n) while the update cost is O(1). Thus, the complexity of this implementation is O(n^2). We can improve the complexity by making the memo more efficient. It is clear that to compute fib n, we only need to look at values between fib 0 and fib n. We can thus store the memo in an array indexed by (0,n). This would reduce the lookup cost to log n and bring down the overall complexity to O(n log n). Dynamic programming ------------------- Recall that we said we could compute the nth fibonacci number in linear time by enumerating the list of values from fib 0 as 1,1,2,3,5,... This amounts to filling out the memo table systematically from the bottom up, using the inductive definition of the function to update the memo table. This bottom up approach to computing a memoized function is called "dynamic programming". In other words, if we have understood the structure of the memo sufficiently, we can "directly" fill it up, without making inductive calls to make each update. In an ideal world, both the top down memoized approach, where updates to the memo table happen whenever the inductive call hits an uncomputed value, and the bottom up dynamic programming approach, which updates the memo directly using the inductive definition, should be equally efficient. However, in standard implementations, making recursive function calls incurs some overheads where we have to remember the state of values in the calling function etc, so the top down approach is NOT as efficient as the bottom up approach. However, it is to be emphasized that almost *any* function can be memoized in reasonably uniform way (the list based memo given above should work for any function whose argument satisfies Eq a), but implementing a memoized function using dynamic programming requires understanding the dependency of the memo values on each other and filling them up appropriately. Modularizing the memo --------------------- Ideally, we should make our memoized function independent of the choice of memo. For this, we first define an abstract datatype Table that stores values of functions from a to b and supports the following operations :: emptytable :: returns an empty memo table memofind :: tells whether a value exists in the memo table memolookup :: looks up a value in the memo table memoupdate :: adds a new entry to the memo table Here is an implementation of this datatype using lists, as we have done above. Note the requirement (Eq a) for the domain type of the function. module Memo(Table,emptytable,memofind,memolookup,memoupdate) where data (Eq a) => Table a b = T [(a,b)] deriving (Eq,Show) -- emptytable n v creates a table to hold n values with initial -- "undefined" value v. This is required, for instance, if we -- use an array to represent the table. emptytable :: (Eq a) => Int -> b -> (Table a b) emptytable n v = T [] memofind :: (Eq a) => (Table a b) -> a-> Bool memofind (T []) _ = False memofind (T ((y,m):l)) x | x == y = True | otherwise = memofind (T l) x memolookup :: (Eq a) => (Table a b) -> a -> b memolookup (T ((y,m):l)) x | x == y = m | otherwise = memolookup (T l) x memoupdate :: (Eq a) => (Table a b) -> (a,b) -> (Table a b) memoupdate (T l) (x,n) = T ((x,n):l) Now, we can write our memoized version of the fib function a little more "abstractly", using the Memo module. import Memo fib :: Int -> Int fib n = memofib n emptytable memofib :: Int -> [Table Int Int] -> (Int,Table Int Int) memofib n t | memofind t n = (memolookup t n, t) | n = 0 = (1, memoupdate t (0,1)) | n = 1 = (1, memoupdate t (1,1)) | otherwise = (valc,memoc) where (vala,memoa) = memofib (n-1) l (valb,memob) = memofib (n-2) memoa valc = vala + valb memoc = memoupdate memob (n,valc) The advantage is that if we decide to modify the memo table implementation to be more efficient, we need not change anything in the definition of fib/memofib. ======================================================================