Introduction to Programming, Aug-Dec 2008
Lecture 7, Mon 01 Sep 2008

Measuring efficiency
--------------------

Computation in Haskell is reduction --- that is, one sided
  application of rewriting rules (function definitions)

  Time taken: count the number of reduction steps

Notion of cost is model dependent.

Want the cost of a function  in terms of its input size

  What is the input size?

  For list functions, e.g., number of elements in list

Typically, we denote by T(n) the time taken by an algorithm for
an input of size n

Notation:

  f(n) = O(g(n)) if there is a constant k such that 
    f(n) <= k g(n) for all n > 0

  e.g.  an^2 + bn + c = O(n^2) for all choices of a,b,c
           (take k = a+b+c)

We will be interested in

  -- "asymptotic" complexity : hence express T(n) in terms of
      O(f(n)). 

  -- worst case complexity : what is the worst possible running
       time on an input of size n?  Note that the worst case
       could be very, very rare!

e.g.

Cost of ++:

  (++) :: [a] -> [a] -> [a]

  [] ++ y = y
  (x:xs) ++ y = x:(xs++y)


  Observe that
    [1,2,3] ++ [4,5,6] ->
    1:([2,3] ++ [4,5,6]) ->
    1:(2:([3] ++ [4,5,6])  ->
    1:(2:(3:([] ++ [4,5,6])  ->
    1:(2:(3:([4,5,6])

  We assume that all representations of a list are equivalent, so
  the last line is the same a [1,2,3,4,5,6] or 1:[2,3,4,5,6] or
  ... or 1:2:3:4:5:6:[]

  We note that if we merge lists l1 of length n1 and l2 of length
  n2, then we reduce using the second definition of ++ n1 times
  and reduce using the first definition 1 time.  There is no
  dependance on n2.  In other words, the relevant input size for
  ++ is n1, which we call n.

  Thus, for ++, T(n) = n + 1 = O(n)

Now, let us compute the cost of reverse:

  reverse :: [a] -> [a]

  reverse [] = []
  reverse (x:xs) = (reverse xs) ++ [x]

  We can analyze this directly, like ++, or we can write down the
  following "recurrence":

  T(0) = 1
  T(n) = T(n-1) + n

  This says that to reverse a list of length n, we have to first
  solve the same problem for its tail, a list of length n-1 (this
  yields T(n-1)) and then do O(n) work to retrieve the result for
  the original list (this is because we use ++ with input of size
  n-1, which we know is of cost n).

  We can solve this recurrence by unfolding it:

  T(n) = T(n-1) + n 
       = (T(n-2) + n-1) + n
       = (T(n-3) + n-2) + n-1 + n
         ...
       = T(0) + 1 + 2 + ... + n
       = 1 + 1 + 2 + ... + n = O(n^2)



Can we do better for reverse?  We can use a function transfer
that reads two lists and moves elements from the first list to
the front of the second list, one by one.

  transfer :: [a] -> [a] -> [a]
  transfer [] l = l
  transfer (x:xs) l = transfer xs (x:l)

Clearly, the relevant input size for transfer is the length of
l1.   In this case, 

  T(0 = 1
  T(n) = T(n-1) + 1 (it requires one reduction
                     to get to the problem of size n-1).  

Expanding this, we get T(n) = 1+1+...+1 = O(n).

It should be clear that transfer l1 l2 = (reverse l1)++l2.
Thus, transfer l [] = (reverse l) ++ [] = reverse l, so we can
write: 

  reverse :: [a] -> [a]
  reverse l = transfer l []

  For this version of reverse, T(n) = O(n).

Sorting lists
-------------

Consider the problem of sorting a list in ascending order.  A
natural inductive definition is

  isort [] = []
  isort (x:xs) = insert x (isort xs)

    where

    insert x [] = [x]
    insert x (y:ys) 
       | (x <= y) = x:y:ys 
       | otherwise y:(insert x ys)

This sorting algorithm is called insertion sort, which is why we
have used the name isort for the function.

Clearly, for insert, T(n) = O(n).

Then, for isort

  T(0) = 1
  T(n) = T(n-1) + O(n), 

which we know (see reverse, version 1) yields O(n^2)

(Aside:  Observe that we can also write isort as foldr insert []).


Can we do better?

Merge sort
----------

Suppose we divide the list into two halves and sort them
separately.  Can we combine two sorted lists efficiently into a
single sorted list?  Examine the first element of each list and
pick up the smaller one, and continue to combine what remains
inductively.

  merge [] ys = ys
  merge xs [] = xs

  merge (x:xs) (y:ys) 
    |x <= y    = x:(merge xs (y:ys))
    |otherwise = y:(merge (x:xs) ys)

The running time depends on the length of  both  input lists.  If
l1 is of length n1 and l2 is of length n2, in each step we put
out exactly one element into the final list.  Thus, in n1+n2
steps we achieve the merge.

Now, we split a list of size n into two lists of n/2, recursively
sort them and merge the sorted sublists as follows:

  mergesort [] = []
  mergesort [x] = [x]
  mergesort l = merge (mergesort (front l)) (mergesort (back l))
    where
     front l = take ((length l) `div` 2) l
     back l = drop ((length l) `div` 2) l
     merge [] ys = ys
     merge xs [] = xs
     merge (x:xs) (y:ys) 
       | x < y     = x:(merge xs (y:ys)) 
       | otherwise = y:(merge (x:xs) ys)

Note that we need explicit base cases for both the empty list and
the singleton list.  If we omit the case mergesort [x], we would
end up with the unending sequence of simplifications below

  mergesort [x] = merge (mergesort []) (mergesort [x])
                = merge [] (mergesort [x]))
                = mergesort [x] 
                = ...

Clearly 

   T(0) = 1
   T(n) = 2 T(n/2) +     O(n)
          --------   -------------
          recursive  initial split 
            sort     and  
                     final merge


Solving this, we get T(n) = O(n log n).  It is important to
recognize that the function n log n is much closer to n than to
n^2.


======================================================================