Algorithms: COMP3121/3821/9101/9801

THE UNIVERSITY OFNEW SOUTH WALES

Algorithms:COMP3121/3821/9101/9801

School of Computer Science and EngineeringUniversity of New South Wales Sydney

2. DIVIDE-AND-CONQUER

COMP3121/3821/9101/9801 1 / 1

A Puzzle

Problem : We are given 27 coins of the same denomination; weknow that one of them is counterfeit and that it is lighter than theothers. Find the counterfeit coin by weighing coins on a panbalance only three times.

This method is called “divide-and-conquer”.

We have already seen a prototypical “serious” algorithm designedusing such a method: the Merge-Sort.

We split the array into two, sort the two parts recursively andthen merge the two sorted arrays.

We now look at a closely related but more interesting problem ofcounting inversions in an array.

COMP3121/3821/9101/9801 2 / 1

Counting the number of inversions

Assume that you have m users ranking the same set of n movies.You want to determine for any two users A and B how similartheir tastes are (for example, in order to make a recommendersystem).

How should we measure the degree of similarity of two users A andB?

Lets enumerate the movies on the ranking list of user B byassigning to the top choice of user B index 1, assign to his secondchoice index 2 and so on.

For the ith movie on B′s list we can now look at the position (i.e.,index) of that movie on A′s list, denoted by a(i).

COMP3121/3821/9101/9801 3 / 1


Thus, if the first listed movie on B′s list is the second movie onA′s list, then a(1) = 2; if the second choice of user B is the fifthchoice of user A, then a(2) = 5, and so on.

Note that in this way for all i, 1 ≤ i ≤ n, A[a(i)] = i.

9a(9)=3

7 a(7)=5

4a(4)=8

2a(2)=11

1 a(1)=1

123456789101112 B

A3 568123456789101112

COMP3121/3821/9101/9801 4 / 1


A good measure of how different these two users are, is the total number ofinversions, i.e., total number of pairs of movies i, j such that movie i precedesmovie j on B′s list but movie j is higher up on A′s list than the movie i.

In other words, we count the number of pairs of movies i, j such that i < j(movie i precedes movie j on B′s list) but a(i) > a(j) (movie i is in theposition a(i) on A′s list which is after the position a(j) of movie j on A′s list.

9a(9)=3

7 a(7)=5

4a(4)=8

2a(2)=11

1 a(1)=1

123456789101112 B

A3 568123456789101112

For example 1 and 2 do not form an inversion because a(1) < a(2) (a(1) = 1and a(2) = 11 because a(1) is on the first and a(2) is on the eleventh place inA);

However, for example 4 and 7 do form an inversion because a(7) < a(4)(a(7) = 5 because a(7) is on the fifth place in A and a(4) = 8)

COMP3121/3821/9101/9801 5 / 1


An easy way to count the total number of inversions between twolists is by looking at all pairs i < j of movies on one list anddetermining if they are inverted in the second list, but this wouldproduce a quadratic time algorithm, T (n) = Θ(n2).

We now show that this can be done in a much more efficient way,in time O(n log n), by applying a Divide-And-Conquer strategy.

Clearly, since the total number of pairs is quadratic in n, wecannot afford to inspect all possible pairs.

The main idea is to tweak the Merge-Sort algorithm, byextending it to recursively both sort an array A and determinethe number of inversions in A.

COMP3121/3821/9101/9801 6 / 1


We split the array A into two (approximately) equal partsAtop = A[1 . . . bn/2c] and Abottom = A[bn/2c+ 1 . . . n].

Note that the total number of inversions in array A is equal to thesum of the number of inversions I(Atop) in Atop (such as 9 and 7)plus the number of inversions I(Abottom) in Abottom (such as 8 and2) plus the number of inversions I(Atop, Abottom) across the twohalves (such as 7 and 4).

9

7 4

2

1

A

Atop

Abottom

21

7 9101112

3 568

1

Atop AbottomCOMP3121/3821/9101/9801 7 / 1


We now recursively sort arrays Atop and Abottom and obtain the number ofinversions I(Atop) in the sub-array Atop and the number of inversionsI(Abottom) in the sub-array Abottom.

We now merge the two sorted arrays Atop and Abottom while counting thenumber of inversions I(Atop, Abottom) which are across the two sub-arrays.

When the next smallest element among all elements in both arrays is anelement in Abottom, such an element clearly is in an inversion with all theremaining elements in Atop and we add the total number of elements remainingin Atop to the current value of the number of inversions across Atop andAbottom.

9a9=3

7 a7=5

4a4=8

2a2=11

1 a1=1

A

Atop

Abottom

21

7 9101112

3 568

1

Atop Abottom

COMP3121/3821/9101/9801 8 / 1


Whenever the next smallest element among all elements in botharrays is an element in Atop, such an element clearly is not involvedin any inversions across the two arrays (such as 1, for example).

After the merging operation is completed, we obtain the totalnumber of inversions I(Atop, Abottom) across Atop and Abottom.

The total number of inversions I(A) in array A is finally obtainedas:

I(A) = I(Atop) + I(Abottom) + I(Atop, Abottom)

Next: we study applications of divide and conquer to arithmeticof very large integers.

COMP3121/3821/9101/9801 9 / 1

Basics revisited: how do we add two numbers?

C C C C C carry

X X X X X first integer

+ X X X X X second integer

-----------

X X X X X X result

adding 3 bits can be done in constant time;

the whole algorithm runs in linear time i.e., O(n) many steps.

can we do it faster than in linear time?

no, because we have to read every bit of the input

no asymptotically faster algorithm

COMP3121/3821/9101/9801 10 / 1

Basics revisited: how do we multiply two numbers?

X X X X <- first input integer

* X X X X <- second input integer

-------

X X X X \

X X X X \ O(n^2) intermediate operations:

X X X X / O(n^2) elementary multiplications

X X X X / + O(n^2) elementary additions

---------------

X X X X X X X X <- result of length 2n

We assume that two X’s can be multiplied in O(1). time (each X

could be a bit or a digit in some other base).Thus the above procedure runs in time O(n2).Can we do it in LINEAR time, like addition?No one knows!“Simple” problems can actually turn out to be difficult!

COMP3121/3821/9101/9801 11 / 1

Can we do multiplication faster than O(n2)?

Let us try a divide-and-conquer algorithm:take our two input numbers A and B, and split them into two halves:

A = A12n2 + A0 XX . . .X︸︷︷︸XX . . .X︸︷︷︸

B = B12n2 + B0

n

2

n

2

• A0, B0 - the least significant bits; A1, B1 the most significant bits.• AB can now be calculated as follows:

AB = A1B12n + (A1B0 + B1A0)2

n2 + A0B0 (1)

What we mean is that the product AB can be calculated recursively bythe following program:

COMP3121/3821/9101/9801 12 / 1

1: function Mult(A,B)2: if |A| = |B| = 1 then return AB3: else4: A1 ← MoreSignificantPart(A);5: A0 ← LessSignificantPart(A);6: B1 ← MoreSignificantPart(B);7: B0 ← LessSignificantPart(B);8: X ←Mult(A0,B0);9: Y ←Mult(A0,B1);

10: Z ←Mult(A1,B0);11: W ←Mult(A1,B1);12: return W 2n + (Y + Z) 2n/2 + X13: end if14: end function

COMP3121/3821/9101/9801 13 / 1

How many steps does this algorithm take?

Each multiplication of two n digit numbers is replaced by fourmultiplications of n/2 digit numbers: A1B1, A1B0, B1A0, A0B0,plus we have a linear overhead to shift and add:

T (n) = 4T(n

2

)+ c n (2)

COMP3121/3821/9101/9801 14 / 1


Claim: if T (n) satisfies

T (n) = 4T(n

2

)+ c n (3)

then

T (n) = n2(c + 1)− c n

Proof: By “fast” induction. We assume it is true for bn/2c:

T(n

2

)=(n

2

)2(c + 1)− c

n

2

and prove that it is also true for n:

T (n) = 4T(n2

)+ c n = 4

((n2

)2(c + 1)− n

2 c)

+ c n

= n2(c + 1)− 2c n + c n = n2(c + 1)− c n

COMP3121/3821/9101/9801 15 / 1


Thus, if T (n) satisfies

T (n) = 4T(n

2

)+ c n

then

T (n) = n2(c + 1)− c n = O(n2)

i.e., we gained nothing with our divide-and-conquer!

Some history: In 1952, one of the most famous mathematicians ofthe 20th century, Andrey Kolmogorov, conjectured that you cannotmultiply in less than Ω(n2) elementary operations. In 1960, Karatsuba,then a 23-year-old student, found an algorithm (later it was called“divide and conquer”) that multiplies two n-digit numbers inΘ(nlog2 3) ≈ Θ(n1.58...) elementary steps, thus disproving theconjecture!! Kolmogorov was shocked!

COMP3121/3821/9101/9801 16 / 1

The Karatsuba trick

How did Karatsuba do it??

Take again our two input numbers A and B, and split them into twohalves:

A = A12n2 + A0 XX . . .X︸︷︷︸XX . . .X︸︷︷︸

B = B12n2 + B0

n

2

n

2

• AB can now be calculated as follows:

AB = A1B12n + (A1B0 + A0B1)2

n2 + A0B0

= A1B12n + ((A1 + A0)(B1 + B0)−A1B1 −A0B0)2

n2 + A0B0

Thus, the algorithm will look like this:

COMP3121/3821/9101/9801 17 / 1

1: function Mult(A,B)2: if |A| = |B| = 1 then return AB3: else4: A1 ← MoreSignificantPart(A);5: A0 ← LessSignificantPart(A);6: B1 ← MoreSignificantPart(B);7: B0 ← LessSignificantPart(B);8: U ← A0 + A1;9: V ← B0 + B1;

10: X ←Mult(A0,B0);11: W ←Mult(A1,B1);12: Y ←Mult(U,V);13: return W 2n + (Y −X −W ) 2n/2 + X14: end if15: end function

COMP3121/3821/9101/9801 18 / 1

The Karatsuba trick

Since

T (n) = 3T

(n

2

)+ c n

implies

T

(n

2

)= 3T

(n

22

)+ c

n

2

and

T

(n

22

)= 3T

(n

23

)+ c

n

22

. . .

we get

T (n) = 3T

(n

2

)︸︷︷︸+c n = 3

3T

(n

22

)+ c

n

2︸︷︷︸+ c n

= 32T

(n

22

)︸︷︷︸+c

3n

2+ c n = 3

2

3T

(n

23

)+ c

n

22︸︷︷︸+ c

3n

2+ c n

= 33T

(n

23

)︸︷︷︸+c

32n

22+ c

3n

2+ c n = 3

3

3T

(n

24

)+ c

n

23︸︷︷︸+ c

32n

22+ c

3n

2+ c n = . . .

COMP3121/3821/9101/9801 20 / 1

The Karatsuba trick

T (n) = 3T(n2

)+ c n = 3

(3T(

n22

)+ cn

2

)+ c n = 32 T

(n

22

)︸︷︷︸+c 3n

2 + c n

= 32

3T

(n

23

)+ c

n

22︸︷︷︸+ c 3n

2 + c n = 33T(

n23

)+ c 32n

22+ c 3n

2 + c n

= 33 T

(n

23

)︸︷︷︸+c n

(32

22+ 3

2 + 1)

= 33

3T

(n

24

)+ c

n

23︸︷︷︸+ c n

(32

22+ 3

2 + 1)

= 34T(

n24

)+ c n

(33

23+ 32

22+ 3

2 + 1)

. . .

= 3blog2 ncT

(n

b2log2 nc

)+ c n

((32

)blog2 nc−1 + . . . + 32

22+ 3

2 + 1)

≈ 3log2 nT (1) + c n

(32

)log2 n−1

32−1

= 3log2 nT (1) + 2c n((

32

)log2 n − 1)

COMP3121/3821/9101/9801 21 / 1

The Karatsuba trick

So we got

T (n) ≈ 3log2 nT (1) + 2c n

((3

2

)log2 n

− 1

)

We now use alogb n = nlogb a to get:

T (n) ≈ nlog2 3T (1)+2c n(nlog2

32 − 1

)= nlog2 3T (1)+2c n

(nlog2 3−1 − 1

)= nlog2 3T (1) + 2c nlog2 3 − 2c n

= O(nlog2 3) = O(n1.58...) << n2

Please review the basic properties of logarithms and the asymptoticnotation from the review material (the first item at the class webpageunder “class resources”.)

COMP3121/3821/9101/9801 22 / 1

A Karatsuba style trick also works for matrices: Strassen’salgorithm for faster matrix multiplication

If we want to multiply two n× n matrices A and B, the product will be amatrix C also of size n× n. To obtain each of n2 entries in C we do nmultiplications, so matrix product by brute force is Θ(n3).

However, we can do it faster using Divide-And-Conquer;

We split each matrix into four blocks of (approximate) size n/2 × n/2:(a bc d

)·(e fg h

)=

(r st u

)(4)

We obtain: a e + b g = r a f + b h = s

c e + d g = t c f + d h = u

Prima facie, there are 8 matrix multiplications, each running in time T(n2

)and 4 matrix additions, each running in time O(n2), so such a directcalculation would result in time complexity governed by the recurrence

T (n) = 8T(n

2

)+ c n2

The first case of the Master Theorem gives T (n) = Θ(n3), so nothing gained.

COMP3121/3821/9101/9801 23 / 1

Strassen’s algorithm for faster matrix multiplication

However, we can instead evaluate:

A = a (f − h); B = (a + b)h; C = (c + d) e D = d (g − e);

E = (a + d) (e + h); F = (b− d)(g + h); H = (a− c) (e + f).

We now obtain

E + D −B + F = (a e + d e + a h + d h) + (d g − d e) − (a h + b h) + (b g − d g + b h− d h)

= a e + b g = r;

A + B = (a f − a h) + (a h + b h) = a f + b h = s;

C + D = (c e + d e) + (d g − d e) = c e + d g = t;

E + A− C −H = (a e + d e + a h + d h) + (a f − a h) − (c e + d e) − (a e− c e + a f − c f)

= c f + d h = u.

We have obtained all 4 components of C using only 7 matrix multiplicationsand 18 matrix additions/subtractions.

Thus, the run time of such recursive algorithm satisfiesT (n) = 7T (n/2) + O(n2) and the Master Theorem yieldsT (n) = Θ(nlog2 7) = O(n2.808).

In practice, this algorithm beats the ordinary matrix multiplication for n > 32.

COMP3121/3821/9101/9801 24 / 1

Next time:

1 Can we multiply large integers faster than O(nlog2 3

)??

2 Can we avoid messy computations like:

T (n) = 3T

(n

2

)+ cn = 3

(3T

( n

22

)+ c

n

2

)+ cn = 3

2T

( n

22

)+ c

3n

2+ cn

= 32(3T

( n

23

)+ c

n

22

)+ c

3n

2+ cn = 3

3T

( n

23

)+ c

32n

22+ c

3n

2+ cn

= 33T

( n

23

)+ cn

32

22+

3

2+ 1

=

= 33(3T

( n

24

)+ c

n

23

)+ cn

32

22+

3

2+ 1

=

= 34T

( n

24

)+ cn

33

23+

32

22+

3

2+ 1

=

. . .

= 3blog2 nc

T

(n

b2log2 nc

)+ cn

( 3

2

)blog2 nc−1+ . . . +

32

22+

3

2+ 1

≈ 3log2 n

T (1) + cn

(32

)log2 n − 1

32− 1

= 3log2 n

T (1) + 2cn

(( 3

2

)log2 n− 1

)

COMP3121/3821/9101/9801 25 / 1

That’s All, Folks!!

COMP3121/3821/9101/9801 26 / 1

Documents

Algorithms: COMP3121/3821/9101/9801