Randomized Algorithms

Kyomin JungKAIST

Applied Algorithm Lab

Jan 12, WSAC 2010

1

Randomness in Algorithms

A Probabilistic Turing Machine is a TM given with a (binary) random “coin”.

In the computation process of the PTM, PTM can toss a coin to decide its decision.

Ex) Computer games, random screen savor…

Note: in computer programming, the random number generating function (ex, rand() in C) takes an initial random seed, and generates a “deterministic” sequence of numbers. Hence they are not “truly” random.

2

Why randomness can be helpful?

A Simple example Suppose we want to check whether an integer

set has an even number or not.

Even when A has n/2 many even numbers, if werun a Deterministic Algorithm, it may check n/2+1 many elements in the worst case.

A Randomized Algorithm: At each time, choose an elements (to check) at random. Smooths the “worst case input distribution”

into “randomness of the algorithm”

}...,,,{ 321 naaaaA

3

Property of “independent” random trials Law of Large Number

If we repeat independent random samplings many times according to a fixed distribution D, the “the average becomes close to the expectation” (ex: dice rolling)

Ex) Erdos-Renyi Random Graph G(n,p). Voting poll …

4

• Suppose we have a coin with probability of heads is p.• Let X be a random variable where X=1 if the coin flip gives heads and X=0 otherwise. E[X] = 1*p + 0*(1-p) = p

• After flipping a coin w times we can estimate the heads prob by average of xi. • The Chernoff Bound tells us that this estimate converges exponentially fast to the true mean p. wxp

w

iiw

2

1

1 expPr

Chernoff Bound

5

Las Vegas vs Monte Carlo

A Las Vegas algorithm is a randomized algorithm that always gives correct results

Ex: randomized quick sort

A Monte Carlo algorithm is one whose running time is deterministic, but whose output may be correct only with a certain probability.

Class BPP (Bounded-error, Probabilistic, Polynomial time) problem which has a Monte Carlo algorithm to solve it

6

Randomized complexity classes

Model: probabilistic Turing Machine

BPP L BPP if there is a p.p.t. TM M:

x L Pry[M(x,y) accepts] ≥ 2/3

x L Pry[M(x,y) rejects] ≥ 2/3

“p.p.t” = probabilistic polynomial time y : (a sequence of) random coin tosses

7

Randomized complexity classes

RP (Random Polynomial-time) L RP if there is a p.p.t. TM M:

x L Pry[M(x,y) accepts] ≥ ½

x L Pry[M(x,y) rejects] = 1

coRP (complement of Random Polynomial-time) L coRP if there is a p.p.t. TM M:

x L Pry[M(x,y) accepts] = 1

x L Pry[M(x,y) rejects] ≥ ½

8

Error reduction for BPP

given L, and p.p.t. TM M:x L Pry[M(x,y) accepts] ≥ ½ + ε

x L Pry[M(x,y) rejects] ≥ ½ + ε

new p.p.t. TM M’: simulate M k/ε2 times, each time with

independent coin flips accept if majority of simulations accept otherwise reject

9

Error reduction for BPP

Xi random variable indicating “correct” outcome in i-th simulation (out of m = k/ε2 )

Pr[Xi = 1] ≥ ½ + ε Pr[Xi = 0] ≤ ½ - ε

E[Xi] ≥ ½+ε

X = ΣiXi

μ = E[X] ≥ (½ + ε)m

By Chernoff Bound: Pr[X ≤ m/2] ≤ 2-(ε2 m)

10

Derandomization

Question: is BPP=P? Goal: try to simulate BPP in polynomial time use Pseudo-Random Generator (PRG):

Good if t=O(log m)

Blum-Micali-Yao PRGseed length t = mδ

seed output stringGt bits m

bits

11

Example of randomized algorithm : Minimum Cut Problem

A

B

C

D

Note: Size of the min cut must is no largerthan the smallest node degree in graph

12

Application: Internet Minimum Cut

13

Randomized Algorithm (by David Karger)

While |V| > 2:Pick a random edge (x, y) from EContract the edge:

Keep multi-edges, remove self-loops Combine nodes

The two remaining nodes represent reasonable choices for the minimum cut sets

14

Analysis

Suppose C is a minimum cut (set of edges that disconnects G)

When we contract edge e:Unlikely that e C So, C is likely to be preserved

What is the probability a randomlychosen edge is in C?

15

Random Edge in C? |C| must be degree of every node in G

How many edges in G:|E| = sum of all node degrees / 2 n |C| / 2

Probability a random edge is in C 2/n

16

Iteration

How many iterations? Probability for first iteration:

Prob(e1 C) 1 – 2/n Probability for second iteration:

Prob(e2 C | e1 C) 1 – 2/(n-1) ... Probability for last iteration:

Prob(en-2 C |…) 1 – 2/(n-(n-2-1)) 1 – 2/3

n - 2

17

Probability of finding C?

Probability of not finding C = 1 – 2/(n(n-1))

32

22

122 1...111 nnn

31

42

53

24

132 ...

n

nnn

nn

)1(2 nn

18

Probability of finding C?

Probability of not finding C on one trial: 1 – 2/(n(n-1)) 1 – 2/n2

Probability of not finding C on k trials: [1 – 2/n2]k

If k = cn2, Prob failure (1/e)c

Recall: lim (1 – 1/x)x = 1/e x

19

Example of randomized algorithm :Random Sampling

What is a random sampling?Given a probability distribution , pick a

point according to .e.g. Monte Carlo method for integration

Choose numbers uniformly at random from the integration domain, and sum up the value of f at those points

20

How to use Random Sampling?

Volume computation in Euclidean space.

21

Hit and Run

Hit and Run algorithm is used to sample from a convex set in an n-dimensional Eucliden space.

Assume that we can evaluate its ratios at given points

It converges in time.)( 3nO

22

Example of randomized algorithm :Markov Chain Monte Carlo Let be a probability density

function on S={1,2,..n}. f(‧ ) is any function on S and we want to

estimate

Construct P={Pij}, the transition matrix of an

irreducible Markov chain with states S, where

and π is its unique stationary distribution.

1( ,..., )n

.

1

( ) ( )n

i

i

I E f f i

SXiXjXP tttij },|Pr{ 1

23

Markov Chain Monte Carlo

Run this Markov chain for times t=1,…,N and calculate the Monte Carlo sum

then

1

1ˆ { },N

t

t

I f XN

ˆ as .I I N

24

Applied Algorithm Lab (http://aa.kaist.ac.kr/)

Research Projects

Markov Random Field and Image segmentation

Network consensus algorithm design Community detection in complex networks Routing/scheduling algorithm analysis in

wireless networks Motion surveillance in video images Character recognition in images SAT solver design based on formula

structures Decentralized control algorithms for multi-

agent robot system Financial data mining

Yongsub Lim Boyoung Kim

Professor Graduate Students

Kyomin Jung Nam-ju Kwak

Members