Monte Carlo methods Agent-Based Modeling and Social ......Monte Carlo methods Agent-Based Modeling...

Monte Carlo methodsAgent-Based Modeling and Social SystemSimulationLecturer: Dr. Nino Antulov-Fantulin

Lecturer: Dr. Nino Antulov-Fantulin 1

Outline

1. Estimation

2. Sampling

3. Markov Chain Monte Carlo (MCMC)

4. Simulation* of stochastic dynamical processes

5. Optimization*

*Will be covered in Monte Carlo methods II lecture (18.11).

Lecturer: Dr. Nino Antulov-Fantulin 2

History...

The modern version of the Monte Carlo method was invented in thelate 1940s by Stanislaw Ulam, while he was working on nuclearweapons projects at the Los Alamos National Laboratory.Immediately after Ulam’s breakthrough, John von Neumannunderstood its importance and programmed the ENIAC computer tocarry out Monte Carlo calculations. *https://en.wikipedia.org/wiki/Monte_Carlo_method#History

Lecturer: Dr. Nino Antulov-Fantulin 3

Monte Carlo methods

Monte Carlo methods are a broad class of computational algorithmsthat rely on repeated random sampling to obtain numerical results.Used for:

• Estimation - integrals in high dimensional spaces

• Sampling - probability distributions

• Optimization - maximization of function

• Simulation

Lecturer: Dr. Nino Antulov-Fantulin 4

Why sampling and estimations?

Monte Carlo techniques play a fundamental role in statisticalphysics, modeling stochastic processes, machine learning andothers. ∫ ∫

...

∫f (x1, x2, ..., xn)dx1dx2...dxn (1)

p(x |y ) =p(y |x)p(x)∫

p(y |x ′)p(x ′)dx ′ (2)

ddt~P = A~P, (3)

(1) Statistical Mechanics: Algorithms and Computations, Book by Werner Krauth(2) Andrieu et al., An Introduction to MCMC for Machine Learning,(3) Gillespie, D. T. A rigorous derivation of the chemical master equation. Physica A: Statistical Mechanics and itsApplications 188, 404–425 (1992)

Lecturer: Dr. Nino Antulov-Fantulin 5

1. Monte Carlo Estimation

• Geometric view on estimating integrals over a functionf (x) : [a, b]→ [c, d ]:∫ b

a f (x)dx(b − a)(d − c)

≈ MN

=⇒∫ b

af (x)dx ≈ M

N(b − a)(d − c) (4)

where M denotes the number of samples that fall below f (x)and N denotes total number of samples.

• How good is this estimate?

Lecturer: Dr. Nino Antulov-Fantulin 6

1. Monte Carlo EstimationExample: Estimate the value of constant π if we have access to thegenerator of random numbers from uniform distributions [-1,1].

• Draw circle with radius 1 and square of length 2 centered atorigin.

• X,Y ∼ U[-1,1]

Z =

1, if√

X 2 + Y 2 ≤ 1

0, otherwise(5)

• M =∑N

i=1 Zi ,

2∫ b

af (x)dx = π̂ =

MN

(b − a)(d − c) =MN

4

How good is this estimate?Lecturer: Dr. Nino Antulov-Fantulin 7

1. Monte Carlo Estimation

Example: Estimate the value of constant π if we have access to thegenerator of random numbers from uniform distributions [-1,1].

• E [π̂] = π

• Chernoff bound: Let X1, ..., XN be independent Poisson trialsP(Xi = 1) = pi . Let X =

∑Ni=1 Xi and µ = E [x ]. For 0 < ε < 1,

P(|X − µ| ≥ εµ) ≤ 2exp(−µε2/3).

• P(|π̂ − π| ≥ επ) ≤ 2exp(−Nπε2/12)The last bound can be generalized for any set of i.i.d. indicator variables to provide (ε, δ)-approximation (see Probabilityand Computing, book by Mitzenmacher and Upfal).

Lecturer: Dr. Nino Antulov-Fantulin 8

1. Monte Carlo Estimation

• Algebra view on estimating integrals:

I =∫ b

af (x)dx = (b−a)

∫ b

a

1(b − a)

f (x)dx = E [(b−a)f (X )] (6)

where X denotes the random variable distributed with Uniformdistribution i.e. density f (x) = 1/(b − a).

Lecturer: Dr. Nino Antulov-Fantulin 9

1. Monte Carlo Estimation

E [(b − a)f (X )] ≈ IN =(b − a)

N

N∑i=1

f (xi ) (7)

• This estimate IN is unbiased and by the strong law of largenumbers, it will almost surely (a.s.) converge to I

• If the standard deviation of f (x) is finite σ <∞, then thestandard deviation of estimation IN is σ/

√N

• Central Limit Theorem (Lindeberg-Levy) states that thedistribution of errors converge in distribution to standardNormal distribution.

√N(IN − I) =⇒

N=∞N (0, σ2) (8)

Lecturer: Dr. Nino Antulov-Fantulin 10

1. Monte Carlo Estimation example

Estimate the integral

f (x) =∫ b

a

e−x

1 + x2 dx , (9)

where a = 0, b = 10 with N = 1000. What happens if b = 100?

Lecturer: Dr. Nino Antulov-Fantulin 11

2. Sampling

Up to this point we were assuming that we have access to thegenerator of Uniform numbers.What happens when we need to get the sample from differentdensity function p(x)?

Lecturer: Dr. Nino Antulov-Fantulin 12

2. Sampling

Rejection method for sampling with easy to sample proposal q(x).You can also use Uniform distribution for q(x).

Lecturer: Dr. Nino Antulov-Fantulin 13

2. SamplingMethod for sampling with easy to sample proposal q(x). Functionq(x) multiplied by constant M is always above function p(x) i.e.Mq(x) ≥ p(x).

i = 1repeat

Sample x (i) ∼ q(x), u ∼ U(0, 1)if u ∗Mq(x (i)) ≤ p(x (i)) then

accept x (i)

i + +else

rejectend if

until i==NLecturer: Dr. Nino Antulov-Fantulin 14

2. SamplingRejection-free method for sampling - Inverse Smirnov Transform.Probability integral transform: if F is c.d.f of X , thenY = F (X ) ∼ U[0, 1].

Lecturer: Dr. Nino Antulov-Fantulin 15

2. Sampling

Rejection-free method for sampling - Inverse Smirnov Transform

1. sample from f (x) density function

2. calculate cumulative density function F (x) =∫ x−∞ f (x)dx

3. calculate the inverse function F−1(x)

4. generate u from Uniform distribution [0,1]

5. sample is F−1(u)

Corollary of probability integral transform X = F−1(Y ) has c.d.f. F ifY ∼ U[0, 1].

Lecturer: Dr. Nino Antulov-Fantulin 16

2. Monte Carlo Importance sampling

We choose density function g(x) which is easy to sample and whichis positive g(x) > 0 whenever f (x) > 0.

I =∫ b

af (x)dx =

∫ b

a

f (x)g(x)

g(x)dx = E [f (Y )/g(Y )], (10)

where Y is a random variable with g(y ) density function.

E [f (Y )/g(Y )] ≈ IN =1N

∑i

f (G−1(xi ))g(G−1(xi ))

, (11)

where xi is the sample from Uniform distribution [0,1] and G−1(x) isthe inverse function of cumulative density functionG(x) =

∫ x−∞ g(x)dx i.e. Inverse Transform Sampling.

Lecturer: Dr. Nino Antulov-Fantulin 17

2. Importance sampling example

Estimate the integral

f (x) =∫ b

a

e−x

1 + x2 dx , (12)

where a = 0, b = 100.Which proposal to take?

Lecturer: Dr. Nino Antulov-Fantulin 18

2. Importance sampling example

Estimate the integral

f (x) =∫ b

a

e−x

1 + x2 dx , (13)

where a = 0, b = 100.

g(x) = C ∗ e−x . (14)

Lecturer: Dr. Nino Antulov-Fantulin 19

3. MCMC methods

Metropolis-Hastings is one algorithm in the MCMC class, whichfrom current sample σ randomly proposes another state σ′ byg(σ → σ′) proposal distribution and accepts it with probability

min(

1,P(σ′)g(σ′ → σ)P(σ)g(σ → σ′)

)

Lecturer: Dr. Nino Antulov-Fantulin 20

3. MCMC methods

The existence of stationary distribution of the Markov Chain indiscrete time and discrete states is guaranteed by detailed balanceproperty and the uniqueness by ergodicity.

Prove it!

Lecturer: Dr. Nino Antulov-Fantulin 21

3. MCMC methods

min(

1,P(σ′)g(σ′ → σ)P(σ)g(σ → σ′)

)This acceptance rule is direct consequence of imposing a detailedbalance condition P(σj )Wσj ,σi = P(σi )Wσi ,σj , which assures theconvergence to equilibrium.

Lecturer: Dr. Nino Antulov-Fantulin 22

3. MCMC methods

Andrieu et. al., An Introduction to MCMC for Machine LearningLecturer: Dr. Nino Antulov-Fantulin 23

Readings

• Christophe Andrieu, Nando de Freitas, Arnaud Doucet, MichaelI. Jordan, An Introduction to MCMC for Machine Learning,doi:10.1023/A:1020281327116

• Statistical Mechanics: Algorithms and Computations, Book byWerner Krauth

• Probability and Statistics, Book by DeGroot and Schervish

• Probability and Computing, Book by Mitzenmacher and Upfal

• Information, Physics and Computation, Book by Mezard andMontanari

Lecturer: Dr. Nino Antulov-Fantulin 24