Numerical Methods for Engineering Design and Optimization

Slide 1

18-660: Numerical Methods for Engineering Design and Optimization

Xin Li Department of ECE

Carnegie Mellon University Pittsburgh, PA 15213

Slide 2

Overview

Monte Carlo Analysis Latin hypercube sampling Importance sampling

Slide 3

Latin Hypercube Sampling (LHS)

A great number of samples are typically required in traditional Monte Carlo to achieve good accuracy

Various techniques exist to improve Monte Carlo accuracy

Controlling sampling points is the key Latin hypercube sampling is a widely-used method to generate

controlled random samples The basic idea is to make sampling point distribution close to

probability density function (PDF)

M. Mckay, R. Beckman and W. Conover, “A comparison of three methods for selecting values of input variables in the analysis of output from a computer code,” Technometrics, vol. 21, no. 2, pp. 239-245, May. 1979

Slide 4


One dimensional Latin hypercube sampling Evenly partition CDF into N regions Randomly pick up one sampling point in each region

x

CDF

0

1

Evenly partitioned regions

One random sample in one region

Avoid the probability that all sampling points come from the same local region

Slide 5


Two dimensional Latin hypercube sampling x1 and x2 must be independent Generate one-dimensional LHS samples for x1

Generate one-dimensional LHS samples for x2

Randomly combine the LHS samples to two-dimensional pairs

0 0.2 0.4 0.6 0.8 1 0

0.2

0.4

0.6

0.8

1

One sample in each row and each column

Sampling is random in each grid

Higher-dimensional LHS samples can be similarly generated

Prob

abilit

y of x

2

Probability of x1

Slide 6


Matlab code for LHS sampling of independent standard Normal distributions

data = rand(NSample,NVar); for i = 1:NVar index = randperm(NSample); prob = (index'-data(:,i))/NSample; data(:,i) = sqrt(2)*erfinv(2*prob-1); end;

NVar: # of random variables NSample: # of samples data: LHS sampling points

Slide 7


Compare Monte Carlo accuracy for a simple example x ~ N(0,1) (standard Normal distribution) Repeatedly estimate the mean value by Monte Carlo analysis

100 102 104

-0.5

0

0.5

# of Samples

Estim

ated

Mea

n Va

lue

100 102 104

-0.5

0

0.5

# of SamplesEs

timat

ed M

ean

Valu

e

Random sampling Latin hypercube sampling

Accuracy is improved by increasing sampling #

Monte Carlo is not deterministic

Slide 8

Importance Sampling

Even with Latin hypercube sampling, Monte Carlo analysis requires a HUGE number of sampling points

Example: rare event estimation

The theoretical answer for P(x ≤ -5) is equal to 2.87×10-7

~100M sampling points are required if we attempt to estimate this probability by random sampling or LHS

( )1,0~ Nx Standard Normal distribution

Estimate ( ) ???5 =−≤xP

Slide 9

Importance Sampling

Key idea: Do not generate random samples from pdfx(t) Instead, find a good distorted pdfy(t) to improve Monte Carlo

sampling accuracy

Example: if x ~ N(0,1), what is P(x ≤ -5)?

Intuitively, if we draw sampling points based on pdfy(t), more samples will fall into the grey area

t

PDF

0

pdfx(t) pdfy(t)

P(x≤-5)

How do we calculate P(x ≤ -5) when sampling pdfy(t)?

Slide 10

Importance Sampling

Assume that we want to estimate the following expected value

Example: if we want to estimate P(x ≤ -5), then

( )[ ] ( ) ( )∫+∞

∞−

⋅⋅= dttpdftfxfE x

( ) ( )( )

−>−≤

=5051

xx

xf

( )[ ] ( ) ( ) ( )5015

5

−≤=⋅⋅+⋅⋅= ∫∫+∞

−

−

∞−

xPdttpdfdttpdfxfE xx

Slide 11

Importance Sampling

Estimate E[f(x)] where x ~ pdfx(t) by importance sampling

( )[ ] ( ) ( ) ( ) ( ) ( )( )∫∫

+∞

∞−

+∞

∞−

⋅⋅⋅=⋅⋅= dttpdf

tpdftpdftfdttpdftfxfE

y

yxx

( )[ ] ( ) ( )( ) ( ) ( ) ( ) ( )[ ]xgEdttpdftgdttpdftpdftpdftfxfE yy

y

x =⋅⋅=⋅⋅

⋅= ∫∫

+∞

∞−

+∞

∞−

Slide 12

Importance Sampling

Estimate E[f(x)] where x ~ pdfx(t) by traditional sampling Step 1: draw M random samples {t1,t2,...,tM} based on pdfx(t) Step 2: calculate fm = f(tm) at each sampling point m = 1,2,...,M Step 3: calculate E[f] ≈ (f1+f2+...+fM)/M

Estimate E[f(x)] where x ~ pdfx(t) by importance sampling

Step 1: draw M random samples {t1,t2,...,tM} based on pdfy(t) Step 2: calculate gm = f(tm)⋅pdfx(tm)/pdfy(tm) at each sampling

point m = 1,2,...,M Step 3: calculate E[f] ≈ (g1+g2+...+gM)/M

How do we decide the optimal pdfy(t) to achieve minimal Monte Carlo analysis error?

Slide 13

Importance Sampling

Determine optimal pdfy(t) for importance sampling

The accuracy of an estimator can be quantitatively measured by its variance

To improve Monte Carlo analysis accuracy, we should minimize VAR[μf]

[ ] ( ) ( )( )∑

=

⋅⋅=≈M

m my

mxmf tpdf

tpdftfM

fE1

1µ

Estimator

[ ]fVARError µ~

Slide 14

Importance Sampling

Determine optimal pdfy(t) for importance sampling

We achieve the minimal VAR[μf] = 0 if

[ ] ( ) ( )( )∑

=

⋅⋅=≈M

m my

mxmf tpdf

tpdftfM

fE1

1µ

( ) ( )( )

ktpdf

tpdftfy

x =⋅ (Constant)

( ) ( ) ( )k

tpdftftpdf xy

⋅= (Optimal PDF)

f(tm)⋅pdfx(tm)/pdfy(tm) is equal to the same constant for all

sampling points

Slide 15

Importance Sampling

How do we decide the value k?

K cannot be arbitrarily selected pdfy(t) must be a valid PDF that satisfies the following condition

( ) ( ) ( )k

tpdftftpdf xy

⋅=

( ) ( ) ( ) 1=⋅⋅

=⋅ ∫∫+∞

∞−

+∞

∞−

dtk

tpdftfdttpdf xy

( ) ( ) [ ]fEdttpdftfk x =⋅⋅= ∫+∞

∞−

Finding the optimal pdfy(t) requires to know E[f], i.e., the

answer of our Monte Carlo analysis!!!

Slide 16

Importance Sampling

In practice, such an optimal pdfy(t) cannot be easily applied

Instead, we typically look for a sub-optimal solution that satisfies the following constraints Easy to construct – we do not have to know k = E[f] Easy to sample – not all random distributions can be easily

sampled by a random number generator Minimal estimator variance – the sub-optimal pdfy(t) is close to

the optimal case as much as possible

Slide 17

Importance Sampling

Finding the right pdfy(t) is nontrivial for practical problems No magic equation exists in general Engineering approach is based on heuristics Sometimes require a lot of human experience and numerical

optimization

The criterion to choose pdfy(t) is also application-dependent

Slide 18

Importance Sampling

Example: if x ~ N(0,1), what is P(x ≤ -5)?

Several possible choices for pdfy(t)

t

PDF

0

pdfx(t) Shifted Normal

P(x≤-5) Uniform

Slide 19

Summary

Monte Carlo analysis Latin hypercube sampling Importance sampling

Documents

Numerical Methods for Engineering Design and Optimization