Steady-State Statistical Analysis
By Dr. Jason Merrick
Simulation with Arena - Steady-state Output AnalysisC7/2
Warm Up and Run Length
• Most models start empty and idle– Empty: No entities present at time 0– Idle: All resources idle at time 0– In a terminating simulation this is OK if realistic– In a steady-state simulation, though, this can bias the
output for a while after startup• Bias can go either way
• Usually downward (results are biased low) in queueing-type models that eventually get congested
• Depending on model, parameters, and run length, the bias can be very severe
Simulation with Arena - Steady-state Output AnalysisC7/3
Warm Up and Run Length (cont’d.)
• The period up to 1500 minutes is less congested
• Thus average output measures will be biased down
• How can we get rid of this bias?
Simulation with Arena - Steady-state Output AnalysisC7/4
Intelligent Initial Conditions
• Collect data– Observe an actual state of the real system that has been
running for a reasonable period of time– Use this state as the initial conditions– Not possible if system does not exist or you are modifying
the system
• Use another model– Queuing models, inventory models etc.– Give steady-state results under more restrictive
assumptions than simulation– Use these results as initial conditions
Simulation with Arena - Steady-state Output AnalysisC7/5
Warm-up
• Define some time tW until which no statistics are collected
– Suppose mW observations are collected up to time tW
– Suppose m observations are collected after time tW
• The idea is that Ymw+1,…,Ym+mw are drawn from the
“steady state” distribution, while Ym,…,Ymw are
from a different warm-up distribution– So truncating the warm-up observations removes the bias
WWW mmmm YYYY ,.....,,,.... 11
Simulation with Arena - Steady-state Output AnalysisC7/6
Determining Warm-up Times
• Ensemble averages
– The average across replications of the first, second, third, … observations
– Each ensemble average is an iid sample from the distribution of that observation
– Put t-distribution confidence interval around each average
– See when the ensemble averages settle down
222
21
21
21
222221
111211
,,,
,,,
,,,
,,,
,,,
m
m
nnmnn
m
m
sss
YYY
YYYY
YYYY
YYYY
Series Averages
EnsembleAverages
Simulation with Arena - Steady-state Output AnalysisC7/7
Determining Warm-up Times
-20
-15
-10
-5
0
5
10
15
20
25
1 11 21 31 41 51 61 71 81 91
Observations
-8
-6
-4
-2
0
2
4
6
8
1 11 21 31 41 51 61 71 81 91
Ensemble Averages
Simulation with Arena - Steady-state Output AnalysisC7/8
Determining Warm-up Times
-100
-50
0
50
100
150
1 11 21 31 41 51 61 71 81 91
Observations
-40
-30
-20
-10
0
10
20
30
40
50
60
1 11 21 31 41 51 61 71 81 91
Ensemble Averages
Simulation with Arena - Steady-state Output AnalysisC7/9
Truncated Replications
• If you can identify appropriate warm-up and run-length times, just make replications as for terminating simulations– Only difference: Specify Warm-Up Period in Simulate
module
nmmnmn
mmm
mmm
YYY
YYY
YYY
nWW
WW
WW
,,,
,,
,,
,1,
2,21,2
1,11,1
2
1
Y
2s
Simulation with Arena - Steady-state Output AnalysisC7/10
Batching in a Single Run
• If model warms up very slowly, truncated replications can be costly– Have to “pay” warm-up on each replication– Throw away
W
W
W
mnn
m
m
YY
YY
YY
,1,
,21,2
,11,1
,,,
,,
,,
Simulation with Arena - Steady-state Output AnalysisC7/11
Batching in a Single Run
• Alternative: Just one R E A L L Y long run– Only have to “pay” warm-up once
– Problem: Have only one “replication” and you need more than that to form a variance estimate (the basic quantity needed for statistical analysis)• Big no-no: Use the individual points within the run as “data” for
variance estimate
• Usually correlated (not indep.), variance estimate biased
mmmm WWWYYYYY ,...,,,, 121
throw away sample
Simulation with Arena - Steady-state Output AnalysisC7/12
Batching in a Single Run (cont’d.)
• Break each output record from the run into a few large batches ckmkmkmkmkmmm WWWWWWW
YYYYYYYY ,...,,,..,,,...,,,..., 122111
Batch 1 Batch 2Warm-up…...
k
iimW
Yk
Y1
1
1
k
iikmW
Yk
Y1
2
1
k
iikjmj W
Yk
Y1
)1(
1
Simulation with Arena - Steady-state Output AnalysisC7/13
• The batch means will not actually be independent– The idea is to reduce the correlation to a level that it will not introduce a
significant bias in the estimate of the standard deviation
– The individual observations in the series are correlated with the previous observations
– The correlogram shows that the correlation reduces the higher the lag– So if the batch size is long enough then most of the observations
making up batch 1 will be approximately independent of those making up batch 2
– Only the observations near the end of the batches will be correlated
Batching in a Single Run (cont’d.)
kmkmkmm WWWWYYYY 211 ,..,,,...,
Batch 1 Batch 2
1Y 2Y
Simulation with Arena - Steady-state Output AnalysisC7/14
Batching in a Single Run (cont’d.)
• Rules of thumb– Schmeiser (1982) found that for a given run length, there was little
benefit in more than 30 batches– However, less than 10 batches was too few
– There may well be correlation between all lags, looking at the lag 1 correlation is usually enough to ascertain independence
– Auto-correlation estimates are not very good for sample sizes like 30– Use smaller batches (say c > 100) and if the independence test is
passed then the bigger batches will be fine
cYYY ,,, 21
Simulation with Arena - Steady-state Output AnalysisC7/15
Examining # of Batches
• Consider the Simple Processing System– t = 1,000,000 minutes, increase number of batches
15
16
17
18
19
20
8 10 15 20 25 30 35 100
# Batches
Av
era
ge
Flo
wti
me
CI too small CI doesn’t change much
Simulation with Arena - Steady-state Output AnalysisC7/16
Examining Run Length
• Consider the Simple Processing System– n = 25 replications, increase the run length
0
5
10
15
20
25
10 100 1000 10000 100000
Run Length
Av
era
ge
Flo
wti
me
Simulation with Arena - Steady-state Output AnalysisC7/17
• Consider the Simple Processing System– t = 15 minutes, increase the number of replications
Examining # of Replications
0
1
2
3
4
5
6
7
0 50 100 150 200 250
# replications
Av
era
ge
Flo
wti
me