AutoSimOA: A Framework for

Automated Analysis of Simulation Output

Stewart Robinson(stewart.robinson@warwick.ac.uk),

Katy Hoad, Ruth Davies

Funded by EPSRC and SIMUL8 Corporation

The Warwick Simulation Research Group

DES

ABSSD

7 members of staff2 research fellows4 PhD students

Focus on the practice and application of simulation methods

The Warwick Simulation Research Group

Recent/current projects:• Comparison of DES and SD model building• Agent based modelling of social networks• Effect of model reuse on learning• Conceptual modelling for DES• Agent based modelling for service systems• Human interactions in supply chains• Simulation and lean in the health service• …

The Problem

• Prevalence of simulation software: ‘easy-to-develop’ models and use by non-experts.

• Simulation software generally have very limited facilities for directing/advising user how to run the model to get accurate estimates of performance.

• With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

Aim

• To develop an automated output analysis system that can be implemented in commercial simulation software with a view to improving the use of simulation, particularly by non-expert simulation users.

• To develop an automated procedure that obtains unbiased estimators (of specified precision) for the population mean and variance (μ and σ2 respectively) for one or more simulation output statistics.

More formally…

Transient Simulation Output

0

20

40

60

80

100

120

140

08:00 -09:00

09:00 -10:00

10:00 -11:00

11:00 -12:00

12:00 -13:00

13:00 -14:00

14:00 -15:00

15:00 -16:00

16:00 -17:00

17:00 -18:00

Time

Nu

mb

er o

f cu

stom

ers

serv

ed

f(11:00-12:00)

Steady-State Simulation Output

0

20

40

60

80

100

120

140

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Day

Dai

ly t

hrou

ghpu

t

Initial transient Steady-state

f (steady-state)

Steady-statemean

3 Main Decisions

• How long a warm-up is needed?

• How many replications should be run?

• How long a run length is needed?

Work Carried Out for AutoSimOA Project

• Classification of different model types and output data properties.

• Extensive testing of replications algorithm.

• Literature review of (44) warm-up methods.

• Tested MSER-5 to destruction using over 3000 data sets.

• Literature review of batch means methods.

• Development of AutoSimOA.

Enter Analyser

Warm-up Analyser

Replications Calculator

Single Run Analyser

Replications

Single run

EXIT Analyser

AutoSimOAReplications or a single run?

Warm-up? Warm-up?

No NoYes Yes

Run

Model START:

Load Input

Produce Output Results

Run Replication Algorithm

Precision criteria met?

Recommend replication number

Run one more

replication

YES

NO

Replications Calculator

0.8

0.85

0.9

0.95

1

1.05

1.1

1.15

1.2

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Replication number (n)

Pre

cision

≤ 5

%

Pre

cision

> 5

% Precision ≤ 5%

f(kLimit)

Nsol2Nsol2 +

f(kLimit)Nsol1

95% confidence limits

Cumulative mean, X

Confidence Interval Method with ‘Look-ahead’

Warm-up Analyser

• MSER-5 most promising method for automation– Performs robustly and effectively for the

majority of data sets tested. – Not model or data type specific. – No estimation of parameters needed. – Can function without user intervention. – Quick to run. – Fairly simple to understand.

Dealing with Initialisation Bias

Warm-up Period: MSER-5 Heuristic

Minimises mean squared error of output data.

Performs analysis on batch mean data – batch size of 5.

MSER-5 value calculated as follows:

n

didni

jdnjYjY

jdjnjd

1

2,2

0)(

* )()()()(

1minarg)(

Dealing with Initialisation Bias

Warm-up Period: MSER-5 Heuristic

0

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0.016

0.018

0.02

0 50 100 150 200 250 300 350 400

Truncation Point

Test

Sta

tistic

0

1

2

3

4

5

6

Batc

h M

eans

MSER-5 test statistic

Output data (batched means values)

Rejection zone

Estimated warm-up period

Estimated truncation point

Heuristic framework around MSER-5

Adaptation in to a sequential procedure:

• Iterative procedure for procuring more data when required.

• ‘Failsafe’ mechanism - to deal with possibility of data not in steady state; insufficient data provided when highly auto-correlated.

• Graphical feedback to user.

Single Run Analyser

There are 3 possibilites:

1.User wants a mean estimate with a CI of a specific precision.

2.User has a specific run-length in mind & wants a mean estimate with a valid CI at end of run (i.e. no precision requirement).

3.User neither requires a specific precision nor has a set run length in mind.

Use set run-length?

Run-length Calculator

ASAP3(Steiger et al, 2005)

Batch Means Calculator

LABATCH2(Fishman, 1998)

SINGLE RUN ANALYSER

Abort

NO YES

Example Implementation of AutoSimOA

Data: • ‘user support model’ - simulates calls received,

processed and actioned at an IT support help desk (Robinson, 2001).

• Output of interest = average time calls spend in the system.

• Steady-state output with a substantial initial bias.

• True steady-state mean estimated as 2,269 mins (using a long run with 54,000 data points).

Implementation Issues

• Output data type – What should and should not be analysed?– Cumulative values, extreme values– Time or entity data

• Multiple outputs– Analyse all outputs of interest to user.

• Multiple scenarios– Run for all scenarios? Run for just the base

case? – Issues regarding run length with ASAP3.

Automation Issues• Generation of more data when required.

– Run simulation from present termination point.• Single run vs replications.

User involvement:• User decision of ‘what to do’- based on

knowledge of nature of model & output.– Warm-up needed? Multiple replications?– One run? Length of run for replications?

• Determining if recommendations are reasonable – Graphical aids.

Limitations of AutoSimOA

• Not directly able to handle cyclic data.

• Unable to analyse warm-up for transient output data subject to initialisation bias.

• Only performs an analysis on the mean and variance of the output statistics of interest. – Median, mode, quantiles,…

• Provides no facilities for scenario analysis.– Ranking and selection, optimisation,…

ACKNOWLEDGMENTSThis work is part of the Automating Simulation Output

Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by

the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in

collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project.

Stewart RobinsonWarwick Business School

Brunel DISC Seminar December 2009