Parameter Estimation: A Glimpse of Backing Out, Sensitivity Analysis ... · Sensitivity in Initial Value •Frequently we don’t know the exact state of the system at a certain point

Parameter Estimation: A Glimpse of Backing Out, Sensitivity

Analysis & Meta-Analysis

Nathaniel Osgood

CMPT 858

Sources for Parameter Estimates

• Surveillance data

• Controlled trials

• Outbreak data

• Clinical reports data

• Intervention outcomes studies

• Calibration to historic data

• Expert judgement

• Systematic reviews

Content

redacted

for

copyright

compliance

Anderson & May

Introduction of Parameter Estimates

Non Obese

General

Population

Undx Prediabetic

Popn

Obese General

Population

Becoming Obese

Dx Prediabetic Popn

DevelopingDiabetesBeing Born Non

Obese Being Born At

Risk

Annual Likelihood of

Becoming Obese

Annual Likelihood of

Becoming Diabetic

Diagnosis of

prediabetics

undx uncomplicated

dying other causes

dx uncomplicated

dying otehr causes

Annualized ProbabilityDensity of prediabetic

recongnition

Non-Obese

Mortality

Annual Mortality Rate for

non obese population

Annualized MortalityRate for obese

population

<Annual Not at

Risk Births>

Annual Likelihood ofNon-Diabetes Mortality forAsymptomatic Population

<Annual at Risk

Births>

Obese Mortality

Dx Prediabetics

Recovering

Undx Prediabetics

Recovering

Annual Likelihood ofUndx Prediabetic

Recovery

Annual Likelihood of Dx

Prediabetic Recovery

<Annual Likelihood ofNon-Diabetes Mortality forAsymptomatic Population>

Hassmiller, K. M. (2006). The Impact of Smoking on Population-Level Tuberculosis Outcomes. Unpublished PhD, University of Michigan, Ann Arbor, Ann Arbor.

Preparation for Pooling



Example of Other Pooling


Pooled Results


Forest Plot


Forest Plot 2


Sensitivity Analyses

• Same relative or absolute uncertainty in different parameters may have hugely different effect on outcomes or decisions

• Help identify parameters that strongly affect – Key model results

– Choice between policies

• We place more emphasis in parameter estimation into parameters exhibiting high sensitivity

Types of Sensitivity Analyses

• Variables involved – One-way

– Multi-way

• Type of component being varied – Parameter sensitivity

analysis: Parameter values

– Structural sensitivity analysis: Examine effects of Model structure on results

• Type of variation – Single alternative values

– Monte Carlo analyses: Draws from probability distributions (many types of variations)

• Frequency of variation – Static (parameter retains

value all through simulation)

– Ongoing change: Stochastic process • Accomplished via Monte-Carlo

analyses

Example Spider Diagram

Content

redacted

for

copyright

compliance

• Each axis represents a % change in a particular parameter – This proportional change

is identical for the different parameters

• The distance assumed by the curve along that axis represents the magnitude of response to that change – Note that these

sensitivities will depend on the state of system!

http://www.niwotridge.com/images/BLOGImages/SpiderDiagram.jpg

Systematic Examination of Policies

08-13

Female14-18 19-24 25-34 35-44 45-54 55-64 65+ 08-13

Male14-18

19-24 25-3435-44 45-54

55-6465+

Cessation

Relapse

Uptake

0

50000

100000

150000

200000

250000

300000

350000

Cessation

Relapse

Uptake

Apply Different Uptake Rates for Children of Mothers who Smoked During Pregnancy? No linkage Time 2050

Sum of Total QALYs

InterventionGender Intervention Age Category

Smoking Status of Intervention Target

Tengs, T., Osgood, N., & Lin, T. (2001). Public health impact of changes in smoking behavior: results from the Tobacco Policy Model. Medical Care, 39(10), 1131-1141.

Sensitivity in Initial Value

• Frequently we don’t know the exact state of the system at a certain point in time

• A very useful type of sensitivity analysis is to vary the initial value of model stocks

• In Vensim, this can be accomplished by

– Indicating a parameter name within the “initial value” area for a stock

– Varying the parameter value

Imposing a Probability Distribution Monte Carlo Analysis

• We feed in probability distributions to reflect our uncertainty about one or more parameters

• The model is run many, many times (realizations)

– For each realization, the model uses a different draw from those probability distribution

• What emerges is resulting probability distribution for model outputs

Example Resulting Distribution

Empirical Fractiles

Impact on cost of uncertainty regarding mortality and medical costs

50% 60% 70% 80% 90% 95% 97% 99%

Incremental Costs

6 B

4.5 B

3 B

1.5 B

0 2001 2014 2026 2039 2051

Time (Year)

Static Uncertainty

Multi-Way Sensitivity Analyses

• When examining the results of changing multiple variables, need to consider how multiple variables vary together

• If this covariation reflects dependence on some underlying factor, may be able to simulate uncertainty in underlying factor

Performing Monte Carlo Sensitivity Analyses in Vensim

• Need to specify three things

– The parameters to vary

– How to vary those parameters

– Which model variables to save away

How & What Parameters to Vary

Model Values to Save Away

Monte Carlo Analyses

Sensitivity Results (Prevalence)

Sensitivity Results (Fraction of Susceptibles)

Stochastic Processes • Examples of things stochastically approximated

– Stock market

– Rainfall

– Oil prices

– Economic growth

• What considered “stochastic” will depend on the scope of the model

– Detailed model: Individual behaviour, transmission, etc.

– A meteorological model may not consider rainfall stochastic

Stochastic Processes

Baseline50% 60% 70% 80% 90% 95% 98% 100%

Average Variable Cost per Cubic Meter

0.6

0.45

0.3

0.15

00 1457 2914 4371 5828

Time (Day)

Dealing with Data Gradients • Often we don’t have reliable information on some

parameters, but do have other data – Some parameters may not be observable, but some

closely related observable data is available

– Sometimes the data doesn’t have the detailed breakdown needed to specifically address one parameter • Available data could specify sum of a bunch of flows or stocks

• Available data could specify some function of several quantities in the model (e.g. prevalence)

• Some parameters may implicitly capture a large set of factors not explicitly represented in model

• There are two big ways of dealing with this: manually “backing out”, and automated calibration

“Backing Out”

• Sometimes we can manually take several aggregate pieces of data, and use them to collectively figure out what more detailed data might be

• Frequently this process involves imposing some (sometimes quite strong) assumptions – Combining data from different epidemiological

contexts (national data used for provincial study)

– Equilibrium assumptions (e.g. assumes stock is in equilibrium. Cf deriving prevalence from incidence)

– Independence of factors (e.g. two different risk factors convey independent risks)

Example

• Suppose we seek to find out the sex-specific prevalence of diabetes in some population

• Suppose we know from published sources – The breakdown of the population by sex (cM, cF)

– The population-wide prevalence of diabetes (pT)

– The prevalence rate ratio of diabetes in women when compared to men (rrF)

• We can “back out” the sex-specific prevalence from these aggregate data (pF, pM)

• Here we can do this “backing out” without imposing assumptions

Backing Out

# male diabetics + # female diabetics = # diabetics

(pM* cM) + (pF* cF) = pT*(cM+cF)

• Further, we know that pF / pM =rrF=> pF = pM * rrF

• Thus

(pM* cM) + ((pM * rrF)* cF) = pT*(cM+cF)

pM*(cM + rrF* cF) = pT*(cM+cF)

• Thus

– pM = pT*(cM+cF) / (cM + rrF* cF)

– pF = pM * rrF = rrF * pT*(cM+cF) / (cM + rrF* cF)

Disadvantages of “Backing Out”

• Backing out often involves questionable assumptions (independence, equilibrium, etc.)

• Sometimes a model is complex, with several related known pieces

– Even thought we may know a lot of pieces of information, it would be extremely complex (or involve too many assumptions) to try to back out several pieces simultaneously

Another Example: Joint & Marginal Prevalence

Rural Urban

Male pMR pMU pM

Female pFR pMU pF

pR pU

Perhaps we know •The count of people in each { Sex, Geographic } category •The marginal prevalences (pR, pU , pM , pF)

We need at least one more constraint

•One possibility: assume pMR / pMU = pR/ pU We can then derive the prevalences in each { Sex, Geographic } category

Example Tying Together Meta-analysis & Calibration


Documents

Parameter Estimation: A Glimpse of Backing Out, Sensitivity Analysis ... · Sensitivity in Initial Value •Frequently we don’t know the exact state of the system at a certain point