Upload
mliss
View
24
Download
1
Embed Size (px)
DESCRIPTION
Model Validation as an Integrated Social Process. George P. Richardson Rockefeller College of Public Affairs and Policy University at Albany - State University of New York [email protected]. What do we mean by ‘validation’?. - PowerPoint PPT Presentation
Citation preview
1Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Model Validation asModel Validation asan Integrated Social Processan Integrated Social Process
George P. RichardsonRockefeller College of Public Affairs and Policy
University at Albany - State University of New [email protected]
2Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
What do we mean by ‘validation’?What do we mean by ‘validation’?• No model has ever been or ever will be thoroughly
validated. …‘Useful,’ ‘illuminating,’ or ‘inspiring confidence’ are more apt descriptors applying to models than ‘valid’ (Greenberger et al. 1976).
• Validation is a process of establishing confidence in the soundness and usefulness of a model. (Forrester 1973, Forrester and Senge 1980).
3Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
The classic questionsThe classic questions• Not ‘Is the model valid,’ but
• Is the model suitable for its purposes and the problem it addresses?
• Is the model consistent with the slice of reality it tries to capture? (Richardson & Pugh 1981)
4Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
The system dynamics modeling processThe system dynamics modeling process
SystemConceptualization
ModelFormulationRepresentation of
Model Structure
Comparison andReconcilation
Perceptions ofSystem Structure
Empirical andInferred Time
Series
Comparison andReconciliation.
Deduction OfModel Behavior
Adapted from Saeed 1992
5Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Processes focusing on system structureProcesses focusing on system structure
EmpiricalEvidence
SystemConceptualization
ModelFormulationRepresentation of
Model Structure
Comparison andReconcilation
Perceptions ofSystem Structure
Mental Models,Experience,Literature
Diagramming andDescription Tools
6Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Processes focusing on system behaviorProcesses focusing on system behavior
EmpiricalEvidence
SystemConceptualization
ModelFormulation
Literature,Experience
Empirical andInferred Time
Series
Comparison andReconciliation.
Deduction OfModel Behavior
ComputingAids
7Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Two kinds of validating processesTwo kinds of validating processes
EmpiricalEvidence
SystemConceptualization
ModelFormulationRepresentation of
Model Structure
Comparison andReconcilation
Perceptions ofSystem Structure
Mental Models,Experience,Literature
Literature,Experience
Empirical andInferred Time
Series
Comparison andReconciliation.
Deduction OfModel Behavior
Diagramming andDescription Tools
ComputingAids
StructureValidatingProcesses
BehaviorValidatingProcesses
8Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
The classic testsThe classic testsFocusing on
STRUCTUREFocusing on BEHAVIOR
Testing SUITABILITY for PURPOSES
• Dimensional consistency• Extreme conditions• Boundary adequacy
• Parameter insensitivity• Structure insensitivity
Testing CONSISTENCY with REALITY
• Face validity• Parameter values
• Replication of behavior• Surprise behavior• Statistical tests
Contributing to UTILITY & EFFECTIVENESS
• Appropriateness for audience
• Counterintuitive behavior• Generation of insights
Forrester 1973, Forrester & Senge 1980, Richardson and Pugh 1981
9Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Validation is present at every stepValidation is present at every step• Conceptualizing:
• Do we have the right people? • The right dynamic problem definition? • The right level of aggregation?
• Mapping: Developing promising dynamic hypotheses• Formulating: Clarity, logic, and extremes• Simulating: Right behavior for right reasons• Deciding: Implementable conclusions• Implementing: Requires conviction!
10Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Do we have the right people?Do we have the right people?
11Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Problem frame stakeholder mapProblem frame stakeholder map
Weak opponents Strong opponents
Weak supporters Strong supporters
Weak StrongStakeholder Power
High
Low
Low
High
Opp
ositi
onSu
ppor
t
Prob
lem
Fra
me
Bryson, Strategic Planning for Public and Nonprofit Organizations
12Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Power versus Interest gridPower versus Interest grid
Subjects Players
Crowd Context setters
Weak StrongPower
High
Low
Inte
rest
Eden & Ackerman 1998
13Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Pursuing validity in mappingPursuing validity in mapping• Think causally, not correlationally• Think stocks and flows, even if you don’t draw
them• Use units to make the causal logic plausible,
even if you don’t write them down• Be able to tell a story for every link and loop• Move progressively from less precise to more
precise -- from informal map to formal map
14Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
The standard cautionsThe standard cautions
Carbon inatmosphere
Prejudice
Discrimination
Opportunities forthe minority
Achievements ofthe minority
Understandingsof the system
Systemconceptualization
Model formulation& testing
Understandingsof the model
Carbon in algae,plants & trees
Carbon inherbivores
Carbon incarnivores
Carbon in soil
15Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
These arrows mean ‘and then’These arrows mean ‘and then’
Carbon inatmosphere
Prejudice
Discrimination
Opportunities forthe minority
Achievements ofthe minority
Understandingsof the system
Systemconceptualization
Model formulation& testing
Understandingsof the model
Carbon in algae,plants & trees
Carbon inherbivores
Carbon incarnivores
Carbon in soil
• We start with some understandings of the problem and its systemic context, and then we conceptualize (map) the system.
• Then we build the beginnings of a model, which we then test to understand it.
• Then we reformulate, or reconceptualize, or revise our understandings, or do some of all three, and then continue…
16Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Arrows here are Arrows here are flowsflows of material of material
Carbon inatmosphere
Prejudice
Discrimination
Opportunities forthe minority
Achievements ofthe minority
Understandingsof the system
Systemconceptualization
Model formulation& testing
Understandingsof the model
Carbon in algae,plants & trees
Carbon inherbivores
Carbon incarnivores
Carbon in soil
The words here represent stocks.
This is not a causal diagram.
17Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Only this one is a Only this one is a causalcausal loop loop
Carbon inatmosphere
Prejudice
Discrimination
Opportunities forthe minority
Achievements ofthe minority
Understandingsof the system
Systemconceptualization
Model formulation& testing
Understandingsof the model
Carbon in algae,plants & trees
Carbon inherbivores
Carbon incarnivores
Carbon in soil
No explicit stocks or flows, no clear units, but it tells a compelling story – It’s a good start.
18Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Project modeling core structureProject modeling core structure
Work tobe done
Workreally done
Undiscoveredrework
Knownrework rework
discovery
Work in processbeginning
workcompleting
work
doing workincorrectly
startingrework
19Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Identical structure Identical structure without explicit stocks and flowswithout explicit stocks and flows
beginningwork
completingwork
doing workincorrectly
Known rework
reworkdiscovery
startingrework
Undiscoveredrework
Work inprocess
Work reallydoneWork to be
done
20Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Pursuing validity writing equationsPursuing validity writing equations• Recognizable parameters• Robust equation forms• Phase relations • Richardson’s Rule: Every complicated, ugly,
excessively mathematical equation and every equation flaw saps confidence in the model.
21Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Modeling conflict within & between nationsModeling conflict within & between nations
Population
LateralpressureInternational
conflict
Consequences ofconflict
Internalstress
Technology
Populationgrowth rateTechnologygrowth rate
Domesticconflict
+
+
+
--
+
+
+
+
-
+
+
Domesticadaptation
Potential forconflict
Potential forinternational
conflict
Adaptation
22Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Complexity & flaws destroy confidenceComplexity & flaws destroy confidence
• P of int'l conflict = DELAY FIXED ((Lateral pressure/10*Military force effect/Trade and bargaining leverage + International conflict)/Lateral conflict break point, 1 , 0)
• FlawsComplexity, discreteness, units confusion and disagreement, disembodied parameter, confusion of the effect of a concept [leverage] with the concept itself, and the wonder what keeps this probability between 0 and 1?
23Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Robust equation formsRobust equation forms
CumulativeprogressProgress
24Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Causal mish-mashCausal mish-mash
CumulativeprogressProgress
WorkersHours per person
per day
Normal effectiveness(tasks/hour)
Effect ofmotivation
Effect ofschedulepressure
Effect of ...
Workweek(days)
25Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Robust equation formulationsRobust equation formulations
CumulativeprogressProgress
Effort(hours/month)
Effectiveness(tasks/hour)
26Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Robust equation formulationsRobust equation formulations
CumulativeprogressProgress
Effort(hours/month)
Effectiveness(tasks/hour)
WorkersHours per person
per day
Workweek(days)
27Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Robust equation formulationsRobust equation formulations
CumulativeprogressProgress
Effort(hours/month)
Effectiveness(tasks/hour)
Normal effectiveness(tasks/hour)
Effect ofmotivation
Effect ofschedulepressure
Effect of ...
28Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Robust equation formulationsRobust equation formulations
CumulativeprogressProgress
Effort(hours/month)
Effectiveness(tasks/hour)
WorkersHours per person
per day
Normal effectiveness(tasks/hour)
Effect ofmotivation
Effect ofschedulepressure
Effect of ...
Workweek(days)
29Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Pursuing validity in equations: PhasingPursuing validity in equations: Phasing
Integratedinformation
Unintegratedinformation integrating info
Resourcesdevoted to info
integration adding to resourcesto integration
subtractingresources tointegration
Problemsgeneratedintegrating
info
generatingproblems
Effort tointegrating info
Ease ofintegrating
info
Willingness to cedecontrol of info
Perceived valueof integratedinformation
Problems generatedper info unit integrated
Pressure to allocateresources elsewhere
(R) Problemscompound
(R) Successenhances resources
(B) Lowhanging fruit
(B) Problemsimpede progress
Unitegratedinfo within
scopeScope of
integration effort
(B) Problemsthreaten scope
(R) Perceived valueenhances scope
(B) Problems robresources
30Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Phase relationsPhase relations Integratedinformation
Resourcesdevoted to info
integration adding to resourcesto integration
Perceived valueof integratedinformation
Constant Perceived Value suggests continually rising Resources, but that doesn’t seem correct
31Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Phase relationsPhase relations
Resourcesallocated tointegration
project adding to resourcesto integration
Perceived valueof integratedinformation
Resourcesplanned to info
integration
Time to allocateresources
Here, the Perceived Value of Integrated Information sets a planned level of resources
32Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Pursuing validity fitting to dataPursuing validity fitting to data• Generally, a weak test of model validity• Whole-model procedures
• Optimization• Partial-model procedures• Reporting results
• Graphically• Numerically: Theil statistics
33Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Example of weakness of fitting to dataExample of weakness of fitting to data• Logistic curve
• dx/dt = ax - bx2
• Gompertz curve• dx/dt = ax - bx ln(x)
CumulativeproductionPetroleum
production
Discovery / productionexperience & technology
Constraints from theresource remaining
(R)
(B)
34Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Cum Production2 M2 M1 M1 M0018801902192419461968199020122034205620782100Time (year)
Fitting global petroleum with LogisticFitting global petroleum with LogisticProduction40,00040,00020,00020,0000018801902192419461968199020122034205620782100Time (year)
35Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Cum Production4 M2 M2 M1 M0018801902192419461968199020122034205620782100Time (year)
Fitting global petroleum with GompertzFitting global petroleum with GompertzProduction40,00030,00020,00010,000018801902192419461968199020122034205620782100Time (year)
36Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Presenting model fit visuallyPresenting model fit visually
Time1975 1978 1981 1983 1986
-4.00e+9
-2.00e+9
0.0
2.00e+9
4.00e+9
InsuranceAssets Hist_assets
3/21/89 9:00:46 AMTime
1975 1978 1981 1983 1986-4.00e+9
-2.00e+9
0.0
2.00e+9
4.00e+9
LossLiability Hist_liabilities
3/21/89 9:00:46 AM
Time1975 1978 1981 1983 1986
-0.800
-0.400
0.0
0.400
0.800
Prem_inc_granted Hist_prem_increases
3/21/89 9:00:46 AMTime
1975 1978 1981 1983 1986-4.00e+9
-2.00e+9
0.0
2.00e+9
4.00e+9
InsuranceSurplus Hist_surplus
3/21/89 9:00:46 AM
37Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Presenting model fit numericallyPresenting model fit numerically• Theil statistics, for example
• Based on a breakdown of the mean squared error:
• 1 = Bias + Variation + Covariation
1/ n 2(Xi −Yi)∑ = 2(X −Y ) + 2( xs − ys ) + 2(1−r) xs ys
38Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Presenting model fit numericallyPresenting model fit numerically
39Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Learning from surprise model behaviorLearning from surprise model behavior• Have clear a priori expectations• Follow up all unanticipated behavior to
appropriate resolution• Confirm all behavioral hypotheses through
appropriate model tests (Mass 1991/1981)
40Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
Tests to reveal and resolve surprise behaviorTests to reveal and resolve surprise behavior
• Testing the symmetry of policy response (up and down)• Testing large amplitude versus small amplitude response• Testing policies entering at different points • Testing different patterns of behavior• Isolating uniqueness of equilibrium or steady state• Understanding forces producing equilibrium positions
(Mass 1991/1981)
41Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
SummarySummary• Modelers, stakeholders, problem experts, and others in
the modeling process pursue validity at every step along the way.
• We have rigorous traditions guiding model creation, formulation, exploration, and implications.
• We have a powerful, intimidating battery of tests of model structure and behavior.
• Model-based conclusions that make it through all this deserve the confidence of everyone in the process.
42Rockefeller Collegeof Public Affairs and Policy
University at AlbanyState University of New York
EpilogEpilog
• Reason is itself a matter of faith. It is an act of faith to assert that our thoughts have any relation to reality. (G.K. Chesterton)
• I have no exquisite reason for’t, but I have reason good enough. (Sir Andrew, Twelfth Night)