Evaluating System Change Ventures Bobby Milstein Measurement Knowledge Network WHO Commission on Social Determinants of Health March 22, 2005 Bobby Milstein

Evaluating System Change VenturesEvaluating System Change Ventures

Bobby MilsteinMeasurement Knowledge Network

WHO Commission on Social Determinants of HealthMarch 22, 2005

Bobby MilsteinMeasurement Knowledge Network

WHO Commission on Social Determinants of HealthMarch 22, 2005

Innovations Toward a More Dynamic and Democratic Approach

Innovations Toward a More Dynamic and Democratic Approach

Plan for TodayPlan for Today

CDC Framework for Program Evaluation

Difference between research and evaluation

Link to institutional change

Steps and standards

Innovations for System Change Ventures

Syndemic orientation

Place of simulation modeling

CDC Framework for Program Evaluation

Difference between research and evaluation

Link to institutional change

Steps and standards

Innovations for System Change Ventures

Syndemic orientation

Place of simulation modeling

Appreciating the Unique Character of Evaluative Inquiry

Appreciating the Unique Character of Evaluative Inquiry

“It is easier to find facts than it is to face them.”

-- Anonymous

“It is easier to find facts than it is to face them.”

-- Anonymous

SystematicMethods

SystematicMethods

Centers for Disease Control and Prevention. What procedures are available for planning and evaluating initiatives to prevent syndemics? Syndemics Prevention Network, 2001. Available at <http://www.cdc.gov/syndemics/overview-planeval.htm>.

Research

Questions of Fact(descriptions, associations, effects)

Questions of Fact(descriptions, associations, effects)

Evaluation

Questions of Values(merit, worth, significance)

Questions of Values(merit, worth, significance)

Institutional Climate for Evaluative InquiryInstitutional Climate for Evaluative Inquiry

Milstein B, Chapel T, Wetterhall S, Cotton D. Building capacity for program evaluation at the Centers for Disease Control and Prevention. New Directions for Evaluation 2002;2002(93):27-47.

“The CDC’s senior leaders understood that strengthening

evaluation capacity in public health would require a process of culture

change, including significant reforms to their own organization.”

“The CDC’s senior leaders understood that strengthening

evaluation capacity in public health would require a process of culture

change, including significant reforms to their own organization.”

The Evaluation ContinuumThe Evaluation Continuum

Ways of EvaluatingWays of EvaluatingInformal

Low

Formal

HighStakes InvolvedStakes Involved

Framework for Program EvaluationFramework for Program Evaluation

Milstein B, Wetterall S, CDC Evaluation Working Group. Framework for program evaluation in public health. MMWR Recommendations and Reports 1999;48(RR-11):1-40. Available at <http://www.cdc.gov/mmwr/PDF/RR/RR4811.pdf>.

“Both a synthesis of existing evaluation practicesand a standard for further improvement.”

“Both a synthesis of existing evaluation practicesand a standard for further improvement.”

Framework for Program EvaluationFramework for Program Evaluation

Milstein B, Wetterall S, CDC Evaluation Working Group. Framework for program evaluation in public health. MMWR Recommendations and Reports 1999;48(RR-11):1-40. Available at <http://www.cdc.gov/mmwr/PDF/RR/RR4811.pdf>.

“A practical, nonprescriptive tool, designed to summarizeand organize the essential elements of program evaluation.”

“A practical, nonprescriptive tool, designed to summarizeand organize the essential elements of program evaluation.”

Utility (7)Serve information needs of intended users

Feasibility (3)Be realistic, prudent, diplomatic, and frugal

Propriety (8)Behave legally, ethically, and with due regard for the welfare of those involved and those affected

Accuracy (12)Reveal and convey technically accurate information





Standards for Effective EvaluationStandards for Effective Evaluation

Joint Committee on Educational Evaluation, James R. Sanders (Chair). The program evaluation standards: how to assess evaluations of educational programs. 2nd edition ed Thousand Oaks, CA: Sage Publications, 1994.

Catalyst for Complementary ResourcesCatalyst for Complementary Resources

CDC Evaluation Working Group. Framework for program evaluation: adapted versions. Available at <http://www.cdc.gov/eval/framework.htm>.

Are We Posing Questions About Attribution or Contribution?

Are We Posing Questions About Attribution or Contribution?

“…if a program’s activities are aligned with those of other programs operating in the same setting, certain effects (e.g., the creation of new laws or policies) cannot be attributed solely to one program or another. In such situations, the goal for evaluation is to gather credible evidence that describes each program’s contribution in the combined change effort. Establishing accountability for program results is predicated on an ability to conduct evaluations that assess both of these kinds of effects.” p.11-12

“…if a program’s activities are aligned with those of other programs operating in the same setting, certain effects (e.g., the creation of new laws or policies) cannot be attributed solely to one program or another. In such situations, the goal for evaluation is to gather credible evidence that describes each program’s contribution in the combined change effort. Establishing accountability for program results is predicated on an ability to conduct evaluations that assess both of these kinds of effects.” p.11-12

Calls into question the conditions in which one focuses on a “program” as the unit of analysis

Calls into question the conditions in which one focuses on a “program” as the unit of analysis

Acknowledging PluralityAcknowledging Plurality

Efforts to Reduce Population Health ProblemsProblem, problem solver, response

Efforts to Organize a System that Assures Healthful Conditions for All Dynamic interaction among multiple problems, problem solvers, and responses

Efforts to Reduce Population Health ProblemsProblem, problem solver, response

Efforts to Organize a System that Assures Healthful Conditions for All Dynamic interaction among multiple problems, problem solvers, and responses

Bammer G. Integration and implementation sciences: building a new specialisation. Cambridge, MA: The Hauser Center for Nonprofit Organizations, Harvard University 2003.

“You think you understand two because you understand one and one. But you must also understand ‘and’.”

-- Sufi Saying


-- Sufi Saying

A syndemic orientation clarifies the dynamic and democratic character

of public health work

The Syndemics Prevention NetworkThe Syndemics Prevention Network

The word syndemic signals special concern for many kinds of relationships:

mutually reinforcing health problems

health status and living conditions

synergy/fragmentation in the health response system

Learning within innovative ventures

Comprehensive Community InitiativesPhilanthropy

Legacy InitiativesState Tobacco Settlements

Efforts to Eliminate Health Disparities Government and Philanthropy

Responses to Unjust Conditions Broad-based Citizen Organizations

The word syndemic signals special concern for many kinds of relationships:

mutually reinforcing health problems

health status and living conditions

synergy/fragmentation in the health response system

Learning within innovative ventures

Comprehensive Community InitiativesPhilanthropy

Legacy InitiativesState Tobacco Settlements

Efforts to Eliminate Health Disparities Government and Philanthropy

Responses to Unjust Conditions Broad-based Citizen Organizations

Health

LivingConditions

Power toAct

Milstein B. Spotlight on syndemics. Centers for Disease Control and Prevention, 2001. <http://www.cdc.gov/syndemics>


-- Sufi Saying


-- Sufi Saying

Questioning the Character of Public Health WorkQuestioning the Character of Public Health WorkPUBLIC HEALTH WORK

InnovativeHealth

Ventures

SYSTEMS THINKING & MODELING (understanding change)

• What causes population health problems?

• How are efforts to protect the public’s health organized?

• How and when do health systems change (or resist change)?

PUBLIC HEALTH(setting direction)

What are health leaderstrying to accomplish?

SOCIAL NAVIGATION(governing movement)

Directing Change

Charting Progress

• Who does the work?• By what means?• According to whose values?

• How are conditions changing?• In which directions?

Formalizing an OrientationJoining Concepts and Methods

Formalizing an OrientationJoining Concepts and Methods

Syndemic Orientation

Network View

Connections

Proximity Data

What links to what?

Systems View

X Y

Leverage

Causal Data

What influences what?

Navigational View

Directed Public Work

Directional Data

Where are we going?

Changing (and Accumulating) Ideas in Causal Theory

What accounts for poor population health?

Changing (and Accumulating) Ideas in Causal Theory

What accounts for poor population health?

God’s will

Humors, miasma, atmosphere (“epidemic constitution”)

Poor living conditions, immorality (sanitation)

Single disease, single cause (germ theory)

Single disease, multiple causes (heart disease)

Single cause, multiple diseases (tobacco)

Multiple causes, multiple diseases (but no feedback dynamics) (social epidemiology)

Dynamic feedback among afflictions, living conditions, and public strength (syndemic)

God’s will

Humors, miasma, atmosphere (“epidemic constitution”)

Poor living conditions, immorality (sanitation)

Single disease, single cause (germ theory)

Single disease, multiple causes (heart disease)

Single cause, multiple diseases (tobacco)

Multiple causes, multiple diseases (but no feedback dynamics) (social epidemiology)

Dynamic feedback among afflictions, living conditions, and public strength (syndemic)

1880

1950

1960

1980

2000

1840

“When X and Y affect each other, one cannot study the link between X and Y and, independently, the link between Y and X and predict how the system will behave. Only the study of the whole

system as a feedback system will lead to correct results."

“When X and Y affect each other, one cannot study the link between X and Y and, independently, the link between Y and X and predict how the system will behave. Only the study of the whole

system as a feedback system will lead to correct results."

-- System Dynamics Society-- System Dynamics Society

The Feedback ThoughtThe Feedback Thought

System Dynamics Society. What is system dynamics? System Dynamics Society, 2002. Available at <http://www.systemdynamics.org/>.

Richardson GP. Feedback thought in social science and systems theory. Philadelphia: University of Pennsylvania Press, 1991.

System Dynamics Society. What is system dynamics? System Dynamics Society, 2002. Available at <http://www.systemdynamics.org/>.

Richardson GP. Feedback thought in social science and systems theory. Philadelphia: University of Pennsylvania Press, 1991.

X Y

A Very Particular DistanceA Very Particular Distance

“The macroscope filters details and amplifies that which links things together. It is not used

to make things larger or smaller but to observe what is

at once too great, too slow, and too complex for our eyes.”

“The macroscope filters details and amplifies that which links things together. It is not used

to make things larger or smaller but to observe what is

at once too great, too slow, and too complex for our eyes.”

Rosnay J. The macroscope: a book on the systems approach. Principia Cybernetica, 1997. <http://pespmc1.vub.ac.be/MACRBOOK.html

-- Joèel de Rosnay-- Joèel de Rosnay

Looking Through the MacroscopeLooking Through the Macroscope

Toward a Balanced System of Health ProtectionToward a Balanced System of Health Protection

SaferHealthierPeople Becoming

vulnerable

Becoming saferand healthier

VulnerablePeople Becoming

afflicted

Afflictedwithout

Complications Developingcomplications

Afflicted withComplications

Dying fromcomplications

Society's HealthResponse

Demand forresponse

Public Work

Adverse LivingConditions

GeneralProtection

TargetedProtection

PrimaryPrevention

SecondaryPrevention

TertiaryPrevention

From: Milstein B, Homer J. The dynamics of upstream and downstream: why is so hard for the health system to work upstream, and what can be done about it? CDC Futures Health Systems Workgroup; Atlanta, GA; 2003.

Balancing Two Major Areas of EmphasisBalancing Two Major Areas of Emphasis

SaferHealthierPeople

VulnerablePeople

Afflictedwithout

ComplicationsAfflicted with

ComplicationsBecomingvulnerable


Becomingafflicted

Developingcomplications




Demand forresponse

GeneralProtection

TargetedProtection

PrimaryPrevention

SecondaryPrevention

TertiaryPrevention

Public Work

World of Providing…

• Health education• Screening tests• Disease management • Pharmaceuticals• Clinical services• Physical and financial access• Etc…

Medical and Public Health Policy

MANAGEMENT OFRISKS & DISEASES

World of Transforming…

• Deprivation• Dependency• Violence• Disconnection• Environmental decay• Stress• Insecurity• Etc…

By Strengthening…

• Leaders and institutions• Foresight and precaution• The meaning of work• Mutual accountability• Plurality• Democracy• Freedom• Etc…

Healthy Public Policy & Public Work

DEMOCRATIC SELF-GOVERNANCE

Understanding Health as Public WorkUnderstanding Health as Public Work


VulnerablePeople

Afflictedwithout

Complications

Afflicted withComplicationsBecoming

vulnerable


Becomingafflicted





Demand forresponse

GeneralProtection

TargetedProtection

PrimaryPrevention

SecondaryPrevention

TertiaryPrevention

-

Public Work-

Vulnerable andAfflicted People

Fraction of Adversity,Vulnerability and AfflictionBorne by Disadvantaged

Sub-Groups (Inequity)

PublicStrength

-

Citizen Involvementin Public Life

Social Division

How Can Test a Dynamic Hypothesis?How Can Test a Dynamic Hypothesis?

-- How can we learn about the consequences of actions in a system of this kind?-- Could the behavior of this system be analyzed using conventional epidemoiological methods (e.g., logistic or multi-level regression)?


VulnerablePeople

Afflictedwithout

Complications

Afflicted withComplicationsBecoming

vulnerable


Becomingafflicted





Demand forresponse

GeneralProtection

TargetedProtection

PrimaryPrevention

SecondaryPrevention

TertiaryPrevention

-

Public Work-

Vulnerable andAfflicted People

Fraction of Adversity,Vulnerability and AfflictionBorne by Disadvantaged

Sub-Groups (Inequity)

PublicStrength

-

Citizen Involvementin Public Life

Social Division

Time Series ModelsDescribe trends

Multivariate Stat Models

Identify historical trend drivers and correlates

Patterns

Structure

Events

Increasing:

• Depth of causal theory

• Degrees of uncertainty

• Robustness for longer-term projection

• Value for developing policy insights

Increasing:

• Depth of causal theory

• Degrees of uncertainty

• Robustness for longer-term projection

• Value for developing policy insights

Dynamic Models

Anticipate future trends, and find policies that maximize chances

of a desirable path

Tools for Policy AnalysisTools for Policy Analysis

Average Number of Unhealthy Days per Month

2

4

6

8

1993 1995 1997 1999 2001

Year

2005 2025 2050

Redirecting the Course of ChangeQuestions from System Dynamics and Social Navigation

Redirecting the Course of ChangeQuestions from System Dynamics and Social Navigation

14% increase

Zack MM, Moriarty DG, Stroup DF, Ford ES, Mokdad AH. Worsening trends in adult health-related quality of life and self-rated health–United States, 1993-2001. Public Health Reports 2004;119(September-October):493-505.

How?Why?

Where?

Who?

Simulations for Learning in Dynamic SystemsThe Problem of Outside Assistance

Simulations for Learning in Dynamic SystemsThe Problem of Outside Assistance

Dynamic Hypothesis (Structure)Dynamic Hypothesis (Structure)

Behavior Over Time (Experiments)Behavior Over Time (Experiments)Affliction

prevalence& burden

Adverseliving

Publicstrength

R1

At-risk fraction

Afflictioncross-impacts

Effort to alleviate andprevent affliction

B1a

Effort to improveliving conditions

B1b

Effort to build public strength

B2

Social disparityR2c

R2b

R2a

R3a

Public work fraction

United efforts

Divided efforts

R3b

Outside assistance toalleviate and prevent

affliction

Outside assistanceto improve living

conditions

Outside assistanceto build public strength

Magnitude ofameliorative efforts

R4a

R4b B3b

B3a

KeyRectangle: Stock/state variableBlue arrow: same-direction linkGreen arrow: opposite-direction linkCircled “B”: balancing causal loopCircled “R”: reinforcing causal loop

conditions

12

10

8

6

4

0 2 4 6 8 10 12 14 16 18 20Time (years)

Affliction burden : BasicOptAffliction burden : BasicAF111Affliction burden : BasicLC111Affliction burden : BasicCS111

12

10

8

6

4

0 2 4 6 8 10 12 14 16 18 20Time (years)

Affliction burden : BasicOptAffliction burden : BasicAF111Affliction burden : BasicLC111Affliction burden : BasicCS111

Homer J, Milstein B. Optimal decision making in a dynamic model of poor community health. Proceedings of the 37th Hawaii International Conference on System Science; Big Island, Hawaii; January 5-8, 2004. Available at <http://csdl.computer.org/comp/proceedings/hicss/2004/2056/03/205630085a.pdf>.

Simulations for Learning in Dynamic SystemsDiabetes Dynamics in an Era of Epidemic Obesity

Simulations for Learning in Dynamic SystemsDiabetes Dynamics in an Era of Epidemic Obesity

Dynamic Hypothesis (Structure)Dynamic Hypothesis (Structure)

Behavior Over Time (Experiments)Behavior Over Time (Experiments)

Jones A, Homer J, Milstein B, Essien J, Murphy D, Sorensen S, Englegau M. Modeling the population dynamics of a chronic disease: the CDC's diabetes system model. American Journal of Public Health (in press).

People withUndiagnosed,Uncomplicated

Diabetes

People withDiagnosed,

UncomplicatedDiabetes

People withDiagnosed,Complicated

Diabetes

DiagnosingUncomplicated

Diabetes

People withUndiagnosedPreDiabetes

People withDiagnosedPreDiabetes

DiagnosingPreDiabetes

DevelopingComplications from

Undx diab

DevelopingDiabetes from Undx

PreD,People with

Undiagnosed,Complicated

Diabetes

DiagnosingComplicated

Diabetes

Dying from UndxComplications

People withNormal

GlycemicLevels

DiabetesDetection

Obese Fraction ofthe Population

Risk forPreDiabetes & Diabetes

Caloric Intake PhysicalActivity

PreDiabetesControl

DiabetesControl

PreDiabetesDetection

MedicationAffordability

Ability to SelfMonitor

Adoption ofHealthy Lifestyle

ClinicalManagement of

PreDiabetes

Clinical Managementof Diagnosed

Diabetes

LivingConditions

PersonalCapacity

PreDiabetesTesting for

Access toPreventive Health

Services Testing forDiabetes

PreDiabetesOnset

Recovering fromPreDiabete

s

Recovering fromPreDiabete

sDiabetesOnset

Dying fromComplications

DevelopingComplications

Deaths per Population

0.0035

0.003

0.0025

0.002

0.0015

1980 1990 2000 2010 2020 2030 2040 2050

Time (Year)

Blue: Base run; Red: Clinical mgmt up from 66% to 90%;Green: Caloric intake down 4% (99 Kcal/day);Black: Clin mgmt up to 80% & Intake down 2.5% (62 Kcal/day)

Base

Downstream

Upstream

Mixed

Striking an acceptable balance.

Iterative Steps in System Dynamics Simulation Modeling

Iterative Steps in System Dynamics Simulation Modeling

Enact PolicyBuild power and

organize actors to establish chosen policies

Enact PolicyBuild power and

organize actors to establish chosen policies

Learn About Policy Consequences

Test proposed policies, searching for ones that

best govern change

Learn About Policy Consequences

Test proposed policies, searching for ones that

best govern change

Run Simulation Experiments

Compare model’s behavior to expectations and/or data to

build confidence in the model

Run Simulation Experiments

Compare model’s behavior to expectations and/or data to

build confidence in the model

Convert the Map Into a Simulation Model

Formally quantify the hypothesis using allavailable evidence

Convert the Map Into a Simulation Model

Formally quantify the hypothesis using allavailable evidence

Create a Dynamic Hypothesis Identify and map the

main causal forces that create the problem

Create a Dynamic Hypothesis Identify and map the

main causal forces that create the problem

Identify a Persistent Problem Graph its behavior

over time

Identify a Persistent Problem Graph its behavior

over time

Milstein B, Homer J. Background on system dynamics simulation modeling, with a summary of major public health studies. Atlanta, GA: Syndemics Prevention Network, Centers for Disease Control and Prevention; February 1, 2005.

Learning In and About Dynamic SystemsLearning In and About Dynamic Systems

Benefits of Simulation/Game-based Learning

Formal means of evaluating options

Experimental control of conditions

Compressed time

Complete, undistorted results

Actions can be stopped or reversed

Visceral engagement and learning

Tests for extreme conditions

Early warning of unintended effects

Opportunity to assemble stronger support

Benefits of Simulation/Game-based Learning



Compressed time







Complexity Hinders

Generation of evidence (by eroding the conditions for experimentation)

Learning from evidence (by demanding new heuristics for interpretation)

Acting upon evidence (by including the behaviors of other powerful actors)

Complexity Hinders

Generation of evidence (by eroding the conditions for experimentation)

Learning from evidence (by demanding new heuristics for interpretation)

Acting upon evidence (by including the behaviors of other powerful actors)

Sterman JD. Learning from evidence in a complex world. American Journal of Public Health (in press).

Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston, MA: Irwin McGraw-Hill, 2000.

“The complexity of our models vastly exceeds our ability to understand their implications without simulation."

-- John Sterman


-- John Sterman

“Simulation is a third way of doing science. Like deduction, it

starts with a set of explicit assumptions. But unlike deduction,

it does not prove theorems. Instead, a simulation generates

data that can be analyzed inductively. Unlike typical induction,

however, the simulated data comes from a rigorously specified

set of rules rather than direct measurement of the real world.

While induction can be used to find patterns in data, and

deduction can be used to find consequences of assumptions,

simulation modeling can be used as an aid to intuition.”

“Simulation is a third way of doing science. Like deduction, it

starts with a set of explicit assumptions. But unlike deduction,

it does not prove theorems. Instead, a simulation generates

data that can be analyzed inductively. Unlike typical induction,

however, the simulated data comes from a rigorously specified

set of rules rather than direct measurement of the real world.

While induction can be used to find patterns in data, and

deduction can be used to find consequences of assumptions,

simulation modeling can be used as an aid to intuition.”

-- Robert Axelrod-- Robert Axelrod

Axelrod R. Advancing the art of simulation in the social sciences. In: Conte R, Hegselmann R, Terna P, editors. Simulating Social Phenomena. New York, NY: Springer; 1997. p. 21-40. <http://www.pscs.umich.edu/pub/papers/AdvancingArtofSim.pdf>.

A Third Branch of ScienceA Third Branch of Science

Revisiting the FrameworkRevisiting the Framework

Simulation Modeling Offers

Support for multi-stakeholder dialogue

A larger conception of the “program” context

Another avenue for experimentation and visceral learning

Ability to track interrelated indicators (both states and rates)

An emphasis on pragmatism (learning through action)

Simulation Modeling Offers

Support for multi-stakeholder dialogue

A larger conception of the “program” context

Another avenue for experimentation and visceral learning

Ability to track interrelated indicators (both states and rates)

An emphasis on pragmatism (learning through action)

“Steps in the framework are starting points for tailoring an evaluation to a particular public health effort at a particular time.”

“Steps in the framework are starting points for tailoring an evaluation to a particular public health effort at a particular time.”

For Additional Informationhttp://www.cdc.gov/syndemics

http://www.sp.uconn.edu/~geo101vc/Lecture22/sld030.htm

Enhancing Learning Through SimulationEnhancing Learning Through Simulation

Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.

Policy ResistancePolicy Resistance

Lee P, Paxman D. Reinventing public health. Annual Reviews of Public Health 1997;18:1-35.

Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982.

“At least six times since the

Depression, the United States has

tried and failed to enact a national

health insurance program.”

“At least six times since the

Depression, the United States has

tried and failed to enact a national

health insurance program.”

-- Lee & Paxman-- Lee & Paxman

“The tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself.”

“The tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself.”

-- Meadows, Richardson, Bruckman-- Meadows, Richardson, Bruckman

Flaws in Previous Attempts at Health System Reform in America (and Elsewhere)

Flaws in Previous Attempts at Health System Reform in America (and Elsewhere)

Heirich M. Rethinking health care: innovation and change in America. Boulder CO: Westview Press, 1999.

Kari NN, Boyte HC, Jennings B. Health as a civic question. American Civic Forum, 1994. Available at <http://www.cpn.org/topics/health/healthquestion.html>.

Piecemeal approaches that do not address the full scope of the problem

Comprehensive strategies that are opposed by special interests

Assumption that healthcare dynamics are separate from other areas of public concern

Conventional analytic methods make it difficult to

Observe the health system as a large, dynamic enterprise

Craft high-leverage strategies that can overcome policy resistance

Been thinking of health and healthcare primarily as nouns (i.e., commodities distributed to consumers), not as verbs (i.e., public work to be done by citizens)

Piecemeal approaches that do not address the full scope of the problem

Comprehensive strategies that are opposed by special interests

Assumption that healthcare dynamics are separate from other areas of public concern

Conventional analytic methods make it difficult to

Observe the health system as a large, dynamic enterprise

Craft high-leverage strategies that can overcome policy resistance

Been thinking of health and healthcare primarily as nouns (i.e., commodities distributed to consumers), not as verbs (i.e., public work to be done by citizens)

Basic Problem Solving OrientationsBasic Problem Solving Orientations

Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.

“Side Effects”

Feedback View

Goals

Environment

Action

Goals ofOthers

Action ofOthers

“Side Effects”

Delay Delay

Delay

Delay

DelayDelay

Delay

Delay

Delay

Delay

Delay

Delay

Event Oriented View

Problem Results

Goals

Situation

Action

Building on Decades of SD Health StudiesBuilding on Decades of SD Health Studies

Disease epidemiology heart disease, diabetes, HIV/AIDS, cervical cancer, chlamydia, dengue fever, drug-resistant infections

Substance abuse epidemiology heroin, cocaine, tobacco

Health care patient flows emergency care, extended care

Health care capacity and deliveryHMO planning, dental care capacity, mental health care, disaster preparedness

Interactions between health capacity and disease epidemiology

chronic illness management, syndemics

Disease epidemiology heart disease, diabetes, HIV/AIDS, cervical cancer, chlamydia, dengue fever, drug-resistant infections

Substance abuse epidemiology heroin, cocaine, tobacco

Health care patient flows emergency care, extended care

Health care capacity and deliveryHMO planning, dental care capacity, mental health care, disaster preparedness

Interactions between health capacity and disease epidemiology

chronic illness management, syndemics

Homer JB, Hirsch GB. System dynamics modeling for public health: background and opportunities. American Journal of Public Health (in press).

Progress in Dynamic ModelingProgress in Dynamic ModelingProblem Focus Dynamic Focus

Grantmaking strategy in areas challenged by multiple afflictions

Designed to explore interactions between afflictions, living conditions, and public strength

Balancing upstream and downstream effort

Designed to understand an observed phenomenon, the 97% -- 3% split in health care expenditures

Diabetes in an era of increasing obesity

Most empirically supported

Fetal and infant health

Health care reform policies

Life-course dynamics of obesity

Transforming the Future of Diabetes…Transforming the Future of Diabetes…

"Every new insight into Type 2 diabetes...

makes clear that it can be avoided--and

that the earlier you intervene the better.

The real question is whether we as a

society are up to the challenge...

Comprehensive prevention programs

aren't cheap, but the cost of doing

nothing is far greater..."

"Every new insight into Type 2 diabetes...

makes clear that it can be avoided--and

that the earlier you intervene the better.

The real question is whether we as a

society are up to the challenge...

Comprehensive prevention programs

aren't cheap, but the cost of doing

nothing is far greater..."

Gorman C. Why so many of us are getting diabetes: never have doctors known so much about how to prevent or control this disease, yet the epidemic keeps on raging. how you can protect yourself. Time 2003 December 8. Accessed at http://www.time.com/time/covers/1101031208/story.html.

…in an Era of Epidemic Obesity…in an Era of Epidemic Obesity

http://www.time.com/time/covers/1101031208/story.html

Re-Directing the Course of ChangeQuestions from System Modeling and Social Navigation

Re-Directing the Course of ChangeQuestions from System Modeling and Social Navigation

20202010

Prevalence of Diagnosed Diabetes, US

0

5

10

15

1980 1985 1990 1995 2000

Mill

ion

peop

le

Data Source: CDC DDT and NCCDPHP. -- Change in measurement in 1996.

How?

Why?

Where?

Who?

Diabetes System Modeling ProjectWhere is the Leverage for Health Protection?




Diabetes




Diabetes


People withDiagnosed

PreDiabetes

People withUndiagnosed,Complicated

DiabetesPeople with

NormalGlycemic

Levels

DiagnosingDiabetes

DiagnosingDiabetes

Diabetes Detection



Diabetes Control

PreDiabetes Detection


DiabetesOnset

PreDiabetes Control

PreDiabetesOnset

Recovering fromPreDiabetes


Obesity Prevention




Diabetes




Diabetes

DiagnosingUncomplicated

Diabetes


People withDiagnosed

PreDiabetes


DevelopingComplications from

Undx diab

DevelopingDiabetes from Undx

PreD,People with

Undiagnosed,Complicated

Diabetes

DiagnosingComplicated

Diabetes

Dying from UndxComplications

People withNormal

GlycemicLevels

DiabetesDetection

Obese Fraction ofthe Population

Risk forPreDiabetes & Diabetes

Caloric Intake PhysicalActivity

PreDiabetesControl

DiabetesControl

PreDiabetesDetection

MedicationAffordability

Ability to SelfMonitor

Adoption ofHealthy Lifestyle

ClinicalManagement of

PreDiabetes

Clinical Managementof Diagnosed

Diabetes

LivingConditions

PersonalCapacity

PreDiabetesTesting for

Access toPreventive Health

Services Testing forDiabetes

PreDiabetesOnset


Recovering fromPreDiabetes Diabetes

Onset



Deaths per Population

0.0035

0.003

0.0025

0.002

0.0015

1980 1990 2000 2010 2020 2030 2040 2050

Time (Year)


Base

Downstream

Upstream

Mixed


Using Available Data to Calibrate the ModelUsing Available Data to Calibrate the Model

Information Sources Data

U.S. Census Adult population and death rates

Health insurance coverage

National Health Interview Survey Diabetes prevalence

Diabetes detection

National Health and Nutrition Examination Survey

Prediabetes prevalence

Weight, height, and body fat

Caloric intake

Behavioral Risk Factor Surveillance System

Glucose self-monitoring

Eye and foot exams

Participation in health education

Use of medications

Professional Literature

Physical activity trends

Effects of control and aging on onset, progression, death, and costs

Expenditures

Diabetes System Modeling ProjectConfirming the Model’s Fit to History

Diabetes System Modeling ProjectConfirming the Model’s Fit to History


0%

2%

4%

6%

8%

1980 1985 1990 1995 2000 2005 2010

Diagnosed diabetes % of adults

Data (NHIS)

Simulated

0%

2%

4%

6%

8%

1980 1985 1990 1995 2000 2005 2010

Diagnosed diabetes % of adults

Data (NHIS)

Simulated

0%

10%

20%

30%

40%

1980 1985 1990 1995 2000 2005 2010

Obese % of adults

Data (NHANES)

Simulated

0%

10%

20%

30%

40%

1980 1985 1990 1995 2000 2005 2010

Obese % of adults

Data (NHANES)

Simulated

Diagnosed Diabetes % of AdultsObese % of Adults

Explaining the PastWhat Has Driven the Burden of Diabetes?

Explaining the PastWhat Has Driven the Burden of Diabetes?

Great Progress in Reducing the Burden

for the Average Person with Diabetes

Huge Growth in Number of People

with Diabetes

Overall, Total Diabetes Burden Held at Bay

Explaining the Past Deaths Due to Diabetes Have Fallen

Explaining the Past Deaths Due to Diabetes Have Fallen

Complications Deaths per Thous People w Diabetes40

30

20

10

0

1980 1985 1990 1995 2000 2005Time (Year)

People with Diabetes per Thousand Adults

100

90

80

70

60

501980 1985 1990 1995 2000 2005

Time (Year)

More people with diabetes

Deaths from Comps of Diabetes Per Thous Adults2.5

2

1.5

1

0.5

0

1980 1985 1990 1995 2000 2005Time (Year)

Model OutputModel Output

Model Output

Among people with diabetes, fewer dying every year

Combine to mean fewer U.S. adults dying 1980-2004

Anticipating the FuturePrevalence Under ‘Status Quo’ Assumptions

Anticipating the FuturePrevalence Under ‘Status Quo’ Assumptions

After adelay

Obese Fraction of Adult Population

0.6

0.45

0.3

0.15

0

Even if we assume the obesity epidemic has peaked…

People with Diabetes per Thousand Adults

130

110

90

70

50

1980 1990 2000 2010 2020 2030 2040 2050Time (Year)

Prevalence continues to increase.

Model OutputModel Output

1980 1990 2000 2010 2020 2030 2040 2050Time (Year)

People with Diabetes per Thousand Adults130

110

90

70

50

1980 1990 2000 2010 2020 2030 2040 2050Time (Year)

Complications Deaths per Thous w Diabetes40

30

20

10

0

1980 1990 2000 2010 2020 2030 2040 2050Time (Year)

Deaths from Complications of Diabetes Per Thousand Adults2.5

1.25

1980 1990 2000 2010 2020 2030 2040 2050Time (Year)

Then prevalence overwhelms the improved care to boost the burden

Anticipating the Future Deaths Under ‘Status Quo’ Assumptions

Anticipating the Future Deaths Under ‘Status Quo’ Assumptions

And if we can maintain current levels of care

but no continued improvement…

If prevalence continues to increase,

Policy ExperimentsPolicy Experiments

Continued downstream improvements

Upstream efforts

Downstream & Upstream

Continued downstream improvements

Upstream efforts

Downstream & Upstream

Deaths per Population0.0035

0.003

0.0025

0.002

0.0015

1980 1990 2000 2010 2020 2030 2040 2050Time (Year)

Downstream-Only InterventionDownstream-Only Intervention

Blue: Base run; Red: Clinical mgmt of diagnosed up from 66% to 90%

Base

Downstream

Disease control acts quickly but does not slow the growth in deaths.

Upstream-Only InterventionUpstream-Only InterventionDeaths per Population

0.0035

0.003

0.0025

0.002

0.00151980 1990 2000 2010 2020 2030 2040 2050

Time (Year)

Blue: Base run; Red: Clinical mgmt up from 66% to 90%;Green: Caloric intake down 4% (99 Kcal/day)

Downstream

UpstreamBase

Obesity prevention slows the growth but takes a long time to do so.

Deaths per Population0.0035

0.003

0.0025

0.002

0.00151980 1990 2000 2010 2020 2030 2040 2050

Time (Year)

Mixed InterventionMixed Intervention


Base

Downstream

Upstream

Mixed


QuestionsQuestions

Where might the results of simulation experiments fit in the hierarchies of evidence used for systematic reviews?

Where might the results of simulation experiments fit in the hierarchies of evidence used for systematic reviews?

Understanding Health as Public WorkUnderstanding Health as Public Work

Safer,Healthier

Population

VulnerablePopulation

Becomingvulnerable

Becoming nolonger vulnerable

Afflictedwithout

ComplicationsBecomingafflicted

Afflicted withComplications



Primary preventioneffect on incidence

-

Secondary preventioneffect on progression

-

Tertiary preventioneffect on complications

-

General protectioneffect on living

conditions

Targeted protectioneffect on vulnerability

reduction

GeneralProtection

TargetedProtection

TertiaryPrevention

SecondaryPrevention

PrimaryPrevention

Contacts withafflicted

DirectContagion

Vulnerable andAfflicted Popn

RiskDiffusionAdverse

LivingConditions

contributingto

Vulnerability

-

PublicStrength

Upstream work

Downstream work

Social disparity

-

SocialDivision

CitizenInvolvement

Effort to buildpublic strength

Social and economicadaptation to norms and

needs of risk popn

Professionalconcern

Public concern

-

Efforts to reduce ALC forreasons other than health

protection

-

Public workInstitutional/organizationalemphasis on disease rather

than vulnerability

-

Wickelgren I. How the brain 'sees' borders. Science 1992;256(5063):1520-1521.

How Many Triangles Do You See?How Many Triangles Do You See?

“Solutions” Can Also Create New Problems“Solutions” Can Also Create New Problems

Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936:894-904.

Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.

New Word for a Familiar PhenomenonNew Word for a Familiar Phenomenon

Singer M, Snipes C. Generations of suffering: experiences of a treatment program for substance abuse during pregnancy. Journal of Health Care for the Poor and Underserved 1992;3(1):222-34.

Singer M. 1994. AIDS and the health crisis of the US urban poor: The perspective of critical medical anthropology. Social Science and Medicine 39(7): 931-948.

Singer M. 1996. A dose of drugs, a touch of violence, a case of AIDS: Conceptualizing the SAVA syndemic. Free Inquiry in Creative Sociology 24(2): 99-110.

Singer M, Clair S. Syndemics and public health: reconceptualizing disease in bio-social context. Medical Anthropology Quarterly 2003;17(4):423-441.

“We have introduced the term ‘syndemic’ to refer to the set of synergistic

or intertwined and mutually enhancing health and social problems facing

the urban poor. Violence, substance abuse, and AIDS, in this sense, are not

concurrent in that they are not completely separable phenomena.”

“We have introduced the term ‘syndemic’ to refer to the set of synergistic

or intertwined and mutually enhancing health and social problems facing

the urban poor. Violence, substance abuse, and AIDS, in this sense, are not

concurrent in that they are not completely separable phenomena.”

-- Merrill Singer-- Merrill Singer

Strengths and Further FrontiersStrengths and Further Frontiers

Positioned Evaluation As…

Essential, not a question of whether, but rather how

Continuous, not left to the end

Inclusive, powerful roles for all stakeholders

Situational, decisions change based on participants, context, questions, uses, resources, etc.

Practical, judged against explicit standards

Humane, primarily about values and learning

Positioned Evaluation As…

Essential, not a question of whether, but rather how

Continuous, not left to the end

Inclusive, powerful roles for all stakeholders

Situational, decisions change based on participants, context, questions, uses, resources, etc.

Practical, judged against explicit standards

Humane, primarily about values and learning

Left Unexamined…

Focus on a singular “program” as the unit of inquiry (N=1 depth)

Limitations of logic (counterintuitive effects)

Dynamic aspects of public health work (e.g., better-before-worse patterns of change)

Democratic aspects of public health work (e.g., actions by citizens who don’t self-identify as health professionals)

Evaluative aspects of planning

Left Unexamined…

Focus on a singular “program” as the unit of inquiry (N=1 depth)

Limitations of logic (counterintuitive effects)

Dynamic aspects of public health work (e.g., better-before-worse patterns of change)

Democratic aspects of public health work (e.g., actions by citizens who don’t self-identify as health professionals)

Evaluative aspects of planning

“When we attribute behavior to

people rather than system structure

the focus of management becomes

scapegoating and blame rather than

the design of organizations in which

ordinary people can achieve

extraordinary results.”

“When we attribute behavior to

people rather than system structure

the focus of management becomes

scapegoating and blame rather than

the design of organizations in which

ordinary people can achieve

extraordinary results.”

-- John Sterman-- John Sterman

Sterman J. System dynamics modeling: tools for learning in a complex world. California Management Review 2001;43(4):8-25.

“The tendency to blame other people instead of the system is so strong

that psychologists call it the fundamental attribution error.”

“The tendency to blame other people instead of the system is so strong

that psychologists call it the fundamental attribution error.”

Beyond ScapegoatingBeyond Scapegoating









Standards for Effective EvaluationStandards for Effective Evaluation

Joint Committee on Educational Evaluation, James R. Sanders (Chair). The program evaluation standards: how to assess evaluations of educational programs. 2nd edition ed Thousand Oaks, CA: Sage Publications, 1994.

Institutional Climate for Evaluative InquiryInstitutional Climate for Evaluative Inquiry

Milstein B, Chapel T, Wetterhall S, Cotton D. Building capacity for program evaluation at the Centers for Disease Control and Prevention. New Directions for Evaluation 2002;2002(93):27-47.

Links to Cross-Cutting Trends

Expanding goals (length, quality, equity)

Wider ecological/systems view (assure the conditions for health)

Deepening democratic processes/outlook

Public accountability

Multi-sector action

Information systems integration

Public health research

Knowledge management

Systematic reviews

Performance standards

Ethical and moral considerations

Excellence in science and practice

Links to Cross-Cutting Trends

Expanding goals (length, quality, equity)

Wider ecological/systems view (assure the conditions for health)

Deepening democratic processes/outlook

Public accountability

Multi-sector action

Information systems integration

Public health research

Knowledge management

Systematic reviews

Performance standards

Ethical and moral considerations

Excellence in science and practice

“Living conditions are the everyday environment of people, where they live, play and work. These living conditions are a product of social and economic

circumstances and the physical environment–all of which can impact

upon health–and are largely outside of the immediate control of the individual.”

“Living conditions are the everyday environment of people, where they live, play and work. These living conditions are a product of social and economic

circumstances and the physical environment–all of which can impact

upon health–and are largely outside of the immediate control of the individual.”

-- World Health Organization-- World Health Organization

Definition:Living Conditions

Definition:Living Conditions

World Health Organization. Health promotion glossary. World Health Organization, 1998. Accessed July 15 at http://www.who.int/hpr/NPH/docs/hp_glossary_en.pdf.

http://broadcast.forio.com/sims/syndemic2003/

Public work is sustained, visible, serious effort by a diverse mix of ordinary people that creates things of lasting civic or public significance.

Public work is sustained, visible, serious effort by a diverse mix of ordinary people that creates things of lasting civic or public significance.

Mitchell Siporin. Jane Addams memorial. Illinois Federal Art Project, WPA, 1936. Fine Arts Collection, General Services Administration.

Center for Democracy and Citizenship. The concept and philosophy of public work. Center for Democracy and Citizenship, 2001. Available at <http://www.publicwork.org/1_2_philosophy.html>.

Definition:Public WorkDefinition:Public Work

-- Center for Democracy and Citizenship-- Center for Democracy and Citizenship

1. The power of citizens to direct the course of change toward a negotiated set of valued conditions/goals.

2. Vitality of a society’s public sphere, the health of its polis.

1. The power of citizens to direct the course of change toward a negotiated set of valued conditions/goals.

2. Vitality of a society’s public sphere, the health of its polis.

Lyon D. Tottle House. . .occupied during a sit-in by some of America's most effective organizers. Washington, DC: Library of Congress; 2002.

Lyon D. Tottle House. . .occupied during a sit-in by some of America's most effective organizers. Washington, DC: Library of Congress; 2002.

Definition:Public Strength

Definition:Public Strength

Milstein B. Syndemic. In: Mathison S, editor. Encyclopedia of Evaluation. Thousand Oaks, CA: Sage Publications; 2004.

What are the Prospects for Using aSyndemic Orientation?

What are the Prospects for Using aSyndemic Orientation?

A way of thinking about public health work that focuses on connections among

health-related problems, considers those connections when developing

health policies, and aligns with other avenues of social change to assure the

conditions in which people can be healthy

A way of thinking about public health work that focuses on connections among

health-related problems, considers those connections when developing

health policies, and aligns with other avenues of social change to assure the

conditions in which people can be healthy

Complements single-issue prevention strategies, which can be effective for discrete problems but often are mismatched to the goal of assuring conditions for health in its widest sense

Incorporates 21st century systems science and political sensibilities, but the underlying concepts are not new. Still, the implications of adhering to this orientation remain largely unexplored.

Complements single-issue prevention strategies, which can be effective for discrete problems but often are mismatched to the goal of assuring conditions for health in its widest sense

Incorporates 21st century systems science and political sensibilities, but the underlying concepts are not new. Still, the implications of adhering to this orientation remain largely unexplored.

Seeing Conditions as FreedomsSeeing Conditions as Freedoms

Adverse living conditions are circumstances that inhibit people's freedom to be safe and healthy and develop their full potential

They include, at a minimum, any deviation from prerequisite conditions for life and human dignity (e.g., physical extremes, violence, deprivation, disconnection)

Phenomena like hunger, homelessness, joblessness, illiteracy, war, environmental decay, and various forms of injustice, including racism, are all examples of adverse living conditions

Adverse living conditions are circumstances that inhibit people's freedom to be safe and healthy and develop their full potential

They include, at a minimum, any deviation from prerequisite conditions for life and human dignity (e.g., physical extremes, violence, deprivation, disconnection)

Phenomena like hunger, homelessness, joblessness, illiteracy, war, environmental decay, and various forms of injustice, including racism, are all examples of adverse living conditions

Choice and Non-ChoiceChoice and Non-Choice

Levins R, Lopez C. Toward an ecosocial view of health. International Journal of Health Services 1999;29(2):261-93.

“Choices are always made from among

alternatives presented by the social

environment, or by circumstances that were

themselves not chosen…When we recognize the

elements of non-choice in choice, we can escape

the contradiction between social causation and

individual responsibility and understand the

interactiveness of the two.”

“Choices are always made from among

alternatives presented by the social

environment, or by circumstances that were

themselves not chosen…When we recognize the

elements of non-choice in choice, we can escape

the contradiction between social causation and

individual responsibility and understand the

interactiveness of the two.”

-- Richard Levins & Cynthia Lopez

-- Richard Levins & Cynthia Lopez

Learning In and About Dynamic SystemsLearning In and About Dynamic Systems

Benefits of Simulation



Compressed time







Benefits of Simulation



Compressed time







Barriers to Learning

Dynamic complexity

Time delays

Inadequate and ambiguous feedback

Poor reasoning skills

Defensive reactions

Inability and costs of experimentation

Barriers to Learning

Dynamic complexity

Time delays

Inadequate and ambiguous feedback

Poor reasoning skills

Defensive reactions

Inability and costs of experimentation

Sterman JD. Learning from evidence in a complex world. American Journal of Public Health (in press).

Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston, MA: Irwin McGraw-Hill, 2000.


-- John Sterman


-- John Sterman

Curve-Fitting Models

Dynamic Models


Dynamic Models


Dynamic Models

Insight Generating

Capacity

Descriptive Realism

Mode Reproduction

Ability

Transparency

Relevance

Ease of Enrichment

Fertility

Formal Correspondence

with Data

Point Predictive Ability

Source: Randers JR. Elements of the System Dynamics Method. Cambridge Mass: Productivity Press, 1980. p. xx.


Dynamic Models

Different Modeling Approaches For Different Purposes

Different Modeling Approaches For Different Purposes

Logic Models

(flowcharts, maps or diagrams)

System Dynamics

(causal loop diagrams and simulation models)

Forecasting

Models

Articulate steps between program actions and results

Improve understanding about the possible effects of a policy over time

Focus on patterns of change over time (e.g., long delays, worse before better)

Make accurate forecasts of key variables

Focus on precision of point predictions and confidence intervals

Documents

Evaluating System Change Ventures Bobby Milstein Measurement Knowledge Network WHO Commission on Social Determinants of Health March 22, 2005 Bobby Milstein