Markov Models: OverviewMarkov Models: Overview

Gerald F. Kominski, Ph.D.Gerald F. Kominski, Ph.D.Professor, Department of Health ServicesProfessor, Department of Health Services

Markov Models: Why Are They Necessary?Markov Models: Why Are They Necessary?

Conventional decision analysis models Conventional decision analysis models assume:assume:- Chance eventsChance events- Limited time horizonLimited time horizon- Events that do not recurEvents that do not recur

What happens if we have a problem with:What happens if we have a problem with:- An extended time horizon, say, over a lifetimeAn extended time horizon, say, over a lifetime- Events can reoccur throughout a lifetimeEvents can reoccur throughout a lifetime

Decision Tree for Atrial FibrillationDecision Tree for Atrial Fibrillation

State-Transition Diagram for Atrial FibrillationState-Transition Diagram for Atrial Fibrillation

WellWell

Post-Post-StrokeStroke

DeadDead

pp1212=0.2=0.2

pp2222=0.9=0.9

pp3333=1.0=1.0

pp1111=0.7=0.7

pp2323=0.1=0.1

pp1313=0.1=0.1

The probabilities for all paths out of a state must sum to 1.0.The probabilities for all paths out of a state must sum to 1.0.

Death is known as an Death is known as an absorbing state,absorbing state, because individuals who enter that because individuals who enter that state cannot transition out of it.state cannot transition out of it.

Transition ProbabilitiesTransition Probabilities

WellWell Post-Post-StrokeStroke

DeadDead

WellWell 0.70.7 0.20.2 0.10.1

Post-Post-StrokeStroke

0.00.0 0.90.9 0.10.1

DeadDead 0.00.0 0.00.0 1.01.0

State of State of Current CycleCurrent Cycle

State of Next CycleState of Next Cycle

Transition probabilities that remain constant over time Transition probabilities that remain constant over time are characteristic of are characteristic of stationary Markov models, aka stationary Markov models, aka Markov chainsMarkov chains

Markov Model DefinitionsMarkov Model Definitions

Any process evolving over time with uncertainty is a Any process evolving over time with uncertainty is a stochastic process, stochastic process, and models based on such and models based on such processes are stochastic or probabilistic modelsprocesses are stochastic or probabilistic models

If the process is both stochastic and the behavior of the If the process is both stochastic and the behavior of the model in one time period (i.e., cycle) does not depend model in one time period (i.e., cycle) does not depend on the previous time period, the process is on the previous time period, the process is MarkovianMarkovian- The process has “lack of memory”The process has “lack of memory”- Even processes where the previous state does matter can be Even processes where the previous state does matter can be

made Markovian through definition of temporary states know made Markovian through definition of temporary states know as as tunnel statestunnel states

Tunnel StatesTunnel States

WellWellPost-Post-StokeStoke 11

Post-Post-StrokeStroke 22

Post-Post-StrokeStroke 33

Post-Post-StrokeStroke

DeadDead

Defining a Markov ModelDefining a Markov Model

Define the initial statesDefine the initial states

Determine the cycle lengthDetermine the cycle length

Consider possible transitions among statesConsider possible transitions among states

Determine transition probabilitiesDetermine transition probabilities

Determine utilities, and costs (if cost-effectiveness Determine utilities, and costs (if cost-effectiveness analysis), for each stateanalysis), for each state

Evaluating Markov Models:Evaluating Markov Models:Cohort SimulationCohort Simulation

StateState

CycleCycle WellWell Post-Post-StrokeStroke

DeadDead Sum of Sum of Years LivedYears Lived

SurvivalSurvival

00 10,00010,000 00 00

11 7,0007,000 2,0002,000 1,0001,000 9,0009,000 0.90000.9000

22 4,9004,900 3,2003,200 1,9001,900 8,1008,100 0.81000.8100

33 3,4303,430 3,8603,860 2,7102,710 7,2907,290 0.72900.7290

44 2,4012,401 4,1604,160 3,4393,439 6,5616,561 0.65610.6561

55 1,6811,681 4,2244,224 4,0954,095 5,9055,905 0.59050.5905

66 1,1761,176 4,1384,138 4,6864,686 5,3145,314 0.53140.5314

77 824824 3,9593,959 5,2175,217 4,7834,783 0.47830.4783

9393 00 11 9,9999,999 11 0.00010.0001

9494 00 00 10,00010,000 00 0.00000.0000

The data in the last column is used to produce a survival curve, aka a Markov trace.The data in the last column is used to produce a survival curve, aka a Markov trace.

Estimating Markov Models:Estimating Markov Models:Monte Carlo SimulationMonte Carlo Simulation

Instead of processing an entire cohort and applying Instead of processing an entire cohort and applying probabilities to the cohort, simulate a large number probabilities to the cohort, simulate a large number (e.g., 10,000) cases proceeding through the transition (e.g., 10,000) cases proceeding through the transition matrixmatrix- Monte Carlo simulationMonte Carlo simulation- TreeAge will do this for you quickly, without programmingTreeAge will do this for you quickly, without programming

The advantage of this approach is that it provides The advantage of this approach is that it provides estimates of variation around the meanestimates of variation around the mean

Monte Carlo simulation is most valuable because it Monte Carlo simulation is most valuable because it permits efficient modeling of complex prior historypermits efficient modeling of complex prior history- Such variables are known as Such variables are known as tracker variablestracker variables

Example of a 5-State MarkovExample of a 5-State Markov

Source: Kominski GF, Varon SF, Morisky DE, Malotte CK, Ebin VJ, Coly A, Chiao C. Costs and cost-effectiveness of adolescent compliance with treatment for latent tuberculosis infection: results from a randomized trial. Journal of Adolescent Health 2007;40(1):61-68.

Key Assumptions of the Markov ModelKey Assumptions of the Markov Model

Variable Value (Range) ReferenceEfficacy of IPT 0.85 (0.75-0.98) 19Cost of treating active TB $22,500 ($17,000-$30,000) 17 Cost of IPT Varies by study group and whether 6-

month IPT is completedCurrent study

TB cases per 100,000 250 (120-560) 20TB case fatality rate 0.0045-0.16 (varies with age) 17All-cause mortality rate per 100,000

19-15,476 (varies with age) National Center for Health Statistics, 1999 mortality tables

Hepatotoxicity of IPT 0.0008 (age<35, started IPT)0.0012 (age<35, completed IPT)

21

Hepatitis fatality rate 0.002 21  Cost of treating IPT-induced hepatitis

$11,250 ($8,500-$15,000) Authors’ assumption

QALY – Healthy 1.00 (0.95-1.00) Authors’ assumptionQALY – Positive Skin Test, but Incomplete IPT 0.90 (0.80-0.95) Authors’ assumption

QALY – Active TB 0.50 (0.20-0.90) Harvard Center for Risk Analysis

QALY – IPT-induced hepatitis 0.75 (75-0.90) Harvard Center for Risk Analysis

Discount rate 0.03 (0.00-0.07) Panel on Cost-Effectiveness