1
Possible Roles for Reinforcement Learning in
Clinical Research
S.A. Murphy
November 14, 2007
2
OutlineGoal: Improving Clinical Decision Support
Systems Using Data
– Clinical Decision Support Systems– Critical Decisions– Types of Data– Challenges
• Incomplete, primitive, mechanistic models
• Measures of Confidence
– Clinical Trials
3
4
5
QuestionsPatient Evaluation Screen with MSE
6
7
Outline
– Clinical Decision Support Systems– Critical Decisions– Types of Data– Challenges
• Incomplete mechanistic models
• Measures of Confidence
– Clinical Trials
8
Critical Decisions
• Which treatments should be offered first?
• How long should we wait for these treatments to work?
• How long should we wait before offering a transition to a maintenance stage?
• Which treatments should be offered next?
9
Critical Decisions
• All of these questions relate to the formulation of a policy.
• Actions include medications, behavioral therapies, delivery mechanisms, monitoring
• Observations include biological measures, family history, severity, side effects, functionality, symptoms
• Rewards include functionality, side effects, symptoms
10
Outline
– Clinical Decision Support Systems– Critical Decisions– Types of Data– Challenges
• Incomplete mechanistic models
• Measures of Confidence
– Clinical Trials
11
Types of Data
• Large Observational Data Sets– Actions are not manipulated by scientist
• Clinical Trial Data– Actions are manipulated by scientist
• Bench research on cells/animals/humans
12
Clinical Trial Data Sets
• Experimental trial data collected for research purposes– Scientists decide proactively which data to collect and
how to collect this data
– Use scientific knowledge to enhance the quality of the proxies for observation, reward
– Actions are manipulated (randomized) by scientist
– Short Horizon (less than 5)
– Hundreds of subjects.
13
Observational Data Sets
• Observational data collected for research purposes– Scientists decide proactively which data to
collect and how to collect this data– Use scientific knowledge to enhance the quality
of the proxies for observation, action, reward– Actions are not manipulated by scientist– Moderate Horizon– Hundreds to thousands of subjects.
14
Observational Data Sets
• Clinical databases or registries– (an example in the US would be the VA registries)– Data was not collected for research purposes– Use gross proxies to define observation, action,
reward– Moderate to Long Horizon– Thousands to Millions of subjects
15
Outline
– Clinical Decision Support Systems– Critical Decisions– Types of Data– Challenges
• Incomplete mechanistic models
• Measures of Confidence
– Clinical Trials
16
Availability of Mechanistic Models
• In many areas of RL, scientists can use mechanistic theory, e.g., physical laws, to model or simulate the interrelationships between observations and how the actions might impact the observations.
• Scientists know many (the most important) of the causes of the observations and know a model for how the observations relate to one another.
17
Incomplete Mechanistic Models in Medical Sciences
• Scientists who want to use data on individuals to construct policies must confront the fact that non-causal “associations” occur due to the unknown causes of the observations.
18
Unknown Unknown Causes Causes
Observations Action Observations Action RewardTime 1 Time 2
Time 2 Time 3
Conceptual Structure in the Medical Sciences (observational data)
19
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
Maturity/Unknown DecisionCauses to join "Adult"
Society
+
+
Binge Drinking Treatment Binge Drinking Counseling Functionality Time 1 Time 2
Time 2 Time 3
20
Unknown, Unobserved Causes (Incomplete Mechanistic Models)
• Problem: Non-causal associations between treatment (here counseling) and rewards are likely.
• Solutions:– Collect clinical trial data in which treatments are
randomized. This breaks the non-causal associations yet permits causal associations.
– Prior to applying methods to observational data proactively brainstorm with domain experts to ascertain and measure the main determinants of treatment selection. Then take advantage of causal inference methods designed to minimize assumptions on the data
21
Maturity/
Unknown DecisionCauses to join "Adult"
Society
"+"
Observations Treatment Binge Drinking Counseling FunctionalityTime 1 Time 2
Time 2 Time 3
Unknown, Unobserved Causes
(Incomplete Mechanistic Models)
22
Maturity/
Unknown DecisionCauses to join "Adult"
Society
+ -
Binge Drinking Counseling on - Binge Drinking Sanctions FunctionalityYes Health Yes/No + counseling
Consequences Time 2 Yes/No Time 3 Yes/No
Unknown, Unobserved Causes (Incomplete Mechanistic Models)
23
• The problem: Even when treatments are randomized, non-causal associations occur in the data.
• Solutions:– Recognize that parts of the Q-function/transition probabilities can
not be informed by domain expertise as these parts reflect non-causal associations
– Or use methods for constructing policies that “average” over the non-causal associations between action and reward.
• I think that the importance of this second causal inference problem depends on the kind of data and how you use it.
Unknown, Unobserved Causes (Incomplete Mechanistic Models)
24
Measures of Confidence
– Measures of confidence are essential• Noisy data
• Need to know when any one of a subset of actions will yield the best rewards –that is, when there is no or little evidence otherwise.
• It is important to minimize the number of observations that must be collected in the clinical setting
25
Measures of Confidence
• We would like measures of confidence for the following:– To compare the value of two estimated policies
(both estimated using the training data).– To assess if there is sufficient evidence that a
particular observation (e.g. output of a biological test) should be part of the policy.
– To assess if there is sufficient evidence that a subset of the actions lead to better rewards for a given observation than the remaining actions.
26
Measures of Confidence
• I must both learn the policy and provide an evaluation of the policy using one data set.
• The data set is small
27
Measures of Confidence
• Traditional methods for constructing measures of conference require differentiability (if frequentist properties are desired).
• Q-functions are constructed via non-differentiable operations (e.g. maximization).
• The value of a policy is a non-differentiable function of the policy.
28
Outline
– Clinical Decision Support Systems– Critical Decisions– Types of Data– Challenges
• Causal ::: Unknown, unobserved causes
• Measures of Confidence
– Clinical Trials
29
Clinical Trials• Data from the --short horizon– clinical trials make
excellent test beds for combinations of supervised/unsupervised and reinforcement learning methods.– Developing methods for variable selection in decision
making (in addition to variable selection for prediction)
– Model selection when goal is learning good policies.
– Confidence intervals for the difference in value between two policies.
– Feature Construction
30
ExTENd
• Ongoing study at U. Pennsylvania (D. Oslin)
• Goal is to learn how best to help alcohol dependent individuals reduce alcohol consumption.
31
Oslin ExTENd
Late Trigger forNonresponse
8 wks Response
TDM + Naltrexone
CBIRandom
assignment:
CBI +Naltrexone
Nonresponse
Early Trigger for Nonresponse
Randomassignment:
Randomassignment:
Randomassignment:
Naltrexone
8 wks Response
Randomassignment:
CBI +Naltrexone
CBI
TDM + Naltrexone
Naltrexone
Nonresponse
32
Adaptive Treatment for ADHD
• Ongoing study at the State U. of NY at Buffalo (B. Pelham)
• Goal is to learn how best to help children with ADHD improve functioning at home and school.
33
ADHD Study
B. Begin low dosemedication
8 weeks
Assess-Adequate response?
B1. Continue, reassess monthly; randomize if deteriorate
B2. Increase dose of medication with monthly changes
as neededRandom
assignment:B3. Add behavioral
treatment; medication dose remains stable but intensity
of bemod may increase with adaptive modifications
based on impairment
No
A. Begin low-intensity behavior modification
8 weeks
Assess-Adequate response?
A1. Continue, reassess monthly;randomize if deteriorate
A2. Add medication;bemod remains stable butmedication dose may vary
Randomassignment:
A3. Increase intensity of bemod with adaptive modifi-
cations based on impairment
Yes
No
Randomassignment:
34
Studies under review
• H. Jones study of drug-addicted pregnant women (goal is to reduce cocaine/heroin use during pregnancy and thereby improve neonatal outcomes)
• J. Sacks study of parolees with substance abuse disorders (goal is reduce recidivism and substance use)
35
Jones’ Study for Drug-Addicted Pregnant Women
rRBT
2 wks Response
rRBT
tRBTRandom
assignment:
rRBT
Nonresponse
tRBT
Randomassignment:
Randomassignment:
Randomassignment:
aRBT
2 wks Response
Randomassignment:
eRBT
tRBT
tRBT
rRBT
Nonresponse
36
Sack’s Study of Adaptive Transitional Case Management
Standard Services
Standard TCM
Randomassignment:
Randomassignment:
4 wks Response
Standard TCM
Augmented TCM
Standard TCM
Nonresponse
37
Discussion
• Methods for online updating the policy as data accumulates.
• Methods for producing composite rewards.– High quality elicitation of functionality
• Human-Computer interface
• Improving tactics
38
This seminar can be found at:
http://www.stat.lsa.umich.edu/~samurphy/
seminars/UAlberta07.ppt
Email me with questions or if you would like a copy:
39
Unknown, Unobserved Causes
• Problem: We recruit students via flyers posted in dormitories. Associations between observations and rewards are highly likely to be (due to the unknown causes) non-representative.
• Solution: Sample a representative group of college students.
40
STAR*D
• This trial is over and the data is being analyzed (PI: J. Rush).
• One goal of the trial is construct good treatment sequences for patients suffering from treatment resistant depression.
www.star-d.org
41