A Primer on Outcomes Research

Vidya Sharma MBBS, MPH.

A Primer on Outcomes Research

The Children’s Mercy Hospital

Objectives

• Understand the principles of outcomes research.

• Define outcome measures and variables

• Understand the advantages/limitations of the outcomes research.

The Promise of Health ServicesResearch

“Health services research can save more

lives in the next decade then can bench

science, work on the human genome,

stem cell therapy and cancer research..”

Atul Gawande MD

Health Services Research inPediatrics

• A core body of pediatrics relevant

health services literature already exists

• Outcomes research

• Patient satisfaction research

• Expert practice

• Utilization research

• Health disparities

Outcomes Research

• What is outcomes research?– Outcomes research studies the end results of

medical care– End results include effects that people experience

and care about, such as change in the ability to function

– For individuals with chronic conditions end results include quality of life as well as mortality

– The effects of the health care process on the health of patients and populations

Outcomes Research

• What works?

• For which patients?

• At what cost?

• From whose perspective?

Case

• 5 year old fell from a grocery cart

• Dazed 60 seconds

• Lethargic 20 minutes

• Exam –no focal findings

• CT frontal fracture no i/c bleed

• Observed 6 hours and d/c with follow up with PCP

Clinical Research

• Duration of altered mental status

• Vomiting

• Cost of care

• Rates of return to ED

• Disposition: discharge, admit, death

Outcomes Research

• Clinical effectiveness: evidenced based treatment• Quality of care: adherence to clinical guidelines• Quality of life: return to normal activities• Patient satisfaction: satisfaction with discharge

instructions, seen in a timely manner• Cost effectiveness: comparisons of admission vs

observation, CT or not• Organization of care: pre-hospital, referral

Domains of Outcomes Research

• Clinical effectiveness• Quality of care• Quality of life• Patient satisfaction• Organization of care

• Death• Disease• Disability/Discomfort• Dissatisfaction• Dollars

Types of Outcomes Research

– Evaluating effectiveness of medical/ surgical procedures

– Measuring impact of insurance/ reimbursement policies on outcomes of care

– Development and use of tools to measure health status

– Analyze the best way to disseminate results to encourage physician/consumer behavior change

What makes Outcomes Research different?

• The question it asks

• The settings it studies

• The method it uses

• The health status it measures

What makes Outcomes Research different?

• The questions it asks– all aspects of the health care delivery system– clinical encounter– organization of the health care system– financing of the health care system– regulation of the health care system

What makes Outcomes Research different?

• The question it asks

• The settings it studies

• The method it uses

• The health status it measures

What makes Outcomes Research different?

• The setting it studies

– contrast with randomized controlled studies• efficacy studies

– research in real-life settings• effectiveness studies

Definitions

• Efficacy– Probability of benefit from an intervention

under standardized or ideal conclusions– Randomized control trials

• Effectiveness Studies– Results achieved in actual practice with

different patients and providers– Research in real life settings

What makes Outcomes Research different?

• The question it asks

• The settings it studies

• The method it uses

• The health status it measures

What makes Outcomes Research different?

• The methods it uses– use existing computerized databases– patient questionnaires– meta-analysis– complement standard strategies

Types of Projects Using Databases

• Health Services Research– Data is used to examine access, use, costs,

delivery, and organization of health services to better understand the structure and effects of health services for populations

• Intervention Projects– A therapy or practice is compared with similar

populations which do not use therapy

Types of Studies

– Adherence to clinical practice guidelines– Costs of Care– Resource utilization of special groups of

patients– Use of specific therapies– Pharmaceutical use

Observational Studies

• Cohort study– Patients with exposure X differ from patients

without exposure X on an outcome

• Case Control Study– Do cases differ from controls on some exposure

• Historical Controls Study– Outcome of interest compared with historical

records

Contrasting RCTs with Observational studies

RCT• error on side of not finding

an effect • generalizability limited to

patient group studied

• populations are equivalent

• small differences can be detected

• small numbers of patients

Observational• error on side of finding an

effect

• ordinary medical practice

• adjust for differences by multivariate analysis

• large differences can be detected

• large numbers of patients

Observational Studies

• Association is not always causation

• Causation is implied when:– Strength (large risk, rules out other factors)

– Consistency (replicated by different researchers and in different conditions)

– Specificity (exposure related to a specific disease than a wide range)

– Temporality (exposure preceded disease)

Hill B. The Environment and Disease, Association or Causation? 1995.Proc R Soc Med;58:295.

Observational Studies

• Causation is implied when:– Biological Gradient (dose related response)– Plausibility (credible scientific explanation)– Coherence (association consistent with natural history of the

disease)– Experimental evidence (physical intervention shows results

associated with the association)– Analogy (similar result that we can draw a relationship to)

http://www.childrens-mercy.org/stats/ask/causation.asp

Health Services Research

Goodman DM, Mendez E, Throop C, Ogata ES: Adult survivors of pediatric illness: the impact on pediatric hospitals. Pediatrics 2002;110:583-589.

Adult Survivors

• Observational study using PHIS, covering the years 1994-1999

• Described the demographics of adult in-patients in pediatric facilities

• Used census data to estimate national impact

• Used HCFA data to estimate financial impact across continuum of care

Adult Survivors

6.198.497.48Average LOS

86,18817853863Number of patients

103,73321275051Number of discharge episodes

All Ages21-64 Years Old

18-64 Years Old

Adult Survivors

3.1%6.3%Adult charges/ total charges

2.1%4.9%Adult disch/ total disch

35%/0.8%27%/13.6%32%/7.4%Medicaid/

Medicare

$20,444$31,114$26,640Avg charge/

discharge

All Ages21-64 Years Old

18-64 Years Old

Adult Survivors

11.09.9Congenital heart disease

5.67.6Malignancy

34.521.8Cystic fibrosis

10.710.2Mental retardation/cerebral palsy

21-64 years old

(% of pts)

18-64 years old

(% of pts)

Diagnostic Group

Total Number of Patients 1994-1999

0

500

1000

1500

2000

2500

1994 1995 1996 1997 1998 1999Year

18+_#pts21+_#pts

Adult Survivors

Intervention StudyTodd J, Bertoch D, Dolan S: Use of a large

national database for comparative evaluation of the effect of a bronchiolitis/viral pneumonia clinical care guideline on patient outcome and resource utilization. Arch Pediatr Adolesc Med 2002;156:1086-1090

Bronchiolitis/Viral pneumonia

• Clinical guideline developed and implemented

• Effect of guideline determined by internal comparison across time and external comparison using PHIS

Bronchiolitis/Viral pneumonia

0

10

20

30

40

50

60

70

80

1994(pre) 1995 1996 1997YEAR

% U

TIL

IZA

TIO

N

CPT RSV TEST ICU Admission RIBAVIRIN

O.R =0.67 (0.60, 0.75)

O.R. = 0.43 (0.28, 0.68)

O.R. = 0.77 (0.66, 0.90)

O.R. = 0.72 (0.64, 0.80)

Bronchiolitis/Viral pneumonia

Reduction of Low Severity Cases Admitted to TCH

0

10

20

30

40

50

60

70

1994 1995 1996 1997YEAR

% o

f C

AS

ES

Severity 1 Severity 2 Severity 3 Severity 4

P = 0.003

Bronchiolitis/Viral pneumonia

0

10

20

30

40

50

60

70

80

1994(pre) 1995 1996 1997

YEAR

% U

TIL

IZA

TIO

N

$0

$50,000

$100,000

$150,000

$200,000

$250,000

Est

ima

ted

Ch

arg

e

Re

du

ctio

n

Estimated Charge Reduction CPTRSV TEST ICU AdmissionRIBAVIRIN

Care for Cystic Fibrosis

Best or Worst Patient Outcomes?

• Best and worst Cystic Fibrosis centers

offered the virtually the same

treatments to patients!!

• Striking differences in the way that care

was delivered within centers

Results

• The best Cystic Fibrosis Center had leadership from someone who:

• Aggressively pursued consistency of care

• Individualized his approach to each patient

• Engaged patients in their own care process

Potential Pitfalls

• Original goals for which the data was collected may not coincide with subsequent use

• Dataset may not have been designed or maintained to maximize consistency or data quality

Potential Pitfalls

• More data collected than needed

• Infrequently used data elements may be unreliably coded

• Codes may change over time

• Geographic variation in coding

Issues with Databases

• Required data elements– Are they present?– Are they properly coded?– Principal diagnosis only– All diagnoses

Issues with Databases

• Sampling – what population?– which hospitals?– what data?

• Record Structure– Discharge episode– Clinical data

Approach to using secondary Data

• Clearly define your question:– P population– I intervention– C comparison groups– O outcome

Planning a Database Project

• It always takes longer than you think• Would you read it if someone else wrote it?• Being better is more important than being the first• Will the results be worthwhile no matter what you

find?• The probability of success is inversely related to the

number of collaborators.

Goldman l, et al. J of Gen Int Med. 1986;1 (Suppl):S25-S30.

What makes Outcomes Research different?

• The question it asks

• The settings it studies

• The method it uses

• The health status it measures

What makes Outcomes Research different?

• The health staus it measures– Traditional: more proximal outcomes

• laboratory test results

• complication rates i.e. infections

• death

– Outcomes: measures distal outcomes• functional status

• patient well-being / quality of life

• satisfaction with care

How is health status measured?

• Questionnaires that measure aspects of health– Functional Status– Well- being / quality of life– Satisfaction with Care

How is health status measured?

• Functional status– physical functioning i.e. walking,etc– role functioning i.e. going to school etc– social functioning i.e. playing with friends

etc

How is health status measured?

• Well-Being / quality of life– mental health i.e. general mood– health perceptions i.e. person’s view of

health– quality of life– pain i.e. the extent of pain experienced

How is health status measured?

• Satisfaction with care– access– convenience– information received– financial coverage– technical quality

C h ar ac te r is t ic s o f th ein d iv id u a l

B io lo gic a l a n dp h y sio lo gic a l

v a r ia ble sS y m p to n s ta tu s

F u n c tio n a ls ta tu s

G en er a l h ea lthp er c ep tio n s

O v er a ll q u a lityo f lif e

C h aract eris t ics o ft h e en v iro n m en t

N o n m ed ic a lf ac to r s

Ps y ch o lo g ica lS u ppo rt s

S o cia l a n d Ps y ch o lo g ica lS u ppo rt s

V a lu e s a n d Pre fe re n ce sS y m pto m A m plif ica t io n

Conceptual Model of Health Related Quality of Life (HRQL)

Adapted from Wilson IB 1995

Outcomes Research is relevant across the disease continuum

Prevention Detection Diagnosis Treatment Long Term

• Obesity diagnosis and management• Risk AssessmentCommunication/Counseling• Prevention

• ScreeningRecommendationsandPractices

• RiskAssessment(genetics)• TimelyDiagnosisand Referral

• End-of-LifeCare• Use ofEvidence-Based Therapies• PainManagement

• Follow-up Long term

Cross Cutting Topics• Access to Care• Health Disparities• Economics of Care• Quality of Care

What studies are included in Outcomes Research?

• Studies of variations in medical practice patterns

• Studies of effectiveness of various treatments

• Appropriateness studies to determine circumstances when a procedure should/should not be performed

• development of tools to identify patient preferences

• measuring changes in health status

• measuring patient satisfaction with the health care process

Measure Example

Mortality Infant death rate

Physiologic measures Blood pressure

Clinical events Stroke

Symptoms Difficulty breathing

Functional measures SF-36, a 36-item health survey

Patients' experiences with care

Consumer Assessment of Health Plans Survey

Examples of Health Outcomes

Challenges to Conducting Outcomes Research

• Acceptance of outcomes research by clinicians

• Unique aspects of child health outcomes research

Unique Aspects of Outcomes Research in Children

• Development

• Dependency

• Different epidemiology

• Demographic patterns

Unique Aspects of Outcomes Research in Children

• Development

return to normal activities construct requires

unique approaches for infants, preschoolers and teens

Unique Aspects of Outcomes Research in Children

• Dependency

– parent-child dyad complicates– health education, disease management and

access to health care

Unique Aspects of Outcomes Research in Children

• Different epidemiology

– <5% of children affected by heart disease, diabetes and arthritis

– these are the conditions most frequently addressed in adult outcomes research

Unique Aspects of Outcomes Research in Children

• Demographic patterns

– 1 in 5 children live in poverty– 15% are uninsured– 1 in 4 families headed by a single parent

Who uses Outcomes Research?

• Clinicians

• Payers

• Administrators

• Policy makers

• Patients

Where does the funding come from?

• Federal Grants – Agency for Health Care Policy and Research– NIH (Effectiveness studies)– Maternal Bureau of Child Health

• Private Foundations – Robert Wood Johnson– Pew Charitable Trusts– Kaiser Foundation

Incorporating Results into Practice

• Publications and conferences• Educational interactive videos• Continuous Quality Improvement/Management

tools• Clinical Practice Guidelines• Clinical Pathways• Disease Management tools

Summary

• Field of Outcomes Research broad and complex

• Studies all aspects of the health care process• Requires collaboration among physicians,

nurses, economists, sociologists, biostatisticians, epidemiologists, political scientists

• Rewards are great!