Schneider Institute for Health Policy,The Heller School for Social Policy and Management,Brandeis University

Introduction To Evaluating Process And Outcomes In Research Design

Sarita BhalotraDonald Shepard

August 18, 2004

Human Services Evaluation

•Johnson’s “Great Society” proliferation of experimental and pragmatic H.S. programs

•Results in need for evaluation

1960s

Human Services Evaluation

•Experimental approach of programs inadequate to understand human services programs

•Traditional approach of evaluation over-reliant on bio-medical model

1970s

Human Services Evaluation1980s

•Post modernist approach emphasizes practical problem-solving orientation to program evaluation

•“Outcomes” are personal or organizational changes or benefits as a result of activity, intervention, or service

Lifestyle Modification Program Demonstration: Evaluation

1. Outcomes Evaluationa. Clinicalb. Cost/Utilizationc. Cost-effectiveness

2. a. MBMI and LA/PMRIb. Program Sitesc. Del Marvad. CMS

PopulationAge > 65 yearsClinically high-risk for progression of cardiovascular disease

ResourcesAdditional Medicare Funds

Additional Demonstration Site funds

In-Kind Contributions

Additional ServicesNutrition ExerciseStress ManagementGroup Support

Health StatusReduced Cardiovascular EventsImproved Q.O.L.Reversal of Cardiovascular Disease(Cost-saving/cost-neutral)

Appropriate

Efficie

nt

Effective

PROGRAM THEORY FOR

MEDICARE LIFESTYLE MODIFICATION DEMONSTRATION

Research QuestionWhat organizational characteristics explain the number of Medicare beneficiaries who participate in the Medicare LMPD?

Specific Aims1. Determine the critical stakeholders and their roles in the Medicare LMPD2. Establish the critical steps and their sequencing needed for the successful

participation of Medicare beneficiaries in the Medicare LMPD.3. Analyze the characteristics of, and interactions between and among stakeholders

that impacts the critical steps and the extent to which these affect the participation of Medicare beneficiaries in the Medicare LMPD.

4. Develop a program model of the structure, processes, and intermediate outcomes of a successful organization in terms of Medicare beneficiary participation.

ContextLMPD is testing two lifestyle interventions nationwide:1. Mind/Body Medical Institute’s Cardiac Wellness Program2. Dr. Dean Ornish Program for Reversing Heart Disease

Hypotheses1. Successful sites will have identified and cooperated with critical stakeholders,

especially referral sources, in the planning phase. 2. Successful sites will have designed an effective senior management team, and

selected and train motivated staff.3. Successful sites will have developed and implemented participant-focused

marketing, recruitment, enrollment and retention techniques.4. Successful sites will have developed and implemented procedures for tracking

and maintaining relationships with stakeholders

LMPD Evaluation

Institutionalization

Cost

QualityDemonstration

SuccessParameters

Enrollment

Process Evaluation: Program Theory for Sites

MarketingProvider Relations

HQ AssistanceInformation

Systems

ParticipationAdequateSustained

InstitutionalizationCulturalFinancial

FacilitiesFunding/

InvestmentRevenues from

Payers

Human ResourcesLeadership

ClinicalManagerial

Cumulative Enrollment in Lifestyle demonstration by Program

0

89

00 0 0

218

307

00

50

100

150

200

250

300

350

Jul-0

0

Oct

-00

Jan-

01

Apr

-01

Jul-0

1

Oct

-01

Jan-

02

Apr

-02

Jul-0

2

Oct

-02

Jan-

03

Apr

-03

Jul-0

3

Oct

-03

Jan-

04Time

Cum

ulat

ive

Num

ber o

f Enr

olle

es

Blue Red Total

Medicare Lifestyle Modification DemonstrationEnrollment Review and Participation Status

Status Program A Program B Overall Totals

Months of Participation 5/00-3/31/04 8/01–3/31/04

Total Cases Reviewed 140 259 399Total Cases Approved 96 224 320Total Cases Not Approved/Pending 44 35 79Total Overall Enrolled 89 214 303Total Currently Enrolled in Year 1 51 90 141Total Currently Enrolled in Year 2 (Monitoring)

6 49 55

Total Currently Enrolled in Year 3 (Monitoring)

3 10 13

Total Completed Demonstration 4 0 4Total Disenrolled 26 64 90

Enrollment Barriers For Patients Contacted by Nurse Recruiter, First Ten Months (excluding those in Process), n = 1, 387

Note: Information was updated on slightly different dates. Minordiscrepancies in totals reflect differences in dates of data collection.

9%10%

17%9%

6%5%

5%4%

12%8%

Did not meet criteria

Age < 65

Cardiac event > 1 year

Met clinical exclusion criteria

Lives > 90 minutes away

Other barriers

Transportation issues

Too much time

Eating Plan

Money

Physical Limitations

Not interested

Other Barriers

Did Not Meet Criteria

Results of Nurse Recruiter, First 10.5 Months (n = 1706)

Program A, 4 sites

ContinuedInterest andSubmittedTo CMS forMedicare

Part B Eligibility

0%Initially

InterestedAnd InitiatedApplication

StillInterested

And SubmittedTo DelmarvaFor ClinicalEligibility

RemainInterested

AndObtaining

ClinicalInformation

1%

2%

3%

4%

5%

6%

7%

8%

9%

10% 9.1%

3.8%

1.7% 1.5%0.9%

ApprovedBy DelmarvaAnd Ready

to JoinNext Cohort

Nurse Recruiter Outcomes, First 10.5 Months (n=1706

0.9%

44.1%

37.2%

17.8%

Did not meet criteria

Other barriers

Indeterminate or inprocessEnrolled thru nurserecruiter

Selected Stakeholder Groups and Key Steps Leading to Participation(Specific Aims 1 and 2)

MedicarePatients

Program Staff

Referring Providers

HostFacility

CardiacRehab

Program

MarketingRecruitmentEnrollmentRetention

Barriers to Enrollment Across ProgramsParticipants 1. Out of pocket cost 2. Time of day (e.g., ends after dark) 3. Time demands 4. Rigorous diet; too time-consuming in preparation5. Program is too stringent to follow 6. Lifestyle changes are not supported by environment7. Extensive documentation needed for enrollment 8. Time consuming screening required

Program staff 1. General mailing is expensive 2. Limited time available for some targeted recruitment despite its

value (e.g. inpatient nurses) 3. No time for other targeted recruitment (e.g., via cath lab) 4. Documentation is labor intensive5. No benefits to program staff apparent from documentation

Barriers to Enrollment Across ProgramsHost Facility1. Not convinced of clinical efficacy2. Financial loss leader3. Takes away from other programs (e.g. cardiac rehab)4. Impinges on turf of other physicians5. Highly politicized in some cases (e.g. “champion’s cause”)

Barriers to Enrollment Across ProgramsCommunity physicians/ Referral sources1. Not convinced of efficacy of programs2. Not sure patients will comply even if efficacious3. Easier to prescribe drugs, other medical or surgical intervention4. Concerned about loss of revenue/patients5. Do not want to refer patients who they think will fail6. Frustrated when referred patients face long delays or are not found to

be eligible7. Frustrated by amount of paperwork required to document a patient’s

eligibility8. Concerned by the time of the doctors and nurses to explain the

program to a patient9. Information available is incomplete and outdated

Key Activities, Interactions, and Characteristics of Successful Organizations (Specific Aims 3 and 4) 1. Marketing is multi-modal, but targeted especially to referring

physicians2. Marketing anticipates physician resistance, and is convincing of the

clinical efficacy, feasibility, and lack of threat to current practice3. Institutional supports financially, culturally, and clinically4. Eligibility criteria do not pose either a substantive or logistic barrier5. Reimbursement is by all payors or majority6. Program is a substitution for or complementary with traditional

cardiac rehab 7. Program Leadership is conducive to trust and credibility 8. HQ Leadership is conducive to trust and credibility

Examples From Site VisitsCharacteristic Program B

SITE 1Program B

SITE 21. Multimodal marketing 1 4

2. Resistance Management 1 33. Institutional Support 1 54. Eligibility Barrier 2 25. Reimbursement 4 46. Cardiac Rehab Synergy 1 57. Program Leadership 1 38. HQ Leadership 3 3

TOTAL 14(site closed since visit)

29(site continues to enroll to capacity)

On a scale of 1 to 5, from ineffective to effective.

Major Issues

1. The target community opposes the change being advocated2. The adoption costs exceed tangible benefits3. Effective structural and motivational systems designed to engage and reward

are needed at every level4. Consumer needs, preferences and lifestyles are paramount

Conclusions

1. Enrollment in LMPD, as in other preventive programs, is a challenge 2. Environmental scans help identify barriers and ways to overcome them 3. Interventions, such as the nurse recruiter, are achieving partial success 4. Lifestyle Modification enrollment remains below program expectations