Program evaluation action case study

Group 1: Cheryl Tang, Divya Balasubramanian, Hsu Yee Wint, Lai Yi Feng, Valerie Koh

CO5222 Program Evaluation Assignment #3

ACTION Case Study Process Evaluation Plan

Outline

1. Purpose of Evaluationa. Engaging stakeholdersb. Roles and responsibilities of evaluation team members

2. Description of Program Aspects to be Evaluateda. Logic modelb. Key process objectives

3. Process Evaluation Design4. Data Collection and Analysis5. Recommendations and Dissemination Plan

Framework for Program Evaluation in Public Health

Engaging Stakeholders

Stakeholder Name Interest or Perspective

MOH ● Interventions leading to improved health outcomes● “Cost - effectiveness” of the interventions

AIC – Program Owner ● Capabilities of ACTION teams ● Effective partnership with step-down care facilities● Development and sustainability of the program and

resources involved

CMB & Hospital Management

● Burden of the workload on involved staff ● Reduction in hospital’s readmission rates / medical

errors upon discharge

Engaging Stakeholders

Stakeholder Name Interest or Perspective

Hospital Care Transition teams (clinicians, nurses and

case managers)

● Accuracy of decision making tool ● Adequacy of recruited patients ● Clear referral mechanism● Extent of patient’s needs addressed throughout

the program

Patients & Caregivers ● Clarity of information and communication● Satisfaction level towards care coordinators

Community Providers (GPs,Polyclinics, FMCs, Social

Support Groups)

● Clear referral mechanism in place to enable smooth transition

Members Roles and responsibilitiesPrincipal evaluator (Program Director)

● Overall coordinator, ensure evaluation carried out as planned.

● Data interpretation and final report vetting.● Communicate and Disseminate findings to key stakeholders

Co-evaluators (Program Managers)

● Study design.● Pilot testing of survey questionnaire and topic guides.● Implementation of evaluation plan.● Data collection, analyze and interpret findings● Drafting of report

Statistician (external evaluator)

● Quantitative Analysis

Sociologist/Psychologist (external evaluator)

● Draft FGD & IDI structure ● Perform thematic analysis

Roles and Responsibilities of Evaluation Team Members

Logic Model

Key Process ObjectivesStages of Program Process Objectives

At Admission Increased coverage of target population for enrolment into ACTION

Admission to Discharge Early initiation of discharge planning for patients according to their individual needs

Post-discharge Increased utilisation of post-discharge interventions

● Follow-up telephone calls● Home visits● Referrals to appropriate community service

providers

Evaluation Design

Time frame of evaluation: 4 years over entire implementation of the program.

Evaluation questions will focus on key process objectives and mapped onto the following areas of evaluation:

1. Reach/Utilization: Coverage analysis from hospital/program records2. Fidelity/Dose: Adherence to procedures from program records review3. Satisfaction: Patient/caregivers and staff feedback from surveys, interviews,

focus group discussions.

A combination of qualitative and quantitative methods will be employed for specific evaluation questions.

Timeline for Process Evaluation

Activities Month

Year 1Y2 Y3 Y4

1 2 3 4 5 6 7 8 9 10 11 12

Coverage analysis - Participation by target group

Quarterly reports

Review of program records- Fidelity and dose of services delivered

Quarterly reports

Engagement of participants and relevant stakeholders for satisfaction ratings and feedback on the program

Yearly reports

Evaluation variables

● Dependent variables = outcome variables from evaluation questions● Independent variables = Factors to stratify evaluation results by

○ Clinical disciplines○ Individual care coordinators○ Geographical regions of patients/clients

Data Collection and Data Analysis

Process Evaluation: Reach /UtilizationEvaluation Question

Data Collection

MethodSource(s) of Data Process Indicator(s) and

Standard/Criteria (if applicable) Data Analysis

How many patients were enrolled in the program?

Quantitative Program records Number of participants enrolled per hospital per quarter.

Total number of patients enrolled in ACTION

What proportion of the target population participated in the program?

Quantitative Hospital admissions and program records

Not more than 20% of patients who are assessed as eligible by the care coordinator decline to participate in the program per quarter.

Numerator: # of Patients enrolled in ACTION program in the quarterDenominator: # of patients assessed as eligible for ACTION in the same quarter

Has the program served the intended clients?

Quantitative Program records- Patients’

demographic characteristics

- Past admissions and medical history

- Social needs assessment

At least 80% of participants enrolled per quarter fulfill AIC’s selection criteria.

Numerator: # of patients who satisfy the eligibility criteria of ACTION

Denominator: # of patients enrolled in ACTION

What was the program dropout rate and characteristics of those who dropped out?

Quantitative Program records- Exit records

Not more than 10% of participants drop out prior to being discharged from the program by the care coordinator, at the end of the first quarter of the program.

Numerator: # of patients who discontinue service before planned closure/ exit assessmentDenominator: # of patients enrolled in ACTION

Variable(Patients enrolled in ACTION)

Patients discharged from ACTION Patients dropped out from ACTION

N (%),Mean P-value N(%), Mean P-value

Age

Gender

Ethnicity Chinese Indian Malay

Socio-economic status (Ward Class)

Geographic area

Number of Outpatient visits in previous year

Social Needs Assessment score

Average Length of stay in Hospital

Care Coordinator Incharge

Department Admitted to

Table: Patient Dropout Analysis

Process Evaluation: FidelityEvaluation Question

Data Collection

MethodSource of Data

Process Indicator(s) andStandard/Criteria (if

applicable)Data Analysis

Was there standardized training provided for the CCs?

Quantitative Staff training attendance records

All CCs completed the prescribed trainings on the ACTION program.

Numerator: # of recruited CCs who completed the prescribed trainings

Denominator: # of CCs recruited per year

Did the CCs follow the protocol in screening for high-risk patients?

Quantitative Hospital admissions and program records

At least 80% of patients admitted per quarter are screened for eligibility by the CCs within 24 hours of admission.

Numerator: # of inpatients who were screened within 24 hours of admission

Denominator: # of patients admitted per quarter

Was there a proper needs assessment conducted for identified high-risk patients?

Quantitative Program records At least 80% of patients who are identified as high-risk are referred to a CC for a needs assessment using the InterRai assessment suite per quarter.

Numerator: # of high-risk patients who were referred to a CC for a needs assessment using the InterRai assessment suite

Denominator: # of patients admitted per quarter who are eligible for enrolment based on AIC’s selection criteria

Process Evaluation: DoseEvaluation Question

Data Collection

MethodSource(s) of Data Process Indicator(s) and

Standard/Criteria (if applicable) Data Analysis

What interventions, how much and how often did the participants receive the interventions?

Quantitative Patients’ case notes and program records

All participants have a documented, personalized care plan, prior or upon their discharge from the hospital.

Numerator: # of enrolled patients who have a documented, personalized care plan, prior or upon their discharge from the hospital

Denominator: # of patients enrolled in ACTION per quarter

Program records- Activities log of telephone calls and home visits conducted by the CC

At least 75% of participants receive a follow-up telephone call from their assigned CC within 7 days after discharge.

Numerator: # of enrolled patients who receive a follow-up telephone call from their assigned CC within 7 days after discharge

Denominator: # of enrolled patients discharged per quarter

At least 75% of participants receive a home visit by their assigned CC within 1 month after discharge.

Numerator: # of enrolled patients who receive a home visit from their assigned CC within 1 month after discharge

Denominator: # of enrolled patients discharged per quarter

Process Evaluation: Dose (cont’d)Evaluation Question

Data Collection

Method

Source(s) of Data



What interventions, how much and how often did the participants receive the interventions?

Quantitative Program records- Date of referrals- Date patient contact by community service providers- Description of services provided

At least 75% of participants with referrals to community service providers are contacted by their first service provider within 7 days afterdischarge.

Numerator: # of enrolled patients with referrals to community service providers who are contacted by their first service provider within 7 days after discharge

Denominator: # of enrolled patients discharged per quarter with referrals to community service providers

Patient Data Fields

Hospital, Program records

Hospital records

Program records

Program records

Patients’ case notes,

Program records

Hospital records

Program records

Program records

Program records

Patient identifiers

Date and time of

admission

Date and time

screened by CC

Date of needs

assessment by CC

Care plan documented?

Date of discharge

Date of follow-up telephone

call

Date of home

visit by CC

Referral to community

service providers

SXXXXXX 11.10.15 (09:00)

11.10.15 (14:00)

11.10.15 Present / absent

13.10.15 18.10.15 1.11.15 Contacted by service provider on

15.10.15

Process Evaluation: Quality of ImplementationEvaluation Question

Data Collection




What were the strengths, barriers and challenges in implementation?

Qualitativemethod

Sampling method - Purposive sampling (all the healthcare workers from hospital and care coordinators)Sample size - 2 FGD with 4-8 per each FGD

Instrument - open ended questions

Mode of administration - 2 FGD (1) FGD with healthcare workers who communicate with the care coordinators(2) all care coordinators

Feedback from care-coordinators on the strengths, barriers or challenges in implementation and communication with healthcare workers.

Feedback from healthcare workers on the strengths, barriers or challenges in implementation and communication with care coordinators.

Thematic analysis to find out the strengths, barriers and challenges during implementation.

Process Evaluation: Quality of Implementation

Evaluation Question

Data Collection

MethodSource of Data Process

Indicator Data Analysis

What were participants’ perceptions of the program and care coordinators?

Mixed method(Sequential design)

Qualitative methodSampling method - Quota sampling (who complete the program and who drop the program )Sample size - about 15 participants (until reach the data situation)Instrument - Open-ended questions.Mode of administration - IDIs with patients and their caregivers

Qualitative feedback from patients and their caregivers regarding their experience with the program and care-coordinators

Thematic analysis to find out the patient satisfaction towards program and care coordinators during the program regards 1)Discharge process2)Experience with care coordinator3)Experience with Program

Quota sampling of participants for IDI

participant complete the program

participant drop out from the program

care coordinator 1 3 1




Sample questions for the interview;How do you feel about the program before, during and discharge from the program?How do you feel about the care coordinators?

Process Evaluation: Quality of Implementation

Evaluation Question

Data Collection


Process Indicator(s) and

Standard/Criteria (if applicable)

Data Analysis

What were participants’ perceptions of the program and care coordinators?

Mixed method(Sequential design)

Quantitative methodSampling method - convenience sampling (who complete the program)Sample size - all the participants who enrolled in the program and completed the programInstrument - Patient satisfaction surveyMode of administration - Self administered questionnaire upon discharge from the program

At least 60% of participants have good perception towards care coordinators and program.

Calculate the % of participants who rate the program and care coordinators as good at the end of the first year of the program

5 point likert scale - 1:strongly disagree, 2:disagree, 3:neutral, 4:agree, 5:strongly agreeSample questions:

● CC explain details about information on medication.

● I understand the information provided by CC.

● I have confidence in CC.

Recommendations and Dissemination Plan

Proposing evaluation recommendations● At the end of process evaluation, findings will be mapped into conclusions

○ Connecting evaluation questions to key process objectives in admission, discharge and post-discharge activities

○ Considering strengths, value and meaning of the findings

● From the conclusions, recommendations for future directions can be made○ Developing the next steps for adjustments to ongoing or future programs

● Conclusion and recommendations should be discussed during review meeting among key program staff and stakeholders (program owner, implementation site’s management)

○ Deciding and prioritizing recommendations○ Set target dates for implementation○ Depending on the nature of the adjustments, different stakeholders may be required to act

Linking evaluation questions to decisionsIndependent

variables Purpose Potential Recommendations

Clinical disciplines

Identify difficulties and challenges faced by each department

Review allocation of resources across disciplines

Geographical regions of patients/clients

Evaluate performance by patients/clients’ location post-discharge

Identify reason(s) for poor performance

Specific recommendations for program owner and community partners (e.g. modifications to administration of outreach activities, suggestion to expand geographical coverage by community providers)

Individual care coordinators

Evaluate performance by each care coordinator and correlate quantitative results from coverage analysis with qualitative data

Identify reasons for poor performance

Specific recommendation for individuals or group of individuals (e.g. counselling, refresher training for staff)

Implementation of evaluation recommendationsAspects of findings/recommendations Action by

Reach of the intended target audience ( no. of patients recruited, no. of referrals - physician referral to ACTION and patients’ acceptance)

Program owner and key program staff to enhance buy-in from frontline care providers such as physicians, and patients/caregivers

Utilization of services ( no.of home visits; telephone follow-ups)

Program owner and key program staff for re-allocation or resources and re-prioritization of objectives

Fidelity of services provided ( proportion of cases screened within 24 hours after admission & the proportion of cases screened accordingly to criteria/screening tools)

Key program staff and institution management. May include key opinion leaders of front-line staff it recommendation requires adjustments to existing workflows

Satisfaction - feedbacks collected and barriers identified via survey, focus groups or interviews

Frontline care providers if recommendation is care-specific. Institution/partner management if recommendation is system-specific

Target audience● Purpose of the final report on process-evaluation:

○ To understand, report and discuss learning points from ACTion implementation in a hospital over 4 years.■ Coverage and Utilization■ Fidelity and Dose■ Quality of implementation- Satisfaction and Barriers

● The following groups of people will form the target audience for the report○ Funder-MOH/MOF○ Program owner: AIC○ Stakeholders in implementation site: Management and front-line staff○ Clients/patients and caregivers○ Community providers and partner institutions

Target AudienceTarget audience Interest Purpose

Funder- MOH/MOF Coverage and UtilizationFidelity and DoseQuality of implementationAppropriatenessSatisfactionBarriers

Determine future funding potential, impact to population health or system/resource utilization

Program owner- AIC Design program adjustments, react to identified problems and challenges. Determine program success/failure, potential for expansion or need to scale down

Hospital Management Determine program success/failure, learning points for other hospital programs/initiatives

Hospital Front-line Staff SatisfactionBarriers

Determine gaps and problems, providing feedback for program refinement. Understand changes in care seeking/delivery experience by providers and end-users.Patients and caregivers

Community partners Quality of implementationAppropriatenessSatisfactionBarriers

Determine program success/failure, potential for further collaboration. Assist in workload projection and future resource allocation.

Dissemination methodsTarget

audienceInterest Dissemination methods Frequency of

report

Funder- MOH/MOF

Coverage and doseQuality of implementationAppropriatenessSatisfactionBarriers

Formal detailed report +/- presentation Annual report at the end of every year

Program owner- AIC

Formal detailed report +/- presentation Quarterly reports at the end of every quarter

Hospital Management

Formal detailed report +/- presentation Annual report at the end of every year

Hospital Front-line Staff

SatisfactionBarriers

Presentation of key findings during department meeting, and feedback gathering

Annual report at the end of every year

Patients and caregivers

Key findings and message via roadshows, article in hospital publication, short video to be screened on TVs within hospital campus

Annual report at the end of every year

Community partners

Quality of implementationAppropriatenessSatisfactionBarriers

Summary report, newsletters +/- short presentation Annual report at the end of every year

Implications for stakeholdersTarget audience Purpose Implications

Funder- MOH/MOF Determine future funding potential, impact to population health or system/resource utilization

Adjustment to funding model, structure and amount; Adjustment to KPIs or objectives.

Program owner- AIC Determine program success/failure, potential for expansion or need to scale down

Refinement of program details and ongoing/ future implementation plans, mapping findings into conclusion and recommendations

Hospital Management

Determine program success/failure, learning points for other hospital programs/initiatives

Refinement of program implementation in consultation with program owner

Hospital Front-line Staff

Determine gaps and problems, providing feedback for program refinement. Understand changes in care seeking/delivery experience by providers and end-users.

Better understanding of the program; Assisting in decision on program enrolment

Patients and caregivers

Community partners Determine program success/failure, potential for further collaboration. Assist in workload projection and future resource allocation.

Assisting in decision on continuation in partnership/collaboration, strengthening or scaling back of efforts.

Thank You!

Questions?

Healthcare

Program evaluation action case study