Evaluation Plan

Steven Clauser, PhD Chief, Outcomes Research Branch

Applied Research Program

Division of Cancer Control and Population Sciences

NCCCP Launch June 25, 2007

Presentation Overview

• Evaluation Approach

• Key Evaluation Features • Evaluation Design Components

• Patient Survey

• Evaluation Timetable

NCCCP Evaluation Approach

• Formative not Summative Evaluation – What is the feasibility, sustainability and replicability of

the NCCCP model? – How does strong institutional commitment and executive

leadership support influence results?– Less emphasis on classical intervention or impact

assessment– More emphasis on measurement of improvement from

baseline assessment • Emphasizes multi-method evaluation techniques

– Qualitative assessments of organizational change – Quantitative assessment of trajectory of change

• Pre-post evaluation design– Performance assessment largely limited to intra- not inter-hospital and program comparisons

Key Evaluation Features

• Logic maps to rigorously define pilot site hospital and cancer program interventions

• Participatory research to achieve consensus on evaluation metrics and data collection approaches

• Baseline and follow-up patient surveys to assess improvement in patient program knowledge, program experience, and health-related quality of life

• Formal cost analysis of program implementation and operations • Continued feedback of performance to

NCI staff, program leadership and pilot sites

NCCCP Evaluation Design Components

• Year One – Refine Evaluation Design, Metrics, and Measures of Performance for hospital and program– Phase I – Evaluability Assessment and site visit (2-4 months) – Phase II – Participatory Research to select evaluation hypotheses/metrics – with pilot group input

(4-7 months) – Phase III – Finalize evaluation design and collect baseline data (8-12 months)

• Year Two – Interim site visit, data collection, and first progress review by evaluation contractor

• Year Three – Final site visit, data collection and second progress review

• Year Four – Final data collection and report

Phase I – “Evaluability Assessment”

• Baseline data collection on pilot program components

• “Logic mapping” of program components (including hospital and community) to evaluation questions

• Anchored by evaluation contractor 2-day site visit to each pilot program – including multi-system leadership interviews

• Assessment completed within 4 months of contractor award

Phase II – Selecting Measures and Metrics

• NCI staff prepares initial set of evaluation measures and metrics

• Evaluation contractor refines measurement set and distributes to the sites for comment

• Sites meet with evaluator and provide feedback on the appropriateness, feasibility, and availability of metrics

• Evaluation contractor submits final recommendations to NCI

• NCI makes final decision on measures and metrics within 7 months of award

Phase III – Finalize evaluation plan and collect baseline data

• Evaluation contractor completes detailed evaluation plan for NCCCP pilot program

• NCI approves evaluation plan within 9 months of contractor award

• Sites complete baseline data collection on all key NCCCP evaluation questions by end of year one

• The “intervention” period begins for all hospital and community-based projects– Progress reviews annually– Final assessment at conclusion of year three

Pilot Site Role in Supporting Patient Surveys

• Evaluation contractor develops survey content based on their expertise, review of pilot site surveys, and NCI input – NCI approves final survey instrument

• Pilot sites administer patient survey with evaluation contractor support and training – Contractor develops survey administration protocols

(mailed surveys with telephone follow-up)– Pilot Sites obtain their IRB approvals for patient survey– Contractor trains pilot site survey personnel – Pilot sites administer survey to eligible patients – Pilot sites return completed surveys to evaluation

contractor

Patient Survey Schedule

• Patient survey administered twice:– Baseline survey completed at the end of year one

• Baseline survey report prepared for NCI and pilot sites to illustrate differences in site populations

• Results may prove useful in targeting NCCCP program interventions

– Follow-up survey completed at end of year three • Final report assesses change in patient experience and

patient knowledge/use of pilot patient resources • 300 patients will complete the survey per site

and per survey round – Multi-system pilots will administer additional

surveys for each component site

Patient Survey Measurement Domains

• Outcome Metrics: – Health-related quality of life (physical, mental, and social

domains) – Overall program satisfaction

• Process Metrics – Self-perception of needed resources to assist in

personal management of cancer treatment • e.g., understanding disease, treatments, emotional and

financial assistance services – Knowledge and use of pilot site patient resources

• Including access to clinical trials– Patient Experience with care

• trust, communication, decision-making • access to care, appointments, and waiting time• Experience with multi-disciplinary team practice

Evaluation Timetable – Key Year One Events

Activity* DateProjected Contract Award Date July 20, ‘07

Initial Contractor Site Visits Oct-Nov ’07

Sites meet with Evaluation Contractor Dec’07 or Jan’08

Sites get trained on patient survey** April ‘08

Sites receive final evaluation design April ’08

All pilot baseline measurements final May ‘08

*Timetable depends on contract award date** Site survey training depends on OMB clearance requirements