Program EvaluationProgram Evaluation

Planning Programs for Adult LearnersChapter 11: Formulating Evaluation PlansCafarella (2002)

Definitions in Program EvaluationDefinitions in Program Evaluation

Program Evaluation: a process used to determine whether the design and delivery of a program were effective and whether the proposed outcomes were met.

Formative Evaluation: Evaluation done to improve or change a program while it is in progress.

Summative Evaluation: Evaluation focuses on the results or outcomes of a program.

Formative vs. SummativeFormative vs. Summative

Formative Summative

Were changes occurring in knowledge and skill levels as they go through the program?

What context issues came up that might have changed the nature of the program?

Were changes evident from pre / post data?

How applicable would this program data be for other settings / contexts?

Goals of Program EvaluationGoals of Program Evaluation

Judging the value or worth of an education/ training program (cost/benefit analysis).

Demonstrate that program outcomes are ties to what happened in the program.

Develop clear criteria to make your judgments that can be changed as the program evolves.

Be aware of the audience for which the program data will be shared and the population from which it will collected. Each will have their own agendas.

Program met the original objective set forth by administration and/or participants.

Program did not have any unintended consequencesLearners are able to transfer the information obtained

to another setting.Usually specific to the program and not as interested

in generalizability.

Elements of a Systematic EvaluationElements of a Systematic Evaluation

Secure support for the evaluation from the stakeholders.

Identify who will plan and oversee the evaluation.

Define the purpose of the evaluation and the use of the results.

Specify what is judged and formulate evaluation questions.

Determine who supplies needed evidence and/or currently available data.

Delineate the evaluation approach.

Choose data collection and use plan.

Indicate the analysis procedure.

Stipulate how judgments are made.

Determine timeline, budge, needed resources.

Monitor and complete the evaluation and make judgments.

When to Collect Evaluation DataWhen to Collect Evaluation Data

Collecting DataCollecting Data

How to Collect DataHow to Collect Data

ObservationsInterviewsWritten QuestionnairesTestsSurveysProduct ReviewsPerformance ReviewsReview of RecordsPortfoliosCost-benefit AnalysisFocus GroupsSelf-Assessment

Levels of Evaluation ApproachLevels of Evaluation Approach

Evaluate◦Participant reactions◦Participant learning◦Behavior change or use new knowledge/skills◦Results or outcomes

Focus◦Primarily on participant reactions and changes

Accountability Planner ApproachAccountability Planner Approach

Evaluate◦Skills/ knowledge/ attitudes◦Achievement of broad objectives◦Learning, tasks, and materials.◦Anticipated changes◦Evidence of change

Focus◦Data collected at each level of program.◦What should you know or believe at the end?◦What were the techniques to reinforce the

objectives?

Situated Evaluation FrameworkSituated Evaluation Framework

Evaluate◦Review of program◦Reflection on criteria (who determines and how)◦Product Review ◦Performance Review

Focus◦Programming (what and how)◦Valuing (who decides what is valuable and how)◦Knowledge Construction (what counts as evidence)◦Utilization of Evaluation Findings (for what ends and

by whom)

Systems EvaluationsSystems Evaluations

Evaluate◦Written Questionnaires◦Cost-benefit Analysis

Focus◦Educational unit function◦Efficiency of use of resources

Case Study MethodCase Study Method

Evaluate◦Observations◦Interviews◦Review of Records◦Qualitative Data

Focus◦What does the program look like from the different

individuals involved Participants Staff Sponsors Stakeholders

◦How was it implemented and received

Quasi-Legal EvaluationQuasi-Legal Evaluation

Evaluation◦Interviews◦Product Reviews◦Tests◦Cost-Benefit Analyses◦Review of Records

Focus◦Quality determined through adversarial

hearings◦Panel of judges determines effectiveness based

on presented evidence

Professional or Expert ReviewProfessional or Expert Review

Evaluate◦Interviews◦Review of Records

Focus◦Panel of experts evaluate based on set of

standards◦Often done for large education and training

programs

EthnographyEthnography

Evaluate◦Qualitative Focus◦Interviews, Observations, Story-telling◦Building of life histories and memoirs

Focus◦Enter and close, prolonged interaction with

people in their everyday lives◦Purpose: placing actions and changes within a

larger context