Upload
ziptrainapi
View
216
Download
0
Embed Size (px)
Citation preview
8/2/2019 11 EDUC 612 Program Evaluation_slides
1/32
PROGRAM EVALUATIONEDUC 612, March 12, 2012
Rohanna Buchanan, Ph.D
8/2/2019 11 EDUC 612 Program Evaluation_slides
2/32
Are we intuitive evaluators?
Can we be objective evaluators?
Evaluation Research Principles
8/2/2019 11 EDUC 612 Program Evaluation_slides
3/32
What do we evaluate?
People/Animals Weather Consumer Products Restaurants Advertiser Messages Political Arguments Programs (social/educational)
8/2/2019 11 EDUC 612 Program Evaluation_slides
4/32
Evaluation Research
Important for guiding decision-making and policydevelopment
Can be used for program improvement or assessingprogram impact (i.e., formative or summative
evaluation)
8/2/2019 11 EDUC 612 Program Evaluation_slides
5/32
Evaluation vs.
Social Science Research
The methodologies involved in evaluation and socialscience research are quite similar
The purpose and use of findings tends to differ: Decision making vs. Knowledge generation Generalizability Value/merit judgments
8/2/2019 11 EDUC 612 Program Evaluation_slides
6/32
Program Evaluation Defined
Program evaluation:A description of program performanceA judgment about program efficacy
Uses social research methods Examines social intervention programs Adaptation of research to the political and
organizational context Informs social action to improve social conditions
8/2/2019 11 EDUC 612 Program Evaluation_slides
7/32
1. Needs Assessment: Is there a need for a program?
2. Program Theory: Is the program logical and well-
designed?
3. Program Monitoring: Is the program being
implemented effectively and with fidelity?
4. Program Impact: Did the program work? Did it effectthe intended population?
5. Program Efficiency: Was the program cost-effective?
Program Evaluation Domains
8/2/2019 11 EDUC 612 Program Evaluation_slides
8/32
1. Needs Assessment
Identifying & Defining a Social Problem Identifying & Defining a Target Population Reconciling Need with Resource Availability in
Program Design
8/2/2019 11 EDUC 612 Program Evaluation_slides
9/32
Assessing Program Need
Needs Assessment: a systematic approach toidentifying social problems, determining their
extent, and accurately defining target populations
A needs assessment can be used to aid in the designof new programs or to review the protocols of
established programs
Helps prevent implementation of inappropriate orunneeded services (challenges assumptions)
8/2/2019 11 EDUC 612 Program Evaluation_slides
10/32
What is a Social Need/Problem?
A social need is not an objective phenomena A social phenomena becomes a social need when
stakeholders become dissatisfied with existing
policy
Social needs/problems tend to be hot buttontopics that are politically charged
Drug use, gambling, prostitutionTeen pregnancy, child literacy, hazingDUI, seat belt/helmet use, kids/pets left in carsSchool uniforms, second-hand smoke, homelessness
8/2/2019 11 EDUC 612 Program Evaluation_slides
11/32
Class Activity:
Identify a population and apotential social or educational
programDescribe the need
8/2/2019 11 EDUC 612 Program Evaluation_slides
12/32
2. Program Theory
The set of assumptions and expectations about howa program is designed to work and the connections
presumed between its various activities, functions,
and outcomes Does the conceptualization and design of the
program reflect valid assumptions about the natureof the target problem and represent a well-
founded and feasible approach to resolving it?
8/2/2019 11 EDUC 612 Program Evaluation_slides
13/32
Program Theory
Assumptions and expectations about programoperations:
Program
Theory
ProcessTheory
(Function)
ImpactTheory
(Causality)
OrganizationalPlan
Service Delivery
8/2/2019 11 EDUC 612 Program Evaluation_slides
14/32
Organizational Plan
Details the programs service objectives and theresources that are necessary to carry out those
objectives
i.e., how program management decides to garner, configure,and deploy resources and organize program activities so that
the intended service delivery system is developed andmaintained
Necessary conditions: what type of service, who is to receive services (target
population), how many and what type of employees,
educational and work experience qualifications, location and
hours of service, etc.
8/2/2019 11 EDUC 612 Program Evaluation_slides
15/32
Organizational Plan (continued)
After School Mathematics Enrichment Example:Program location and timesResources: facilities, equipment, program materialsHuman capital (teachers, support staff): application
process, qualifications, stipend, professional
development/training requirements, number needed
Students: application process; selection criteria
8/2/2019 11 EDUC 612 Program Evaluation_slides
16/32
Services Utilization Plan
The assumptions and expectations about how thetarget population will make initial contact with
the program and be engaged with it through the
completion of the intended services
Assumptions and Expectations:beliefs about how the intended target population is to
become aware of (e.g., information leaflets, public
service announcements, mailings, billboards) and
receive the intended amount of the intervention
8/2/2019 11 EDUC 612 Program Evaluation_slides
17/32
Services Utilization Plan (continued)
After School Mathematics Enrichment Example:Contact: letters to parents, advertisements in schools,
announcements by teachers, counselors, principal
Engagement: arrange transportation, provide snacks,provide incentives & awards for participation and
completion of the program
8/2/2019 11 EDUC 612 Program Evaluation_slides
18/32
Impact Theory
The beliefs, assumptions, and expectationsinherent in a program about the nature of the
change to be brought about by program action
and how the change is to result in the intendedimprovement in social conditions
Causal Theory:the hypothesized cause-effect sequence between
program activities and the programs proximal and
distal outcomes
8/2/2019 11 EDUC 612 Program Evaluation_slides
19/32
Impact Theory (continued)
After School Mathematics Enrichment Example:
Problem: Low mathematics achievement Solution: After-school mathematics program Activities: Focused instruction in computation, problem-
solving, logical reasoning
Proximal Outcomes: Increased self-efficacy, improvedattitudes toward mathematics, improved mathematics
achievement
Distal Outcomes: Increased mathematics course-taking,increased pursuit of math/science degrees in college
8/2/2019 11 EDUC 612 Program Evaluation_slides
20/32
Using Logic Models to
Represent Program Theory
Logic Model:
A logic model is a schematic representation ofprogram theory
The logic model provides a road map of theprogram, highlighting how it is expected to work,
what activities need to come before others, and how
desired outcomes are achieved Logic models facilitate communication with
stakeholders, and serve as an effective
management tool
8/2/2019 11 EDUC 612 Program Evaluation_slides
21/32
Logic Model Example
Problem:
Low Math Achievement
Distal Outcomes:
Increased math course taking
in H;, increased math/science
degrees in college
Proximal Outcomes:
Improved attitudes, self-
efficacy, math achievement
Instructional Focus:
Problem solving, logical
reasoning, computation
Solution:
Afterschool Math Program
8/2/2019 11 EDUC 612 Program Evaluation_slides
22/32
Class Activity:
Describe the program theoryProcess (organizational plan, service
delivery) or causality (impact theory)
8/2/2019 11 EDUC 612 Program Evaluation_slides
23/32
3. Program Monitoring
A form of evaluation to answer questions about program operationsand service delivery (follows logically from program theory)
Is the program being implemented according to the intended design? Is the program staffed and well-organized? Is the program reaching the intended target population? Is the program in compliance with federal law, state statutes, and professional
standards?
Program monitoring can be thought of as the systematic study of keyaspects of program performance (a process or implementation
evaluation)
Program monitoring focuses on documenting the extent to which anintervention is actually delivered, received, and adhered to bytargets
8/2/2019 11 EDUC 612 Program Evaluation_slides
24/32
Monitoring Program Processes
Program monitoring is used to provide quality assuranceinformation (e.g., accountability), identify impediments to
service delivery, and document the processes by which a
program works
Program monitoring can also uncover service delivery breakdowns (e.g., program materials not received or
utilized)
unwanted side effects (e.g., NCLB: narrowing of content, teaching to thetest)
and confounds (e.g., extra treatments) early in the service deliveryprocess
Program monitoring thus has great diagnostic value for theimpact evaluation
8/2/2019 11 EDUC 612 Program Evaluation_slides
25/32
Implementation Fidelity
The monitoring of program processes is the means by whichimplementation fidelity is assessed
Definition: The degree to which a program is delivered as intended(adheres to program protocol)
Synonyms: treatment integrity, treatment fidelity, programadherence, protocol compliance
Implementation fidelity is a relatively new concept (emerged inthe 1970s) and is an often overlooked aspect of the
evaluation process increasingly common Reasons for not closely considering implementation fidelity:
the general lack of clearly defined treatment models, resource
costs, and attitudes about the professional discretion of
providers
Th I t f
8/2/2019 11 EDUC 612 Program Evaluation_slides
26/32
The Importance of
Implementation Fidelity
When null results are obtained, close monitoring of the degreeto which a program is implemented as designed enables one
to distinguish between implementation and theory failure
Implementation failure: the failure of the program to adequatelydeliver the service or perform the actions to elicit change (i.e., theintervention is delivered inconsistently or not at all)
Theory failure: a failure of the programs conceptual model(correctly implemented program services do not bring about the
expected proximal or distal outcomes) When positive results are obtained, fidelity data can be used
to ascertain whether the results were due to program or the
non-program activities of providers
8/2/2019 11 EDUC 612 Program Evaluation_slides
27/32
Class Activity:
How will you conduct programmonitoring?
i.e., measure fidelity?
8/2/2019 11 EDUC 612 Program Evaluation_slides
28/32
4. Program Impact
Evaluation to answer questions about programeffectiveness (i.e., did the program have the
intended effect?)
Need to separate the effects of program fromthe intake characteristics & natural maturation of
program participants
Social science research methods provide the key
8/2/2019 11 EDUC 612 Program Evaluation_slides
29/32
Isolating Program Effects
0
10
20
30
40
50
60
Pre Program Post Program
8/2/2019 11 EDUC 612 Program Evaluation_slides
30/32
Isolating Program Effects
0
10
20
30
40
50
60
Pre Program Post Program
ControlGroup
ProgramEffect
TreatmentGroup
8/2/2019 11 EDUC 612 Program Evaluation_slides
31/32
5. Program Efficiency
Evaluation to answer questions about program costand cost effectiveness (i.e., did the ends justify the
means?)
Follows logically from the evaluation of programimplementation and impact necessary to justify
continued support (forces consideration ofalternatives)
8/2/2019 11 EDUC 612 Program Evaluation_slides
32/32
Class Activity:
What are important considerations related toprogram impact and program efficiency?