Upload
gyles-berry
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
Evaluation Criteria
S519
What is evaluation? What are the steps involved? What are step1 and step2? What are absolute or relative merit?
Review
S519
To build a criterion list, consider the following procedures: A needs assessment Logic model of linking the evaluand to
the needs An assesment of other relevant values,
such as process, outcomes, and cost A strategy to organize your criterion
checklist
Step3: Defining evaluative criteria
Make sure that you go into the evaluation with a well-thought-out plan so that you know what you need to know, where to get that information, and how you are going to put it together when you write up your report.
S519
Needs that we identify become the outcome criteria we use for the evaluation
The data collected during the needs assessment phase can be used as baseline data for comparison if we wish to track change in certain outcome variables.
Needs assessment
S519
Understand the true needs of your evaluation end users (consumers or impactees)
Who are your end users? They are the person or entity who buys or uses a
product or service, enroll in a training program,etc. Upstream stakeholder (i.e. People on upper level of the
structure – manager, designer) Immediate recipients (i.e. People who directly consume your
product or service – consumer, trainee) Downstream consumers (i.e. People who indirectly involved in
your evaluation)
Needs assessment
S519
Rules to follow during the evaluation Systematic (step-by-step, thorough) Objective (free from bias) Transparent (easy to repeat and follow)
Needs assessment
S519
Needs vs. Wants Difference and why A need is something without which unsatisfactory functioning
occurs. Different kind of needs
Context dependence Conscious needs vs. Unconsious needs
Needs we know and needs we do not know Met needs vs. Unmet needs
Building a factory (increase job, but create pollution) Performance needs vs. Instrumental needs
„need to do“ something for satisfactory functioning (actual problems) vs. Proposed solutions
Access email vs. Lightweight laptop Most of the case, performance needs is considered, but not the
instrumental needs
Understanding needs
S519
Two phases: Identifying and documenting
performance needs Investigating the underlying causes of
performance needs
Needs assessment method
S519
Steps Examples
Document the extent of the presenting need by gathering evidence
- Ask police for teen drug arrests
Find out more about individuals - Is the drug use more prevalent among boys or girls
Look for unnoticed performance needs - e.g. By combining of open-ended inquiry
-Have there also been problems with violence, or other crimes- asking parents, teachers or community members
Come out the clear picture of the needs
- Lay out the evidence
Identifying performance needs
Identify the performance needs for an increase in drug abuse by teenagers
S519
Using logic model
Investigating the cause of the needs
Training program
Improved skillsImproved
performance
If we implement thisWe will address this underlying need
Which should solve our performance problem
S519
If we implement this
We will address this underlying need
Which should solve our performance problem if that was the true underlying need
Training program Unskilled employee
Education and communication
Improved performance
Performance incentives
Unmotivated staff
Job enrichment Unchallenged job
Provision of resources Lack of resources
Flexible scheduling Home and work conflict
Organizational climate intervention
Bad working enviornment
Exercise
S519
Asking potential participants about their needs is just the tip of the iceberg
Many important information has to be gathered in some other way Working with experienced upstream impactees
Working with project officer for grant writing training program
Identifying correct cause by asking open-end inquiry to downstream and upstream impactees
Poor involvement of grant writing Lack of motivation – increase incentive Lack of confidence – build up self confidence
More to do
S519
Think about other criteria based on Process evaluation Outcome evaluation Comparative cost-effectiveness Exportability Like: legal, ethical, authenticity, scentific, economic,
historical requirements, etc. Being diagnostic Collecting facts
Other relevant criteria
S519
Needs assessment Identify consumers or impactees (e.g. Table3.2) Identify different needs (e.g. Table3.3)
Logic model (e.g. Exhibit3.6 and Exhibit3.7) An assessment of other relavent values
with the consideration of process, outcome and cost (e.g. Table3.4)
Organizing your criteria see step4 output report
Step3: output report
S519
Rolling design unanticipated criteria may add to the list Existing criteria may be modified
Never draw a conclusion based on a single piece of evidence
Try to view the same issue from different angles through: Different types of data (both qualitative and quantitative) Multiple sources of information (e.g., existing document,
observations, input from more than one group of stakeholders)
Step4: Organizing criteria and indentifying sources of evidence
Triangulation: using different ways or data to verify the conclusionStep-by-Step: start the small scope of data collecting, extend it slowly later on.
S519
When organizing your criteria, always keep the followings in mind:
Process How good are the evaluand‘s content and implementation
Outcomes How good are the impacts on immediate recipients and
other impactees Comparative Cost-Effectives
How costly is it? Excessive, quite high, acceptable or reasonable
Exportability How can we extend this to other settings?
Checkpoints
S519
Process evaluation Content
What the evaluand consists of, i.e., basic components or design)
Implementation How well or efficiently the evaluand was
implemented or delivered to the consumers who needed it
Other features Any other features that make the program good
or bad which are not covered by the first two and are not outcomes or cost-related criteria
The process evaluation checkpoint
S519
Now feed into these three categories with all the process-relevant ingredients we generated from: the needs assessment and other relevant
needs (e.g. Table3.3, Table 3.4, Exhibit3.6) Add additional consideration if necessary
The process evaluation checkpoint
category Subcategories and criteria
Source of evdience
Content evaluation
Implementation evlauation
Others (if necessary)
S519
What is outcome Things that happen as a result of the program Outcomes can affect anyone listed as consumers
How to do Based on logic model in step3 (e.g. Exhibit3.6 and
Exhibit3.7) Organize them into subcategories See Table4.3 (D-p60)
The outcome evaluation checkpoint
category Subcategories and criteria
Source of evdience
Knowledge, skill and attitude gain
Application of knowledge, skill and attitudes
S519
Any evaluation has to take cost into account
What are costs? Money Time Effort Space Opportunity costs
The comparative cost-effectiveness checkpoint
S519
Type of costs Money, resources, time, opportunities
Costs When Preparation, implementation,
maintenance, evaluation Costs to whom
Participants, community organizations, workshop provider, others
D-p61
Cost cube (Scriven, 1991)
S519
Preparation
Implementation
Maintenance
Evaluation
Money
Resources
Time
Opportunity
Costs to Whom
Comparison with competitors
Cost cube table
• Find out the most important costs for your evaluation• Compare your cost with other competitors
S519
What elements of the evaluand (i.e., innovative design or approach) might make it potentially valuable or a significant contribution or advance in another setting Think about examples
The exportability checkpoint
S519
Checkpoints for Process Outcomes (e.g., Table4.3) Comparative Cost-Effectives (e.g., cost
cube table) Exportability
Short summary of potential areas for exportability
Step4: Output report
S519