Upload
aletta
View
27
Download
0
Embed Size (px)
DESCRIPTION
Some Practical Tips for Measuring Financial Success Dr. Angela Lyons University of Illinois. Program Evaluation II: Creating Your Evaluation Toolkit Presented by Dr. Angela Lyons University of Illinois October 2009. Identifying the “ideal” approach to program evaluation. - PowerPoint PPT Presentation
Citation preview
Some Practical Tips for Measuring Financial Success
Dr. Angela LyonsUniversity of Illinois
Program Evaluation II: Creating Your Evaluation Toolkit
Presented byDr. Angela Lyons
University of Illinois
October 2009
Evaluation methods and measures vary widely across programs and academic disciplines.
Wide variation in financial outcomes across programs.
Significant differences in financial needs across consumers.
Some participants unable to implement certain financial behaviors.
Identifying the “ideal” approach to program evaluation
Then you need to decide….
“To evaluate or not to evaluate, that is the question?”“To evaluate or not to
evaluate, that is the question?”
Questions to ask yourself …. At the end of the day, what are you trying to show? What is the purpose of the evaluation? Who will use the information – and how? What information do you want to collect? Who is your target audience? What is your primary delivery method? What are your available resources (i.e., time, money,
and staff)? What is your timeline? What is your expertise and evaluation capacity? Who are your partners, funders, and stakeholders?
Common survey methods used to collect impact data
• Post evaluation only • Retrospective pre-test (RPT)• Pre and post evaluation• Follow-up• Stages to Change (TTM)• Control groups and longitudinal studies
Key question to ask: What is the length of your program?
Post evaluation only
When to use: Short programs that are less than 2 hours
Advantages: Only need to survey group once. Good for limited-resource audiences and groups that are
transient. Relatively inexpensive and less time intensive. Can document participants’ levels of knowledge, skills, and
planned behaviors at the end of the program.
Disadvantages: With no pre assessment, it’s difficult to document potential and
actual changes in knowledge, attitudes, and behavior.
Retrospective pre-tests (RPTs); The Post-Then-Pre Evaluation
Retrospective pre-test (RPTs)
When to use: Any program, but typically 2 hours or less
Advantages: Only need to survey group once. Good for limited-resource audiences and groups that are
transient. Controls for “response shift bias.” Can document “relative” change.
Disadvantages: Potential for respondent bias (social desirability factor). Self-assessment measures are subjective.
7
Pre and post evaluations
When to use: Programs that are 2 hours or longer
Advantages: Can compare pre and post responses and document changes in
knowledge, attitudes, and behavior. Can be used to document immediate changes in knowledge, skills
and planned behaviors following the program.
Disadvantages: More time intensive. Identification numbers are needed to match pre and post surveys. May be difficult to show actual behavior change. May be difficult to show that the intervention caused the change. Doesn’t account for other possible reasons for change. Transient populations may lead to low unmatched evaluations.
Follow-ups
When to use:
• Program is comprehensive enough to potentially result in intermediate and long-term impact.
• Must have adequate resources and evaluation capacity.
• Usually administered three to six months after the program.
• Can document changes in actual financial behaviors, ability to achieve financial goals, and overall financial position.
Stages to Change (TTM)
When to use: Programs that have multiple sessions
Advantages: Can document intermediate and long-term change. Easier to measure actual behavior change and to control for other
factors that may lead to change over time. Can identify stage at which individual is ready and able to change
behavior. Behaviors can be recorded at the beginning, middle, and end of the
program so that changes in actual behavior can be observed.
Disadvantages: Time and resource intensive. May require additional progress reporting and long-term follow-up. Can only be used with multi-session programs.
Train-the-trainer evaluations Similar to pre and post evaluation,
but more content specific.
Covers subject material in more detail to ensure that trainers have an adequate level of knowledge to teach the program to others.
Can be used to document changes in both the instructors’ teaching skills and personal financial behaviors.
Follow-ups can document how the curriculum materials are being used and identify additional programming needs.
Designing the evaluation instrument:Key survey content
1. General reactions to the session2. Changes in knowledge3. Changes in motivation, confidence, and abilities4. Intended changes in behavior5. Actual changes in behavior6. Future programming needs and preferences7. Demographics8. Qualitative / open-ended responses
General reactions to the session
Please rate the instructor(s), materials, and the overall program by checking the box that best applies.
Not Helpful
SomewhatHelpful
Helpful Very Helpful
Instructor(s)
Educational materials
Overall program
Measuring changes in knowledge
Testing Knowledge Please circle your answer to each of the following statements.
1. Fixed expenses are expenses that typically change from month to month such as food, clothing, and utilities.
True False Not Sure
2. Financial experts recommend having an emergency fund that is equal to 3-6 months worth of living expenses.
True False Not Sure
3. Goals should only be made for long-term plans such as homeownership, college tuition, or retirement.
True False Not Sure
4. Gross income is defined as income after taxes and other withholdings have been subtracted from net income.
True False Not Sure
5. Credit reports can affect an individual’s ability to get a job, purchase a home, and obtain home and auto insurance.
True False Not Sure
Measuring changes in knowledge (conti.)
Format can be True/False or multiple choice.
True/False is reliable indicator for low literacy audiences and youth.
The more questions you ask, the greater the reliability measure.
May include a “don’t know” option to control for guessing.
Post-test: 10 questions (established standard)
Pre- and post-test: 10-20 questions
Train-the-trainer: 10-25 questions
Changes in motivation, confidence, and abilities
Building Skills/Confidence Indicators Please check the box that best describes your confidence to do the following:
Your confidence to: Not Confident
A little confident
Somewhat confident
Confident Very confident
Set short and long-term financial goals.
Save money regularly.
Use a spending plan to track income and expenses.
Pay bills on time each month.
Changes in motivation, confidence, and abilities (conti.)
Recording Participants’ Attitudes Please check the box that best describes how much you agree with the following statements.
Statement Strongly Disagree
Disagree Undecided Agree StronglyAgree
Saving money regularly is important to me.
Keeping track of spending is a good habit.
Planning my personal budget is a priority.
Intended changes in behavior
Please indicate how often you plan to do each of the followingfinancial practices. There is no “right” or “wrong” answer. (Choose only one)
As a result of this program, you plan to:
No Maybe Yes Already doing this
Does not apply
Set short and long-term financial goals.
Use a spending plan to track income and expenses.
Pay bills on time each month.
Financial Practice Never Rarely Sometimes Usually Always
Set short and long-term financial goals.
Save money regularly.
Use a spending plan to track income and expenses.
Pay bills on time each month.
Actual changes in behavior
Please indicate how often you are currently doing each of the following financial practices. There is no “right” or “wrong” answer. (Choose only one)
Financial Practice Never Rarely Sometimes Usually Always
Setting short and long-term financial goals.
Saving money regularly.
Using a spending plan to track income and expenses.
Paying bills on time each month.
Financial Progress IndicatorsPlease check the box that best describes how your financial position has changed since completing the program. Then, indicate specifically how your financial position has changed.
Capturing behavior change in follow-ups and “stages”
DecreasedStayed
about the same
Increased
Amount of change,
if any (% or $)
Monthly income
Monthly expenses
Amount saved monthly
Current credit card debt
Capturing behavior change in follow-ups and “stages”
Progress ReportingPlease record your financial position based on your current progress in the program.
Financial PositionAt the
beginning of the program
In the middle of the program
At the end of the program
How much credit card debt do you owe? ($)
How many credit cards do you have? (#)
What is the highest interest rate on your credit cards? (%)
How much do you have in savings? ($)
A few words about train-the-trainer programs….
Testing knowledge Building teaching skills Shaping personal skills Taking action for teaching Taking action for personal financial success Follow-ups
Qualitative / Open-Ended Questions (common examples)
“Post Evaluation Only” and “Pre and Post Evaluation” What did you like the most about this program? What did you like the least about this program? How could this program be improved? Would you recommend this program to others?
“Stages to Change Evaluation” What has made it easier for you to improve your financial practices? What has prevented you from improving your financial practices? With respect to the overall program, what did you like the most? What did you like the least? How could this program be improved? Have you shared what you learned with others? Would you recommend this program to others?
Qualitative / Open-Ended Questions (conti.)
“Train-the-Trainer Evaluation” What was the most helpful information you received during this
training program? How could this training program be improved? How do you plan to share this information with your target
audience(s)? What information and materials from this training do you plan to
share with your target audience(s)? Will you share what you learned with other instructors and
colleagues? Would you recommend this training program to other instructors
and colleagues?
Demographic Questions
Age Gender Race, Ethnicity, and Language Marital Status Education Employment Family Structure Health Status Income, Assets, and Debts Region/Location
Financial Experience Students/Youth Instructors/Educators
Longitudinal data?Control groups?
Randomized experiments?
Rigor vs. Reality
NEFE Financial Education Evaluation Toolkit ®
http://www2.nefe.org/eval/intro.html
NEFE Financial Education Evaluation Toolkit®
Database• Post evaluation only with option for follow-up• Pre and post evaluation with option for follow-up• Stages to Change Evaluation• Train-the-Trainer
• Testing Knowledge• Building Skills• Taking Charge
Manual• How-to-guide for grass-roots level organizations• Examples (survey instruments, executive summary, reports)• Guidance on how to organize and present impact data
Manualhttp://www2.nefe.org/eval/manual.html
Manualhttp://www2.nefe.org/eval/manual.html
Part I: Financial Education OverviewPart II: Understanding Program EvaluationPart III: The Evaluation Planning ProcessPart IV: Using the Evaluation Database
Part V: Reporting Program Impact
Part I:Financial Education Overview
Part II:Understanding Program Evaluation
Part III:The Evaluation Planning Process
Part IV:Using the Evaluation Database
Part V:Reporting Program Impact
Appendix:Sample Evaluation Instruments
Databasehttp://www2.nefe.org/eval/index.php
Step 1:Program Info and Follow-up
Step 2a:Knowledge Indicators
Step 2b:Customizing Questions
Step 3:Confidence and Behavior Indicators
Step 4a:Selecting Statements
Step 4b:Customizing Statements
Step 5:Qualitative Info
Step 6:Demographics
Step 7:Follow-Up: Financial Progress Indicators
Step 8:Follow-Up: Personal Achievements
Step 9:Follow-Up: Demographics
Step 10:Finalizing Evaluation
ACTIVITY: Putting It All Together
(Evaluation Action Plan – Part B)
Post-test only Retrospective pre-test Pre and post-test Follow-up survey Stages-to-change Focus groups
Interviews Case studies Observations Stories/anecdotal evidence Tests of ability Other
Think about your signature program, what is the most appropriate evaluation method?
What types of questions will your evaluation seek to answer?
I would really like to know….
1.
2.
3.
Changes in satisfaction levels Changes in knowledge Changes in skills and confidence levels Changes in attitudes Changes in aspirations Anticipated or intended changes in behavior Actual changes in behavior Socio-economic changes Other
What types of indicators will you useto document this impact?
List some specific indicators.
1.
2.
3.
4.
U of I Center for Economic and Financial Educationhttp://www.cefe.illinois.edu/
Educational Tools for Evaluationhttp://www.cefe.illinois.edu/tools/evaluation.html
Checklist of evaluation resources
Program Planning GuideEvaluation Action PlanEvaluation Road MapEvaluation Reading ListSample EvaluationsNEFE® Financial Education Evaluation Toolkit
http://www.cefe.illinois.edu/tools/evaluation.html
University of Wisconsin-Extensionhttp://www.uwex.edu/ces/pdande/evaluation/index.html
Cornell University Extensionhttp://staff.cce.cornell.edu/administration/program/evaluation/evalrefs.htm
Penn State Extensionhttp://www.extension.psu.edu/evaluation/
Contact Information
Dr. Angela Lyons Associate ProfessorDirector, U of I Center for Economic and Financial EducationUniversity of Illinois
Phone: 217-244-2612 E-mail: [email protected]
Questions