Evaluating Financial Education Programs:

A framework for measuring results Ellen Taylor-Powell, Ph.D.

Evaluation Specialist

American Savings Education CouncilApril 18, 2007

Washington, D.C.

University of Wisconsin-ExtensionCooperative Extension

The state of affairs …

• Evidence tells of the situation we face: low saving rates; high debt; high bankruptcy rates, shift in policies, sophisticated financial markets and products…

• Partners have responded in force to build financial literacy with a myriad of investments and activities

• The scope and size of the financial educational effort is impressive

Our purpose today…

What difference is all this effort making?Does financial education work?

How do we know?

• Examine the logic model as a framework for planning and evaluation

• Explore evaluation challenges and options • Discuss how we can make evaluation

meaningful

Accountability age

No longer good enough to report numbers of services delivered; number of participants reached

Need documentation of changes: improvements for individuals, families, communities

Need to be able to answer:So what?

“I think you should be more explicit here in Step Two.”

What is a Logic Model?

A graphic depiction of a program that shows the relationship among activities and intended results – Linkage between and among components: Program

“theory” or “program action”

– Intended results: what the program is to accomplish

Serves as a tool for integrating planning and evaluation – Helps with both planning and evaluation

Is not an evaluation model, method or realityIs widely used: nonprofits, public sector, foundations,

international agencies, evaluation communityOther names: theory of change, program action,

program theory, causal road map

Family Members

Budget

Car

Camping Equipment

Drive to state park

Set up camp

Cook, play, talk, laugh, hike

Family members learn about each

other; family bonds; family has

a good time

A Family Vacation

A youth financial literacy program

Partners invest resources

A high school financial planning program – 7 unit curriculum - is developed and delivered in high schools

Teens gain knowledge and skills in money management

Teens establish sound financial habits

Teens make better decisions about the use of money

Comprehensive Cancer ControlIF THEN

ProvideEducation

BuildPartnerships

EnhanceInfrastructure

UtilizeResearch &

Data

Integrated &CoordinatedPlans withEffective

InterventionsIndividualBehaviorChange

System,Environmental,

& PolicyChanges

AppropriateRehabilitationand Support

EffectivePalliation

New CasesPrevented

CancersDetectedEarlier

State-of-the-ArtTreatment

DecreaseMorbidity

DecreaseMortality

ReduceDisparities

IncreaseQuality of

Life

Common graphic of logic model the “four box approach

INPUTS OUTPUTSACTIVITIES OUTCOMES

A bit more detail

INPUTS OUTPUTS OUTCOMES

Program investments

Activities Participation Short Medium

What we invest

What we do

Who we reach

What results

SO WHAT??

Long-term

Fully detailed

OUTPUTS What we do Who we reach

ACTIVITIES• Assess needs and assets• Design and deliver curriculum • Conduct workshops; teach• Provide one-on-one counseling• Produce products - Exhibits - Printed materials - Electronic materials•Populate/maintain web sites• Hold Conferences • Work with the media• Facilitate policy dialogue• Partner – collaborate• Advocate

PARTICIPATION

•Underserved populations Low income consumers•Youth: elementary, high school, post grad; college students•Homebuyers/homeowners•Military personnel•Teachers/educators•Immigrant populations•Mandated audiences •Financial professionals

Reactions – Satisfaction

OUTCOMESWhat results for individuals, families, communities..…

SHORTLearning

Changes in • Awareness• Knowledge• Attitudes• Skills• Opinion• Aspirations• Motivation• Behavioral intent

MEDIUMAction

Changes in •Behavior

•pay bills on time + savings+ assets–debt•achieve personal goals •spend wisely•track spending and income

•Decision making•Policies•Social action

LONG-TERMConditions

Changes in • Human

•well-being•stability•security

• Economic• Community

–civic activity

C H A I N OF O U T C O M E S

Reactions

Learning

Actions

Social-economic-environmental improvements

Hierarchy of effects

Source: Bennett and Rockwell, 1995, Targeting Outcomes of Programs

Number and characteristics of people reached; frequency and intensity of contact

Degree of satisfaction with program; level of interest; feelings toward activities, educational methods

Changes in knowledge, attitudes, skills, aspirations

Changes in behaviors and practices

Participation

Staff

Money

Partners

Develop parent ed curriculum

Deliver series of 8 interactivesessions

Parents increase knowledge of child dev

Parents better understanding their own parenting style Parents use

effective parenting practices

Improved child-parent relations

Research

INPUTS OUTPUTS OUTCOMES

Facilitate support groups

Parents gain skills in effective parenting practices

Simple logic model

Parents identify appropriate actions to take

Parents of 3-10

year olds

SITUATION: During a county needs assessment, majority of parents reported that they were having difficulty parenting; were often uncertain about what to do and felt stressed as a result.

Parents increase knowledge of community resources

Staff

Time

Money

Volunteer

Partners

MaterialsSupplies

Research- best practices

Develop, deliver parent ed curriculum

Do home visits;Outreach

Parents/ - caregivers:Single parentCouplesDivorcedTeen parentFathersLow-incomeHomelessMandated

Childcare providers: directors, staff

Demonstrate positive parenting practices; improved family communications

Children enter school ready to learn

Demonstrated language, communication and social skills

Use community supports/resources

Increased abilities to socialize and communicate

Implement quality improvement strategies

Changes in knowledge, skills intent re. child dev.; parenting style

Elaborated logic model - Parent support initiative

Increased knowledge of community resources

Increased knowledge of quality improvement

strategies: developmentally appropriate practice, child assessments, learning environments

Early detection of delays and referrals

OUTCOMES

Improved child-family relati’ships

Familiesconnectedto community

Facilitate parent support groups

Provide parent/childActivities:Play group; Toy lending

Provide individual services; counseling

TrainSupportDo assessments

Children

Ages

Low-incomeSpecial needs

Increased ability to advocate for self and families

Advocate for self and family

Inclusive, appropriate services available

Situation

What does a logic model look like? • Graphic display of boxes and

arrows; vertical or horizontal– Relationships, linkages

• Any shape possible– Circular, dynamic– Cultural adaptations; storyboards

• Level of detail– Simple– Complex

• Multiple models– Multi-level programs

– Multi-component programs

EVALUATION: check and verify

What do you want to know? How will you know it?

PLANNING: start with the end in mind

EVALUATION: check and verify

What do you want to know? How will you know it?

PLANNING: start with the end in mind

Plan backwards

Logic Model and Planning

• Apply to any level: national plan, statewide plan, individual plan of work, specific project/activity plan

• Model vs. a more detailed narrative plan or a management plan

• Focus on outcomes: “start with end in mind”

Integrate evaluation into program planning

Think about:• What do you (and other key stakeholders)

want to know?• What data will you want to have?• When will it be appropriate to collect data?• What evidence will be credible; believable

by various consumers? If you told me that people are benefiting, what would I see if I came to visit?

• Is it your program or something else that is making this difference?

EVALUATION: check and verifyWhat do you want to know? How will you know it?

INPUTS OUTPUTS OUTCOMES

Program investments

Activities Participation Short MediumLong-term

Indicators: What evidence do you need to answer your questions?

Match evaluation questions to program

Evaluation questions: What questions do you want to answer?

e.g., accomplishments at each step; expected causal links; unintended consequences or chains of events set into motion

Staff

Money

Partners

Parents increase knowledge of child dev

Parents better understand their own parenting style

Parents use effective parenting practices

Improved child-parent relationsResearch Facilitate

support groups

Parents gain skills in effective parenting practices

Parents identify appropriate actions to take

To what extent are relations improved?

To what extent did behaviorschange? For whom? Why? What else happened?

To what extent did knowledge and skills increase? For whom? Why? What else happened?

Who/how many attended/did not attend? Did they attend all sessions?Supports groups? Were they satisfied – why/why not?

How many sessions were held? Quality of implementation? #, quality of support groups?

What amount of $ and time were invested?

Example: Evaluation questions and indicators

Deliver series of 8 interactive sessions

EVALUATION QUESTIONS

# Staff$ used# partners

# Sessions held

Quality criteria

INDICATORS

#,% attended per session

Certificate of completion

#,% demonstrating increased knowledge/skills

Additional outcomes

#,% demonstrating changes

Types of changes

#,% demonstrating improvements

Types of improvements

Develop parent ed curriculum

Parents of 3-10

year olds

Logic model and evaluation

Needs/asset assessment:

What are the characteristics, needs, priorities of target population?

What are potential barriers/facilitators?

What is most appropriate to do?

Process evaluation:

How is program implemented?

Are activities delivered as intended? Fidelity of implementation?

Are participants being reached as intended?

What are participant reactions?

Outcome evaluation:

To what extent are desired changes occurring? Goals met?

Who is benefiting/not benefiting? How?

What seems to work? Not work?

What are unintended outcomes?

Impact evaluation:

To what extent can changes be attributed to the program?

What are the net effects?

What are final consequences?

Is program worth resources it costs?

Key considerations

• Match evaluation to stage of program’s development

• Define evaluation purpose• Identify specific evaluation

questions• Select measurable indicators• Match design to purpose, audience

and resources• Attend to cultural relevance

Evaluation challenges

Internal• Commitment• Training• Common

language and understanding

• Value• Leadership• Resources

Community context• Complex issues• Lagged outcomes• Attrition• Non-response• “Noisy” systems• Implementation

fidelity• Varied programming:

contexts, content, intensity, audience and implementation

So, what can we do?

Internal• Build capacity• Provide leadership• Showcase use• Institute

operational and structural changes

• Provide resources• Practice, practice,

practice

Community context• Use logic model to

show connections• Match evaluation to

purpose, program, questions

• Use multiple sources of information

• Mix methods• Conduct formative

research

Summing up

Logic model links activities to results integrates planning and evaluation helps match evaluation to the program

effort identifies points for appropriate data

collection

Evaluation challenges are many but not insurmountable! Remember that evaluation equals

learning

Resources

“We build the road and the road builds us.”

-Sri Lankan saying

• www.uwex.edu/ces/lmcourse• www.uwex.edu/ces/pdande• http://www.cdc.gov/eval/index.htm• http://citnews.unl.edu/TOP/english/• Weiss, Carol. 1998, Evaluation,

2nd Ed; Prentice-Hall, Inc.