project evaluationsproject evaluations

a framework for your own project

who needs a project evaluation anyway?

• If your thesis/dissertation involves the implementation of a project or device and you want to prove its effectiveness

• If you are conducting an experiment these might be helpful guidelines

• If you are conducting a workshop or a teaching seminar• Federally funded projects must be evaluated (Government

Performance and Results Act, GPRA, 1993)– If you have an NSF grant to do a project

why do project evaluations?

• Federally funded projects must be evaluated (Government Performance and Results Act, GPRA, 1993)– If you have an NSF grant to do a project

• proves causal connections • provides information on how to improve a project• allows you to understand if goals were met• can account for unanticipated consequences • can better communicate to stakeholders

types of evaluations see also visual on

board

1) Formative evaluation: begins during project and continues throughout project life

a) Implementation evaluation: is the project going as planned?

b) Progress evaluation: is the program meeting its goals? (most important in the beginning stages of the project)

2) Summative evaluation: did the project meet its stated goals? Similar to progress evaluation, but at the end of project

“When the cook tastes the soup, that’s formative.When the guests taste the soup, that’s summative.”

the big picture

1) Develop a conceptual model of the project* 2) Identify project goals*3) Define evaluation questions and measurable

outcomes* 4) Develop evaluation design* 5) Collect data6) Analyze data7) Provide information to interested audiences *the rest of the presentation only focuses on 1-4

1)Develop a conceptual model and 2) identify project goals

The logic modelInputs Activities Short-Term Long-Term

• Fill in the model• Set up a timeframe for project• Sometimes helpful to work backwards (start with long-term

outcomes)

1) my conceptual model and project goals

• Inputs: time to plan, help from other TAs and Donna, research guidance from Ken

• Activity: in recitation sections, TAs choose one of three activities and do the activity with the class to discuss either a) ecotourism, b) participatory development, or c) the causes of poverty

• Short-term outcomes: students actively participate in the activities, explore issue a, b, or c and discuss/reflect on the issues with other students

• Long-term outcomes: students are more aware of international development issues and will continue to learn about them outside of class

3) Define measurable outcomes and evaluation questions

four steps1) Identify key stakeholders

(see pg 7 the NSF Guide)2) Formulate evaluation questions that stakeholders might ask

-may produce more than you’ll actually use, but that’s OK 3) Define outcomes in measurable terms

-use the questions generated in step two to fill out number three in the Goals & Objectives worksheet (see pg 9)

4) Prioritize & eliminate questions -consider your most important stakeholders first -what is success? Statistical significance, etc -consider resources: time, money, staff capability (general rule: 5-10% of cost for evalutions

3) measurable outcomes and evaluation questions

• Example of stakeholder questions • Measurable outcomes: Example of goals

worksheet

• Prioritize: example of survey

4) Develop an evaluation design

• Select methodological approach and data collection instruments – Data collection matrix (see pg 15)

• Determine who will be studied and when– randomized samples are best– Comparison groups if possible– If not, maybe levels of implementation?

evaluation design

• student surveys (450-500)• TA surveys (8)• professor survey (1)• any possible follow-up interviews with any of the

above three• must acknowledge the parameters of my samples

references

• NSF 2002: The 2002 User-Friendly Handbook for Project Evaluation

• NSF 1997: The User-Friendly Handbook for Mixed Method Evaluations

• Stake, R. 1957: Program Evaluation, Particularly Responsive Evaluation

• Yin, R.K. 1989: Case Study Research: Design and Method