Evaluating Your STEM Outreach Program

Evaluating Your STEM Outreach Program

MISO Spring WorkshopMay 7, 2012

MISO Data Analytics TeamJeni Corn, Tricia Townsend, Alana Unfried

http://miso.ncsu.edu/

http://miso.ncsu.edu/

Goals of the Session

• Understand how program evaluation process can be helpful to improving your STEM programs

• Ask evaluation questions that are useful to your STEM programs

• Identify a variety of useful data sources for your STEM programs

Agenda

Introductions

The Evaluation Process - Presentation

Asking Good Evaluation Questions - Small group discussion

Identifying Data – Presentation, Small group discussion

Q & A - Whole group discussion

Formative and Summative Evaluation

Just like formative and summative assessments …

“When the cook tastes the soup, that's formative evaluation. When the guests taste the soup that

is summative evaluation.” ~ Bob Stake

Keep it Simple and Focused

• An evaluation doesn’t have to be big! Match the number of evaluation questions in your plan to your resources.

• Focus on efficient, effective data collection strategies.

A basic chart is a great way to organize your own thinking and to share easily the plan with others. Evaluation Questions

Data Sources

Program Evaluation Steps 1-5, Repeat!

1. Identify the critical elements of the STEM program (logic model).

2. Ask important questions about your STEM program. (evaluation questions).

3. Identify what data are available to help answer your questions and determine the additional data needed (MISO surveys, NCERDC data, program level data).

4. Collect, analyze, and interpret data to answer your questions.

5. What changes to your STEM program should you make based on your results?

Repeat steps 1-5!!6




3. Identify what data are available to help answer your questions and determine the additional data needed (MISO surveys, NCERDC data, program level data).



Repeat steps 1-5!!7

What is a logic model?

A logic model is a graphic

representation of the

relationships among the key elements of a

project: inputs, strategies,

objectives, long-term goals.

• Helps to articulate the key elements of the project.

• Enables evaluation efficiency and effectiveness.

• Promotes stakeholder buy-in by helping clarify how the project works.

• Drafting one can be a great way to involve stakeholders in planning.

What is a logic model?




3. Identify what data are available to help answer your questions and determine the additional data needed.



Repeat steps 1-5!!10

Developing evaluation questions

• The process for identifying the questions to be answered by the evaluation is critical.

• Evaluation questions provide the direction and foundation for the entire evaluation.

EVALUATION QUESTIONS

Data Collection

Data Analysis

Results


Why do we need to ask good questions? To …

• Determine what is really important and to whom• Project leaders, program participants-teachers,

students and their parents, etc.• Focus data collection efforts

• What do we need to find out?• How can we collect that information?• Who is the best person to collect that information?


The main types of evaluation questions are:

1. Questions about STRATEGIES: these questions ask about how well the strategies were implemented.

2. Questions about OBJECTIVES: these questions ask about impacts.

Logic models are great guides for developing evaluation questions.


Quick tips for writing good questions:1. Try to avoid simple “yes or no” questions2. Consider QUANTITY questions, e.g:

• “How many”• “How much”• “How often”

3. Consider QUALITY questions, e.g.:• “How well”• “How effectively”• “In what ways”

4. Be able to be tuned-in to unexpected results.


IMPLEMENTATION questions:

- How many hours of sleep am I getting each week? (quantity)

- How soundly am I sleeping? (quality)

IMPACT questions:- How much weight have I

lost? (quantity)- How has my stress level

changed? (quality)


Every evaluation question can’t be answered - finding out the answers costs time, money and people.

Pick the most important questions that provide the most valuable information to users.

Small Group Activity

With a partner(s) at your table, use the sample STEM program logic model to:

1. Brainstorm 2-3 implementation questions about the program’s strategiesE.g. Strategy: Teachers will engage in face-to-face and online professional development.

Quantity Questions: What percentage of teachers attend the PD regularly? What were the total number of hours teachers attended PD each month? Over the course of the program?

Quality Question: How do teachers rate the professional development?

2. Brainstorm 2-3 impact questions to evaluate how well the outcomes are being met.

Whole group share-out.

STEM Program Implementation Questions

Quantity Questions Quality Questions

STEM Program Impact Questions

Quantity Questions Quality Questions


1.Identify the critical elements of the program (logic model).

2.Ask important questions about your program.

3.Identify what data are available to help answer your questions and determine the additional data needed.

4.Collect, analyze, and interpret data to answer your questions.

5.What changes to your program should you make based on your results?

Repeat steps 1-5!!

Data in Evaluations

For each evaluation question, what information are you going to gather in order to answer it?

1. Consider a wide variety of data types and sources – both quantitative and qualitative.

2. What data do you already have?3. What data do you need? 4. How much time, money and/or other resources will it

cost to collect the data?5. Make a calendar of what you’ll need, from who, by

when.

REMEMBER: Data must be interpreted, not just analyzed. 21

Data in STEM Evaluations

MISO Instruments:• Student STEM

Attitudes Surveys• Upper Elementary• Middle/High

• Teacher STEM Attitudes Surveys• Elementary• Science• Technology• Engineering• Mathematics

Evaluation Questions:• To what extent did

students’ interest in STEM careers increase?

• Did inquiry-based learning increase student engagement?

• How did teachers self efficacy for teaching STEM content change?

Data in Evaluations

“Qualitative data are

measurements that cannot be measured on a

numerical scale; they can only be

classified into one of a group of

categories.”

Qualitative data are analyzed for

patterns or themes.

There are many sources of qualitative data; common in education evaluation:• Interviews and focus groups

with teachers• Interviews and focus group with

students• Open-ended questions on

surveys or questionnaires• Open-ended assessments• Portfolios of student work or

other performance artifacts• Open-ended classroom

observation notes• Journals, logs or other artifacts

of project activities

Data in Evaluations“Quantitative

data are measurements

that are recorded on a

naturally occurring

numerical scale.”

Quantitative data are analyzed

using descriptive or

inferential statistics.

There are many sources of quantitative data;

common in education evaluation:

• Demographics• Grade-level information• Years of teaching experience• Standardized assessment scores• Scaled questions on surveys • Scale-scored classroom

observations• Graduation rates• Rates of course-taking or

course completionMost common and straightforward type of statistics – absolute numbers, percentages, averages, etc.

Data in Evaluations

Beware of common data traps!• Biting off more than you can chew• Not collecting data needed to answer important questions• Collecting data that is not really useful• Neglecting hard-to-quantify data• Not formalizing “informal data” (e.g.,

anecdotes, unrecorded observations)• Not using valuable data after it has been

collected

25

Small Group Activity

1. With a partner(s) at your table, select either the evaluation questions you developed or 4-5 questions shared during report out.

2. Brainstorm 1-3 data sources and collection strategies (how & when) you could use to answer each question.

• Examples:• How many teachers attended the PD regularly?

Attendance counts collected after each PD session throughout the entire evaluation time period.

• How do participants rate the PD sessions? Interviews with teachers collected once or twice throughout the entire evaluation period. Feedback forms administered and collected after each evaluation. Observations of each session.

Whole group share-out.

STEM Evaluation Data Sources

STEM Evaluation Data Sources

Q & A

Jeni Corn [email protected] Townsend tricia_townsend@ncs

u.eduAlana Unfried [email protected]

Thank You

mailto:[email protected]





Documents

Evaluating Your STEM Outreach Program