Dr. AlbrechtResearch TeamEXAMPLE of EVALUATION RESEARCH

SERVICE LEARNING2007 - 2009

Dr. Carol Albrecht was hired by the State of Texas Service Learning Center

18 public school districts in the state of Texas were involved in Service Learning Projects

We met with them to identify how the program impacted public school children, teachers, community partners and parents of students

ABOUT THE PROJECT

THE PROCESS

One

• Identified the Objectives, Conceptualized Variables and Developed Indicators.

Two

• Selected the Sample.

Three

• Constructed the Indicators

Four

• Selected a Research Design.

Five

• Collected and Analyzed the Data.

Six

• Wrote the Report and Presented the Findings

STEP #1

One

• Identified the Objectives, Conceptualized Variables and Developed Indicators.

Two

• Selected the Sample.

Three

• Constructed the Indicators

Four

• Selected a Research Design.

Five

• Collected and Analyzed the Data.

Six

• Wrote the Report and Presented the Findings

Does Service Learning…

Have a positive

impact on students?

Have the support of the

teachers?

Have the support of the

community, parents

& partners?

When, where

and why is it

effective?

IDENTIFY THE OBJECTIVES

Development of Small Group

Leadership SkillsIncrease in a

Positive Affect Toward School

Development of Problem Solving

SkillsDevelopment of Post-High School

Aspirations

IDENTIFY THE OUTCOMES

Concept

Leadership Skills

Liking for School

Problem Solving Skills

Post High School Aspirations

IndicatorStudents’ Perceptions

Measured by Index Questions on Pre and Post

Test

Students’ Perceptions as Measured by Index

Questions on Pre and Post Test

Students’ Perceptions as Measured by Index

Questions on Pre and Post Test

Students’ Perceptions as Measured by Index

Questions on Pre and Post Test

DEVELOP INDICATORS

STEP#2One

• Identified the Objectives, Conceptualized Variables and Developed Indicators.

Two

• Selected the Sample.

Three

• Constructed the Indicators.

Four

• Selected the Research Design.

Five

• Collected and Analyzed the Data.

Six

• Wrote the Report and Presented Findings.

3500 high school students

2500 elementary students

200 of their teachers and 18 of the service learning

coordinators

20 parents and 30 community partners

SELECT A SAMPLE

IDENTIFY DIVERSE SCHOOLS

Middle/High SchoolRaising money for various causes

Tutoring Elementary children

Landscaping Parks

Building playground equipment

Painting murals on boy/girl club walls

SERVICE LEARNING PROJECTS

One

• Identified the Objectives, Conceptualized Variables and Developed Indicators.

Two

• Selected the Sample.

Three

• Constructed the Indicators

Four

• Determined Research Design. Constructed Instrument.

Five

• Collected and Analyzed the Data.

Six

• Wrote the Report and Presented Data .

STEP #3

USE A SURVEY TO DEVELOP INDICATORS

Students report more confidence in

leadership skills after participation in Service Learning

Students report more confidence in

problem solving skills after

participation in Service Learning

Students report greater affinity for teachers and school after participation in Service Learning

Students report greater desire to

pursue post secondary

education after participation in

Service Learning

RELATIONSHIP BETWEEN SUCCESS AND LEVELS OF SERVICE LEARNING

PARTICIPATION

Amount of participation is

positively related to confidence in leadership skills

Amount of participation is

positively related to confidence in problem solving

skills.

Amount of participation if

positively related to affinity for teachers and

school

Amount of participation is

positively related to post secondary

aspirations

Development of Leadership Skills

Development of Problem Solving Skills

Increased Affect Toward School

Increased Academic Aspirations

Positive Affect Toward Program

HOW THE STUDENTS FELT ABOUT PARTICIPATING IN THE PROGRAM

One

• Identified the Objectives, Conceptualized Variables and Developed Indicators.

Two

• Selected the Sample.

Three

• Constructed the Indicators.

Four

• Selected the Research Design.

Five

• Collected and Analyzed the Data.

Six

• Wrote the Report and Presented Findings.

STEP #4

We decided to use multi-methods to increase the validity and reliability of our fi ndings.

Used Experimental Design to evaluate the program and test our hypotheses Pre-test survey Intervention Post-test survey

Used Focus Groups In addition we used focus groups to gain additional

information/insights and further test the validity/reliability of our survey.

SELECT RESEARCH DESIGN

One

• Identified the Objectives, Conceptualized Variables and Developed Indicators.

Two

• Selected the Sample.

Three

• Constructed the Indicators.

Four

• Selected the Research Design.

Five

• Collected and Analyzed the Data.

Six

• Wrote the Report and Presented the Findings.

STEP #5

Texas A&M students went all over Texas and administered the pre-test

Five months later they returned to administer the post-test

12 schools were selected for the focus groups They were recorded by the students They were later transcribed A coding schema was developed to compare and contrast

answers across schools

ADMINISTERING THE TEST

One

• Identified the Objectives, Conceptualized Variables and Developed Indicators.

Two

• Selected the Sample.

Three

• Constructed the Indicators.

Four

• Selected the Research Design.

Five

• Collected and Analyzed the Data.

Six

• Wrote the Report and Presented the Findings.

STEP #6

WRITING THE REPORT

We wrote a fi nal report and prepared a power point presentation for the teachers and administrators.

It was important to interpret the information accurately even though it was not what the Texas Center for Service Learning was hoping to hear. In summary, we found very small and sometimes

even diff erences comparing the pre- and post-tests. Most of the

diff erence were NOT statistically significant. These findings were consistent across two years

However, we had collected data that allowed us to do a more in-depth analysis, and so we proceeded.

We now looked at the quantity and quality of the experience more specifically – and how that influenced outcomes. To measure quantity of Service Learning, we measured

how many total hours students had spent in service learning that year.

To measure quality, we measured sense of ownership

Did students help plan the Service Learning Project?

Did students make decisions about the Service Learning Project?

EVALUATION

Mean Score on Outcome Measurements by Amount of Student Planned or Participated in Project

Students Planned -Four Hours or More

Yes No

Attitudes Toward Attending CollegeAttitudes Toward SchoolProblem Solving SkillsLeadership Skills

12.81***19.84***17.01***17.15***

12.50***18.45***15.58***15.85***

Students Participated - Four Hours or More

Yes No

Attitudes Toward Attending CollegeAttitudes Toward SchoolProblem Solving SkillsLeadership Skills

12.89***19.54***16.91***17.04***

12.18***18.38***15.09***15.46***

***p<.0001

* p< .05 **p< 0.01 ***p< 0.0001

Mean Scores on Outcome Measurements by Sense of Ownership – Whether or Not Students Made Decisions

Students Made Decisions

Yes No

Attitudes Toward CollegeAttitudes Toward SchoolProblem Solving SkillsLeadership Skills

13.04***19.59***

17.26***p17.51***

12.36***18.84***15.58***15.81***

* p< .05 **p< 0.01 ***p< 0.0001

CHART 1. High School Students’ Perception of How Good They are at Speaking in Front of Groups by Whether or Not They Made Decisions about Service Learning Projects

(p <0.0001)

CHART 2. High School Students’ Perception of How Good They are at Finding Resources by Whether or Not They Made Decisions about Service Learning Projects

11%7%

44%

13%7%

28%24%

9%

43%

17%

0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%

100.00%

StronglyDisagree

Disagree Undecided Agree Strongly Agree

Did

Did Not

(p <0.0001)

Even though I had done research for

many years, I learned and “re” learned some important things about

evaluation research from this project.

Click here to link to power point outlining the things we learned about:1.The Impact that Timing can have on Results.2.Asking the “Right” Questions and Selecting the

“Right” Indicators.3.The Impact of Social Desirability on Findings and

how to Deal with it. 4.The Importance of using Multi-Methods when doing Evaluation Research.