Noyce Evaluation

University of MinnesotaApril 20, 2006

Jim AppletonMarjorie Bullitt Bequette

Frances LawrenzAnn Ooms

Deena Wassenberg

Technical assistance: David Ernst

Overall goals for our project To contribute to the knowledge base about

effective strategies for attracting and retaining high quality STEM teachers

To collaboratively develop a plan to evaluate the Noyce Program that will document overall program accomplishments while celebrating the uniqueness of each project

To conduct the evaluation and disseminate findings in a utility-oriented fashion

Our responsibilities We are:

Collecting and categorizing evaluation plans and instruments

Conducting a comprehensive review of the STEM recruitment and retention literature

Working with ORC MACRO to make effective use of their data

We will: Work with all the projects to design a program

evaluation through virtual and face-to-face meetings Conduct the evaluation Disseminate the results in a user-friendly fashion

We need you to be effective

We need your help to: Refine our literature data base Optimize the effectiveness of the

evaluation variables and instrument data bases

Plan and conduct the program level evaluation

Plan for this two session introductory conference

Showcase our materials and explain how we think they might be useful

Obtain feedback on how to improve Discuss what might be useful in an

overall evaluation of the Noyce Program Determine the most effective use of the

evaluation time at the PI conference

Outline of today’s presentation

A tour of the Web site Demo of the literature data base Summary of the literature findings Logic Model Evaluation variables and instruments Setting the stage for tomorrow’s

discussion

Q & A

Questions and Answers about the Introduction

A tour of the Web site

Q&A

Questions and Answers about the Web site overall

Q&A

Questions and Answers about using the Literature Data Base

Ongoing review of the R&R literature Looking for factors that affect recruitment and

retention Chose empirical articles from our database that

(based on abstracts) had significant results on factors affecting recruitment and retention

Starting with larger N, quantitative work; integration of other studies will follow

We summarized recent articles and used RAND (2004) summaries of older work

Factors were grouped into larger categories Our more detailed summary will be posted

Literature examined so far Adams, 1996 Arnold, Choy, & Bobbitt,

1993 Baker, 1988   Ballou, 1996 Ballou & Podgursky, 1997 Brewer, 1996 Bempah, Kaylen, Osburn,

& Birkenholz, 1994 Boe, Bobbitt, Cook,

Whitener, & Weber, 1997 Bond, 2001 Carroll, Reichardt,

Guarino, & Mejia, 2000 Darling-Hammond,

Chung, & Frelow, 2002 Eberhard, Reinhardt-

Mondragon, & Stottlemyer, 2000

Galchus, 1994 Hall, Pearson, & Carroll,

1992 Hansen Lien, Cavalluzzo,

& Wenger, 2004

Hanushek, Kain, & Rivkin, 2001

Hanushek & Pace, 1995 Henke, Geis, Giambattista, &

Knepper, 1996 Henke, Zahn, & Carroll, 2001 Hounshell & Griffin, 1989 Ingersoll, 2001 Ingersoll, 2003 Ingersoll & Kralik, 2004 Jacobson, 1988 Kirby, Berends, Naftel, 1999 Kirby & Grissmer, 1993 Loeb (2000) Lankford, Loeb, Wyckoff, 2002 Marso & Pigge, 1997 Miech & Elder, 1996 Mont & Rees, 1996 Murnane, Singer, Willett,

Kemple, & Olsen, 1991 Odell & Ferraro, 1992 Pigge, 1985 Plecki, Elfers, Loeb, Zahir, &

Knapp, 2005

Rickman & Parker, 1990 Seyfarth & Bost, 1986 Shen, 1997 Shen (Autumn, 1997) Shen, 1998 Shen, 1999 Shin, 1994 Shin, 1995 Shugart and Hounsell,

1995   Stinebrickner, 2001a Stinebrickner, 2001b Stinebickner 2002 Stockard & Lehman, 2004 Theobald, 1990 Tran,

Young, Mathison, & Hahn, 2000 Villar & Strong, 2005

Weiss, 1999 Young, Place, Rinehart,

Jury, & Baits, 1997

What research has shown to affect retention Characteristics of teachers:

Race/ethnicity Gender Experience Age Type of training program Area taught Academic ability/achievement Family and fertility choices Reasons for choosing to teach Certainty of intention to teach

What research has shown to affect retention Teacher preparation program

characteristics Most of the large N quantitative work that

we’ve examined focuses on the type of program (alternative, master’s 5th year, major in education or in a discipline), not program components.

Some studies examine the effects of course requirements generally.

Both program type AND program components matter, though.

What research has shown to affect retention Mentoring and induction programs

Again, details on what helps are underexamined.

Salary Pay affects retention, interacting with

gender, race/ethnicity, other local salaries and conditions, subject taught, potential for advancement (and salaries for those positions), and salary scale/highest salary.

What research has shown to affect retention School and district setting

“School culture” Race/ethnicity of students; also distribution

of race/ethnicity Student ability Student SES; also distribution of SES School size Number of classes taught Classes taught in area of specialization Spending (amount and patterns) Incidence of crime/violence

Putting this all together:

Tracking teacher characteristics (affective as well as demographic), program and mentoring experience (and the connections between those two), salary, district conditions, and more, can help each project improve and can help all projects learn from each other.

Q&A

Questions and Answers about what the literature has shown

Theoretical Framework: LOGIC MODEL DESCRIPTION

Our proposed Noyce Logic Model contains our efforts to delineate several perspectives: The Noyce Program Ideal

Depicted by the main path as well as bold headings preceded by addition signs (e.g., “+Plan to teach”)

Decision points en route to becoming a STEM teacher Indicated by diverging routes from the main path describing

alternative options and the Noyce Ideal in bold headings Dashed boxes denote retention/recruitment by school or

program Important STEM major decision factors

Influenced by attributes of the candidate, pre-service program, and school/district (depicted as bulleted lists on the main path)

Depicted as thought bubbles emerging from the decision point

Theoretical Framework: LOGIC MODEL DESCRIPTION

The Noyce Program Ideal: Diverse and smart STEM majors will be enticed by

scholarships and stipends to enter pre-service programs

Programs will provide adequate and relevant training These STEM majors will graduate, begin teaching in

their field and at high need schools, and fulfill the obligations of their scholarship/stipend.

These new teachers will continue to teach at high-need schools beyond the obligation period?

Theoretical Framework: LOGIC MODEL DESCRIPTION

Decision Points En Route to Becoming a STEM Teacher:

STEM majors may: Plan to teach or plan for a non-teaching STEM

career If planning to teach, either enter a certification

program or teach without certification If entering a program, upon graduation decide to

teach or to not teach If choosing to teach, decide if it will be at a low- or

high-need school If at a high-need school, decide whether to remain

over time.

Theoretical Framework: LOGIC MODEL DESCRIPTION

Important STEM Major Decision Factors along These Paths: Interests, career values, career pay and

importance of monetary compensation, importance of certification, challenge of financial costs, desire and requirement to teach

What value in workplace, social justice beliefs, program/funding requirements, training, fulfillment of job, perception of support and appropriate level of challenge

Theoretical Framework: LOGIC MODEL GRAPHIC

Q&A

Questions and Answers about the logic model

Project Evaluation Resources

Project evaluation variables, methods and instruments Collected from your evaluation

plans – Thank You! Categorize these based on the kind

of information collected and how it was collected

Categorize and present any specific evaluation instruments you provided (most now available on our Web site)

What you are doing:

Of those responding (41 of 65, 63%), 92.7% of Noyce programs submitted a detailed evaluation plan to us.

We’ve categorized these into: Evaluation of the program itself Evaluation of post-program activity Evaluation methods

What you are doing: VARIABLES

What you are doing: VARIABLES

What you are doing: VARIABLES

What you are doing: VARIABLES

What you are doing:THE PROGRAM ITSELF

Program Recruitment (61.0%) Noyce Student Performance in

Program (61.0%) Demographics (58.5%) Program Retention (24.4%)

What you are doing: POST-PROGRAM MONITORING

Noyce Teacher Effectiveness (63.4%) Monitoring of Noyce Teachers (61.0%) School/District Retention (41.5%) Transition Experiences/Support for Teachers

(39.0%) Coordination Between Programs or Institutions

(31.7%) Fulfillment of Scholarship Requirements (29.3%) School/District Recruitment (22.0%) School/District Characteristics (14.6%) Teaching Assignment Characteristics (2.4%)

What you are doing: WAYS OF GATHERING DATA

Interpret carefully: These were only used if the documents, observations, self-report data could not be better classified elsewhere.

Formative Program Effectiveness Data (63.4%) Summative Program Effectiveness Data (61.0%) Self-Report Data (61.0%) Specific Analyses or Methodologies (56.1%) Research Questions/Evaluation Goals (34.1%) Observations (19.5%) Document Analysis (4.9%)

What you are doing: SUMMARY Program itself: many are gathering data on

recruitment, demographics, and student performance; fewer on program retention

Post-program monitoring: several to many are gathering data on teacher effectiveness, monitoring/fulfillment of scholarship requirements, school/district retention, teacher transition experiences/support, and inter-program/institution coordination; fewer on school/district recruitment and characteristics (including teaching assignment characteristics)

Ways of gathering data: mostly formative and summative program effectiveness, and self-report (caution)

What you are doing: INSTRUMENT DESCRIPTIONS

What you are doing: INSTRUMENTS

Q&A

Questions and Answers about the project evaluation resources

Involvement oriented evaluation: We need you!

Evaluations should be designed to document the context and the full range of effectiveness

Involvement-oriented evaluations provide informed objectivity relationship to site goals and context exemplary designs motivation to provide data more use of the evaluation

What we know already

You are doing a great deal of evaluation

This work is varied and addresses many different stages of the progress of an aspiring teacher

Through coordinating and sharing this work, evaluations (individual and overall program) can be improved

Planning for Friday Discussion: Resources

What are the valuable components of the existing resources?

How could the existing resources be improved?

What additional resources could be provided?

Planning for Friday Discussion: Evaluation What might be important questions for

the evaluation to address? What should we emphasize in the

evaluation? From whom should data be gathered? What sort of data gathering methods

would be most appropriate? What data/report dissemination

strategies would be most useful?

Planning for Friday Discussion: PI conference What overall Program evaluation

issues should be addressed at the PI meeting in June?

What presentation format should be used?

What outcomes should we expect? What “deliverables” should we

hope to obtain?

Planning for Friday Discussion: Participation Think about the questions (these are now

posted on our Web site) Feel free to email before the session begins (

lawrenz@umn.edu) if you like, to send a question to be shared or to arrange a time to appear onscreen

During the discussion you can e-mail through chat to have your comments read by presenter

You can also e-mail at any time through chat to be scheduled to join the discussion verbally (and in person if you have a camera)

Please feel free to raise any other questions or issues that you feel are important

Q & A

Questions and Answers about Involvement Oriented Evaluation or Friday’s Presentation

See you tomorrow!

3:30-5:00 Eastern 2:30-4:00 Central 1:30-3:00 Mountain 12:30-2:00 Pacific