Embed Size (px)
Citation preview
Assessment and Evaluation Plan for a Noyce Scholarship Project
Gerunda B. Hughes, Ph.D.Howard UniversityRobert S. Noyce Teacher Scholarship Program WorkshopNew Orleans, LADecember 11, 2010
What Will Be Covered
Definitions of Terms
Benefits of Formative Evaluation
Elements of a Good Assessment and Evaluation Plan
A Sample Template for a Noyce Teacher Scholarship Program Evaluation Plan
An Evaluation Logic Model for a Noyce Teacher Scholarship Project
Questions or Comments???
2
Definitions of Terms
Assessment – the process of obtaining information that is used to make decisions about programs, projects, participants, etc.
Evaluation – the process of making a value judgment about the quality or worth of components or outcomes of a program given the context under which the project is/was implemented.
Formative/Process Evaluation
Summative Evaluation
3
Definitions of Terms
Formative/Process Evaluation
Provides feedback on the project while it is being conducted (i.e., evaluates the degree or fidelity of implementation)
Is concerned with the activities of the project and confirms that outputs have been achieved
Summative Evaluation
Provides evidence that the project fulfilled that which was originally proposed
Is concerned with the stated objectives of the project and confirms that outcomes have been achieved
4
Benefits of Formative Evaluation
If you have a well-implemented formative evaluation plan, intended results from your summative evaluation are (almost) guaranteed.
If you received the following feedback about your project activities from the formative evaluation, how would you use the information to improve project outputs and outcomes?
5
Formative Evaluation: Recruitment
Proposed Activity 1: Recruit STEM majors using electronic media.
Feedback: For Year 1 and Year 2, however, your were able to recruit only 8 and 12, respectively.
Use of Information:
Expand type of media strategies for recruitment
Increase frequency of recruitment activities
6
Formative Evaluation: Curriculum Development
Proposed Activity 2: Development of new curriculum for mathematics minors by end of Year 1.
Feedback: Development of curriculum completed in Year 3.
Use of Information:
Improve collaboration among and between STEM and Education faculty
Next time, apply for a planning grant to develop curriculum
7
Formative Evaluation: Test Preparation Sessions
Proposed Activity 3: Mandatory weekly test preparation sessions for Praxis I and II
Feedback: At first two sessions, only half of the STEM majors/teacher candidates attended
Use of information:
Do not tolerate absences; explain consequences and follow through
Coordinate test prep schedules with participant schedules
Eliminate conflicts with other project activities
8
Formative Evaluation: Monthly Meetings
Proposed Activity 4: Monthly meetings with STEM and Education faculty and P-12 teachers and principals
Feedback: Sporadic meeting attendance by members. P-12 collaborators attend when they can “get away.”
Use of Information:
Send meeting reminders
Rotate location of meetings to accommodate schedules
Consider holding meetings every other month9
Elements of an Assessment and Evaluation Plan
The Assessment and Evaluation (A & E) Plan
The A & E Plan should be developed during the planning of the project – during the proposal phase.
The A & E Plan should not be added to the proposal as an afterthought.
The A & E Plan shows relationships among the components of the project and serves as a management tool (See Evaluation Logic Model).
Every project should have a formative evaluation plan and a summative evaluation plan.
11
Elements of an Assessment and Evaluation Plan
Evaluation Question(s)
What do we want to know?
Indicators
How will we know it?
Timeline
When will data be collected?
Data Sources
From whom or from what will the data be collected?
Methodology
How will the data be collected?
12
Elements of an Assessment and Evaluation Plan
Assessment Instruments
What measures will be used to collect the data?
Methods of Analysis
What quantitative or qualitative methods of analyses will be used?
13
Elements of an Assessment and Evaluation Plan
Interpretation of Results
What do the results mean?
Were the objectives “met” or “not met”?
If met, how well were the objectives met?
If not, what components of the project failed?
Dissemination of Results
How will the results be communicated? To whom, when, and where?
14
Sample Evaluation Plan for a Noyce Teacher Scholarship Project
Evaluation Question
What do we want to know?
Are the academic programs supported by the Noyce Teacher Scholarship program producing an increase in the number of teaching credentialed STEM majors?
Indicators
How will we know it?
Passing performance rates on Praxis I and II by project participants
Levels of satisfaction with academic program elements among participants
Participant rate of persistence in the Noyce Teacher Scholarship Program
Number of participants who earned a teaching credential
15
Sample Evaluation Plan for a Noyce Teacher Scholarship Project
Timeline
When will data be collected?
During the recruitment phase [Praxis I and Praxis II (Content)]; after taking the professional education courses and clinical experiences in the field [Praxis II (Pedagogy)].
Data Sources
From whom or from what will the data be collected?
Project Participants/Candidates, STEM and Education faculty, Teachers in partnering, high-need schools
Project meetings agenda/minutes
Curriculum Committee meetings agenda/minutes
16
Sample Evaluation Plan for a Noyce Teacher Scholarship Project
Methodology/Assessment Instruments
How will the data be collected? What measures will be used?
Tests, questionnaires, surveys, interviews, observations, focus groups, document review, case studies, blogs, portfolios, etc.
Praxis I, Praxis II, Classroom Observation protocol
Methods of Analysis
What quantitative or qualitative methods of analyses will be used?
Descriptive or inferential statistics; Common theme identification in unstructured data, etc.
17
Sample Evaluation Plan for a Noyce Teacher Scholarship Project
Interpretation of Results
What do the results mean?
Was there an increase in the number of teaching credentialed STEM majors?
Were the objectives “met” or “not met”?
Yes or No
If met, how well were the objectives met?
If Yes: To what degree? -- “Met expectations”; “Exceeded expectations”
If not met, why not?
If No: What project components failed to deliver intended outputs?
18
Sample Evaluation Plan for a Noyce Teacher Scholarship Project
Dissemination of Results
How will the results be communicated?
Interim reports; formative evaluation reports; scheduled project meetings; newsletters; journal articles; conference presentations, project website, summative evaluation report, etc.
To whom?
Project managers (PI and Co-PIs), participants, other primary stakeholders (faculty, teachers, etc.); service providers; funding agency
When?
During the implementation of the project – after the first 6 months? Year 1? Year 2? After Year 3 summer program?; At the conclusion of the project
19
A Sample Template for a Noyce Teacher Scholarship Project Evaluation Plan
The following sample template contains two evaluation questions and important components of the evaluation plan for each question.
An Evaluation Logic Model for a Noyce Teacher Scholarship Project
The following example shows the components of a simplified version of a logic model. It does not contain the “outputs” that result from activities nor does it contain “intermediate outcomes”.
22
NTSP Logic Model
Inputs
Activities
Short-term Outcomes
Long-Term Outcomes
23
NTSP Logic Model: Inputs
Inputs
Resources
NSF Funding
Expertise of Faculty
Institutional Support and Infrastructure
Quality of STEM Recruits
Collaborations within and across organizations
STEM and Education Faculty
School Administrators and Teachers
University and School District Personnel
Local and National Professional Organizations
National Accrediting Organizations
Implementation
Formative and Summative Evaluation
Program Sustainability
24
NTSP Logic Model: Activities
Activities
Recruit undergraduate STEM majors
Create a curriculum that prepares STEM majors for classroom experiences
Design field and clinical experiences
Design and implement STEM teacher learning communities
Create a mentoring program for participants
Secure support from school districts and high need schools
Schedule monthly meeting for primary collaborators
Disseminate findings
25
NTSP Logic Model: Short-Term Outcomes
Short-Term Outcomes
Increased numbers of undergraduate STEM majors with teaching credentials
Improved STEM teacher education program(s)
Improved collaborations between university STEM and Education faculty
Improved collaborations between local university and school district personnel
Enhanced literature and knowledge base on effective teaching in high needs school districts
26
NTSP Logic Model: Long-Term Outcomes
Long-Term Outcomes
Increase in the number of STEM teachers in P- 12 schools nationwide
Transformation and strengthening of teacher education programs for all majors (sustainability)
Improve STEM teaching and student learning for all
Increase in the number and diversity of STEM professionals (scientists, engineers, etc.) nationwide over time
27
An Evaluation Logic Model for a Noyce Teacher Scholarship Project
The following diagram is a simplified version of a logic model for a Noyce Teacher Scholarship project. “Outputs” and “intermediate outcomes” are omitted.
28
