Upload
malcolm-boyd
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Evaluation of Efforts to Broaden STEM Participation:
Results from A Two-Day Workshop
Planning Committee:
Bernice AndersonElmima Johnson
Beatriz Chu ClewellNorman Fortenberry
Presenters:
Patricia B. Campbell & Veronica Thomas
Evaluation of Efforts to Broaden STEM Participation: Workshop
GoalsTo develop and validate a strategy by which to demonstrate the value of NSF's investment in broadening participation (BP).
To negotiate answers to two questions:1. What metrics should be used for project
monitoring? 2. What designs and indicators should be
used for program evaluation?
Evaluation of Efforts to Broaden STEM Participation: The Workshop
ReportThe Policy Context for NSF Programs for
Broadening Participation (Fortenbury)
Measuring Success and Effectiveness in NSF’s Broadening Participation Programs (Clewell)
Outcomes and Indicators Related to Broadening Participation (Campbell, Thomas, & Stoll)
Evaluating Efforts to Broaden Participation (Campbell, Stoll, & Thomas)
Implications of the NSF Broader Impacts Statement (Nelson & Bramwell)
The Policy Context: Historically
NSF’s goal of broadening participation has been shaped through a variety of policy actions by the legislative and executive branches of government.
Within the agency itself, policies articulated by the National Science Board (NSB) and the Committee on Equal Opportunity in Science and Engineering (CEOSE) have informed the NSF approach and strategy to address this goal, as referenced in major policy documents issued by NSF.
The Policy Context: Currently
Broadening Participation at the National Science Foundation: A Framework for Action (May 2008), outlines the NSF-wide broadening participation plan.
It provides guidelines for broadening participation both externally and internally, through:• Expanding the reviewer pool• Training NSF staff and reviewers• Enforcing accountability for NSF staff and
principal investigators• Communicating promising practices• Maintaining and monitoring a portfolio of
relevant programs
The Policy Context: A Core Value and A Strategic Goal
Broadly Inclusive: seeking and accommodating contributions from all sources while reaching out especially to groups that have been under-represented; serving scientists, engineers, educators, students, and the public across the nation; and exploring every opportunity for partnerships, both nationally and internationally. Investing in America's Future: Strategic Plan FY 2006-2011, National Science Foundation, NSF 06-48, National Science Foundation, Arlington, VA, 2006.
Measuring Success: NSF – BP Programs
Broadening Participation Focused Programs (28 Programs; 17 Require Evaluations) Programs with Emphasis on Broadening Participation (17 Programs; 8 Require Evaluation) Programs with Broadening Participation Potential (16 Programs; 9 Require Evaluation) Other Broadening Participation Efforts (5 Programs).
Measuring Success: Suggested Monitoring Metrics
Institution Focused Targeted Programs
Goal: Increase research capability and teaching effectiveness
Baseline data: Collaborative relationships, Funding distribution, % URM students, Total enrollment
Follow-up: Collaborative relationships established, Funding support obtained, Teaching reforms effected
Measuring Success: Recommendations
It is recommended that the NSF:•Conduct periodic evaluations, including external reviews ranging from the program level to larger cross-sections of the portfolio•Develop a common framework requiring that BP projects collect uniform data •Review all funded programs to determine:
If program funds serve a representative proportion of members of under-represented groups or institutions;
If positive outcomes of programs are distributed equitably among all groups of participants or institutions.
Broadening Participation (BP) : Critical Issues Related to Indicators and Outcomes
• Developing shared understanding and clarifying meaning
• Addressing ‘‘success” at multiple levels
Inputs
Resources, contributions, investments that gointo the project
Input indicators measure resources,contributions and investments such as:
Staff Volunteers Funding Materials Facilities Investments made to support BP
Outputs
Units of services and goods provided by the project
Output indicators measure things such as the scope/size of activities, services, events, and products reachingunderrepresented
Numbers of students served Numbers of workshops
Process
Ways in which project services andgoods are provided
Process indicators measures extent towhich BP projects, programs, andstrategies delivered as intended(alignment)
Outcomes
Things project hopes to achieve; actualbenefits, impact, or changes
Outcomes indicators expressed in terms of changes forindividuals, groups, communities, institutions, and system :
Knowledge, attitude, and skill changes Behavior changes Value changes Policy, procedural, and practice changes
Considering BP Success at Multiple Levels
Level 1: Having access to the benefits of STEM knowledge
Level 2: Having access to STEM knowledge
Level 3: Studying STEM
Level 4: Working in STEM areas
Level 5: Generating STEM knowledge
Problems in Determining “Success”
Defining in terms of increase in absolute number or
percentage
Defining in terms of increase in both number and percentage
Defining in terms of the end point being “parity” (absolute number)
Other Considerations in Defining Success
Defining “parity” as a range
Achieving parity, as more participate overall
Considering discipline/field size to which definition of success apply
Integrating qualitative indicators (e.g., broadening and transforming perspectives)
Other Indicators of Success Broadening Participation
Individual level indicators
Institutional level indicators
Foundation level indicators
Individual (Student) Level Indicators
Participation
Retention, persistence, and success
Experiences
Attitudes
Institutional Level Indicators
• Staffing
• Policies, programs, and institutional commitment
• Accountability and rewards
• Monitoring, tracking, and using data for improvement
• Collaborations
Foundation Level Indicators
• Inclusion of information about importance of BP
• Review and monitoring of foundation policies/practices in terms of potential to broaden participation
• Diversity of professional involved with the Foundation
• Foundation resources devoted to BP
• Improvements to knowledge base about broadening participation
• Implementation of effective strategies at Foundation level to BP
Evaluating BP: Research vs. Evaluation
GoalResearch: To move the knowledge base forward.Evaluation: To assess quality/effectiveness.
OutcomeResearch: Why something does or doesn’t.Evaluation: If something does or doesn’t work.
FocusResearch: The research.Evaluation: The program/intervention.
Designs/measures/analysis: No difference
Evaluating BP: Longitudinal Tracking
Being able to follow students longitudinally is the key to any sophisticated understanding of how colleges are doing and what's happening to students. - Thomas R. Bailey, 2008 Without longitudinal data, the generation and testing of causal models tied to successful participation in STEM for diverse populations will be difficult if not impossible.
Evaluating BP: Selecting Designs
The appropriateness of the fit between the design of the program or “intervention” and the requirements of more rigorous evaluation methodologies.
The timing of the evaluation.
The balance between the level of investment in the evaluation and the level of investment in and the intensity of the intervention.
The level of evidence expected given the nature of the intervention.
The strength of rival hypotheses.
Evaluating BP: Selecting Designs
Study Type DesignRepresentati
on
Typical questions
answered by the
design
Quantitativ
e Case
Study
One-shot
Post-test
only
Design
X O
After attending a
preview weekend are at
least 50% of the
students planning to
apply to the institution?
Quasi-
experiment
al Study
One-shot
Pre-test-
Post-test
Design
Oa X Ob
Does working with a role
model increase girls’
interest in science
careers?