Embed Size (px)
Citation preview
Evaluating Broader Impacts
Community Evaluation and Research Collaborative
Miles McNall & Robert Griffore
Why Evaluate Broader Impacts?• Two main NSF proposal review criteria:
• Intellectual merit• Broader impacts: Benefits to society or advancement
of desired societal outcomes• Each proposal submitted to the NSF must include a
section about its intended broader impacts and a plan for assessing them.
NABI ‐ Broader Impacts Guiding Principles
• Guiding Questions• Are the goals and objectives clearly defined with measurable
outcomes?• How will the outcomes be measured and who will be conducting
the measurement?• Are the intended target audience/societal impacts of the
activities described?
https://broaderimpacts.net/wp-content/uploads/2016/05/nabi_guiding_principles.pdf
Resources for Broader Impacts
https://engage.msu.edu/
http://coseenow.net/wizard/
Build a Broader Impacts Plan
• COSEE NOW Broader Impacts Wizard• Step 1: Audience• Step 2: Budget• Step 3: Activity• Step 4: Project Description• Step 5: Evaluation
• Example plan:• http://coseenow.net/wizard/files/BI%20Wizard%20Example%20Summary.pdf
Example BI Plan
Example Plan Evaluation Section
EvaluationTo measure the success of this project, I will also perform the following assessments with the help of an external evaluator:• Formative evaluation: Conducted throughout project design
and development to guide improvements during piloting/prototyping of something new, or to improve an existing program or project.
• Summative evaluation: Conducted at the end to document successes, failures and lessons learned.
Weak! You can do better.
BINDERS
• Online platform available to MSU researchers
• Supports:• Planning broader impacts activities
• Evaluating broader impacts activities for high school programs, public events, and undergraduate research experiences
• Submitting grant proposals ‐ Use BINDERS Proposal Narrative
BINDERS Self‐Assessment Rubric for Broadening Participation
• Helps PIs plan projects to broaden the participation of women, persons with disabilities, and underrepresented minorities
• Focused on STEM disciplines• Six rubric dimensions
• Inclusive Stakeholder Involvement• Recruitment and Accessibility• Authentic Participation in STEM• Mentoring and Network Building• STEM Learning Outcomes• Dissemination
BINDERS Proposal Narrative• The BINDERS Proposal Narrative is boilerplate text for use in the
evaluation portion of grant proposals. It describes what BINDERS is.
BINDERS Surveys• Measure whether formal or informal educational activities are
achieving the desired goals at different levels of outreach.• Cannot be modified.• Can be administered online using the Survey Dashboard.• Available surveys:
• General Public Outreach• High School Outreach Survey• Undergraduate Research Survey
Areas of Broader Impact
Broader Impacts
• Full participation of women, persons with disabilities, and underrepresented minorities in science, technology, engineering, and mathematics (STEM)
• Improved STEM education and educator development at any level
• Increased public scientific literacy and public engagement with science and technology
• Improved well-being of individuals in society
Broader Impacts
• Development of a diverse, globally competitive STEM workforce
• Increased partnerships among academia, industry, and others• Improved national security• Increased economic competitiveness of the United States• Enhanced infrastructure for research and education
Evaluating Broader ImpactsTerminology
Evaluation
• A systematic investigation of the merit, worth, or value of something.
Outcomes and Impacts
• Outcomes:• Outcomes are the result of goals being successfully achieved.• They should be measurable and measured.• Outcomes demonstrate changes in awareness, knowledge, skills, attitudes, behavior, motivations, beliefs, values, capacities, or conditions of individuals, groups, organizations, systems, or communities.
• There can be short term, intermediate, and/or long term outcomes
• Impacts:• Benefit(s) within or to the target audience(s)/society due to the BI activity(s) as evidenced by measurable or articulated outcomes.
https://broaderimpacts.net/wp-content/uploads/2016/05/nabi_guiding_principles.pdf
Goals
• The purposes toward which the activity(s) is directed.
• Example Goal for Science Festival: Increased interest in and awareness of careers in science among K‐12 students who are members of groups underrepresented in STEM careers.
BI Logic ModelINPUTS ACTIVITIES OUTPUTS SHORT-TERM
OUTCOMESINTERMEDIATE OUTCOMES
LONG-TERM OUTCOMES/IMPACTS
Resources essential to carry out project activities
What the project does
Immediate products of activities
Changes in awareness, attitudes, knowledge
Changes in behavior
Changes in the status or condition of people or communities
Scientific expertiseScience communication skillsDemonstration supplies and materialsK-12 attendees
Hands-on, engaging demonstration of basic scientific concepts to K-12 students from under-represented groups by scientists
Number of K-12 students from under-represented groups attending the MSU Science Festival
Increased interest in and awareness of careers in science among K-12 students who are members of groups under-represented in STEM careers
Increase in students taking STEM courses in high schoolIncrease in students studying STEM fields in college
Increased number of under-represented groups in STEM careers
OUTPUTS-IMPACT CHAIN
OUTPUTSNumber of K-12 students from
underrepresented groups attending the
Science Festival
SHORT-TERM OUTCOMES
Increased interest in and awareness of careers in science
INTERMEDIATE OUTCOMES
Increase in students taking STEM
courses in high school
Increase in students studying STEM fields in college
LONG-TERM OUTCOMES/
IMPACTSIncreased number of
underrepresented groups in STEM
careers
MEASUREMENT COSTS
$ $$ $$$ $$$$
OUTPUTS-IMPACT CHAIN
OUTPUTSNumber of K-12 students from
underrepresented groups attending the
Science Festival
SHORT-TERM OUTCOMES
Increased interest in and awareness of careers in science
INTERMEDIATE OUTCOMES
Increase in students taking STEM
courses in high school
Increase in students studying STEM fields in college
LONG-TERM OUTCOMES/
IMPACTSIncreased number of
underrepresented groups in STEM
careers
What you might actually measure
EVALUATION PLAN
OUTCOMEOR IMPACT
INFORMATON NEEDED
SAMPLE DESIGN DATACOLLECTION METHOD
DATAANALYSIS
REPORTING• Timing• Format• Audience
Increased interest in and awareness of careers in science among K-12 students who are members of groups under-represented in STEM careers.
Self-reported interest in and awareness of careers in science among K-12 Science Festival attendees
Demographic characteristics of Science Festival attendees (gender, race, ethnicity, etc.)
Systematic sample (every 5th attendee who crosses an intercept point)
Single-group,posttest only
OR
Retrospective pretest-posttest
iPad survey Descriptive statistics
NSF
PublicAudiences
Identify population
Can entire population be included in the
evaluation?
Yes Census
No Sample
Sample vs. Census
Is random (probability) sampling possible?
Yes
Simple Random Sample
Stratified Random Sample
No
•Convenience sampling•Snowball sampling
•Systematic sampling•Purposive sampling
•Quota sampling
Sampling Options
Evaluation Designs
• Designs specify:• Whether there will be an intervention (observational vs. experimental)
• Timing and frequency of measurement (cross-sectional vs. longitudinal)
• Number of groups (treatment vs. control)
• Experimental designs are the gold standard for causal inference, but they are not always feasible or appropriate.
True Experimental Design:Pretest - Posttest Control Group Design
R Treatment O X OR Control O O
Strengths: Controls for everything except pretest effect, including selection bias
R = Random Assignment (NR = Non-Random Assignment)O = ObservationX = Intervention
True Experimental Design:Posttest Only Control Group Design
Strengths:• Controls selection bias• No testing effectWeakness• Cannot measure change over time
R Treatment X OR Control O
No Pretest
Experimental Designs Control Threats to Internal Validity
• Maturation – people change over time• History – Things happen• Mortality – participants drop out• Regression effect – outcomes move toward mean• Selection bias – participant characteristics differ by group• Reactivity – outcomes can be caused by observation• Testing effect – outcomes affected by measurement
Quasi-experimental designs can eliminate some, but not all threats to internal validity
Limitations• No comparison with non-intervention (No counterfactual)• No random assignment• Does not control for any threats to internal validity
Quasi-Experimental DesignOne Group Pretest - Posttest Design
NR Treatment O X O
Quasi-Experimental DesignPretest - Posttest Control Group Design
Strengths:• Controls for several threats to IV (maturation, history, testing)Limitations:• Does not control for pretest effect• No random assignment• Does not control for selection bias
NR Treatment O X ONR Control O O
Data Collection Options
• Surveys • Paper• Phone• Online – computer, tablet, phone
• Interviews• Focus groups• Observations• Secondary data• Document review
http://www.socialbrite.org/author/jd-lasica/
Types of Question Items
• Likert-type – measure degree of agreement with statement
• Multiple choice – select correct or preferred answer
• True-False – assumes an assertion is either true of false
• Rank–order – ranked based on characteristic(s)
• Rating scale – attitude direction and intensity
• Open-ended – no specific response options
Online Resources:Types of questions: http://www.socialresearchmethods.net/kb/questype.php
Likert scale options: https://www.extension.iastate.edu/Documents/ANR/LikertScaleExamplesforSurveys.pdf
Descriptive Statistics• Central tendency
• Mode• Median• Mean
• Dispersion• Range• Percentiles• Quartiles• Variance• Standard deviation
Online Resource: https://www.betterevaluation.org/en/rainbow_framework/describe/analyse_data
https://stats.stackexchange.com/questions/423/what-is-your-favorite-data-analysis-cartoon
Basic Statistical Tests and Hypothesis Testing• Examine variance in dependent variable based on group characteristics
• Independent samples t test
• Paired samples t test
• Analysis of variance
• Factorial analysis of variance
• Examine – associations among variables
• Pearson correlation
• Spearman correlation
• Chi-square and other nonparametric statistics
Online resource: https://www.betterevaluation.org/en/rainbow_framework/describe/analyse_data
EVALUATION PLAN
OUTCOMEOR IMPACT
INFORMATON NEEDED
SAMPLE DESIGN DATACOLLECTION METHOD
DATAANALYSIS
REPORTING• Timing• Format• Audience
Increased interest in and awareness of careers in science among K-12 students who are members of groups under-represented in STEM careers.
Self-reported interest in and awareness of careers in science among K-12 Science Festival attendees
Demographic characteristics of Science Festival attendees (gender, race, ethnicity, etc.)
Systematic sample (every 5th attendee who crosses an intercept point)
Single-group,posttest only
OR
Retrospective pretest-posttest
iPad survey Descriptive statistics
NSF
EVALUATION PLANOUTCOMEOR IMPACT
INFORMATON NEEDED
SAMPLE DESIGN DATACOLLECTION METHOD
DATAANALYSIS
REPORTING• Timing• Format• Audience
The MSU IRB
The Human Research Protection Programhttps://hrpp.msu.edu/
Evaluations are often exempt because they are not considered research, but investigators should always apply for rather than assume exempt status.
Your BI Evaluation Consultants
• Robert Griffore• [email protected]
• Miles McNall• [email protected]
