Evaluating Broader Impacts · 2018-12-05 · Why Evaluate Broader Impacts? • Two main NSF...

Evaluating Broader Impacts

Community Evaluation and Research Collaborative

Miles McNall & Robert Griffore

Why Evaluate Broader Impacts?• Two main NSF proposal review criteria:

• Intellectual merit• Broader impacts: Benefits to society or advancement

of desired societal outcomes• Each proposal submitted to the NSF must include a

section about its intended broader impacts and a plan for assessing them.

NABI ‐ Broader Impacts Guiding Principles

• Guiding Questions• Are the goals and objectives clearly defined with measurable 

outcomes?• How will the outcomes be measured and who will be conducting 

the measurement?• Are the intended target audience/societal impacts of the 

activities described?

https://broaderimpacts.net/wp-content/uploads/2016/05/nabi_guiding_principles.pdf

Resources for Broader Impacts

https://engage.msu.edu/

http://coseenow.net/wizard/

Build a Broader Impacts Plan

• COSEE NOW Broader Impacts Wizard• Step 1: Audience• Step 2: Budget• Step 3: Activity• Step 4: Project Description• Step 5: Evaluation

• Example plan:• http://coseenow.net/wizard/files/BI%20Wizard%20Example%20Summary.pdf

Example BI Plan

Example Plan Evaluation Section

EvaluationTo measure the success of this project, I will also perform the following assessments with the help of an external evaluator:• Formative evaluation: Conducted throughout project design 

and development to guide improvements during piloting/prototyping of something new, or to improve an existing program or project.

• Summative evaluation: Conducted at the end to document successes, failures and lessons learned.

Weak! You can do better.

BINDERS

• Online platform available to MSU researchers 

• Supports:• Planning broader impacts activities

• Evaluating broader impacts activities for high school programs, public events, and undergraduate research experiences 

• Submitting grant proposals ‐ Use BINDERS Proposal Narrative

BINDERS Self‐Assessment Rubric for Broadening Participation

• Helps PIs plan projects to broaden the participation of women, persons with disabilities, and underrepresented minorities

• Focused on STEM disciplines• Six rubric dimensions

• Inclusive Stakeholder Involvement• Recruitment and Accessibility• Authentic Participation in STEM• Mentoring and Network Building• STEM Learning Outcomes• Dissemination

BINDERS Proposal Narrative• The BINDERS Proposal Narrative is boilerplate text for use in the 

evaluation portion of grant proposals. It describes what BINDERS is.

BINDERS Surveys• Measure whether formal or informal educational activities are 

achieving the desired goals at different levels of outreach.• Cannot be modified.• Can be administered online using the Survey Dashboard.• Available surveys:

• General Public Outreach• High School Outreach Survey• Undergraduate Research Survey

Areas of Broader Impact

Broader Impacts

• Full participation of women, persons with disabilities, and underrepresented minorities in science, technology, engineering, and mathematics (STEM)

• Improved STEM education and educator development at any level

• Increased public scientific literacy and public engagement with science and technology

• Improved well-being of individuals in society

Broader Impacts

• Development of a diverse, globally competitive STEM workforce

• Increased partnerships among academia, industry, and others• Improved national security• Increased economic competitiveness of the United States• Enhanced infrastructure for research and education

Evaluating Broader ImpactsTerminology

Evaluation

• A systematic investigation of  the merit, worth, or value of something.

Outcomes and Impacts

• Outcomes:• Outcomes are the result of goals being successfully achieved.• They should be measurable and measured.• Outcomes demonstrate changes in awareness, knowledge, skills, attitudes, behavior, motivations, beliefs, values, capacities, or conditions of individuals, groups, organizations, systems, or communities.

• There can be short term, intermediate, and/or long term outcomes

• Impacts:• Benefit(s) within or to the target audience(s)/society due to the BI activity(s) as evidenced by measurable or articulated outcomes.

https://broaderimpacts.net/wp-content/uploads/2016/05/nabi_guiding_principles.pdf

Goals

• The purposes toward which the activity(s) is directed.

• Example Goal for Science Festival: Increased interest in and awareness of careers in science among K‐12 students who are members of groups underrepresented in STEM careers.

BI Logic ModelINPUTS ACTIVITIES OUTPUTS SHORT-TERM

OUTCOMESINTERMEDIATE OUTCOMES

LONG-TERM OUTCOMES/IMPACTS

Resources essential to carry out project activities

What the project does

Immediate products of activities

Changes in awareness, attitudes, knowledge

Changes in behavior

Changes in the status or condition of people or communities

Scientific expertiseScience communication skillsDemonstration supplies and materialsK-12 attendees

Hands-on, engaging demonstration of basic scientific concepts to K-12 students from under-represented groups by scientists

Number of K-12 students from under-represented groups attending the MSU Science Festival

Increased interest in and awareness of careers in science among K-12 students who are members of groups under-represented in STEM careers

Increase in students taking STEM courses in high schoolIncrease in students studying STEM fields in college

Increased number of under-represented groups in STEM careers

OUTPUTS-IMPACT CHAIN

OUTPUTSNumber of K-12 students from

underrepresented groups attending the

Science Festival

SHORT-TERM OUTCOMES

Increased interest in and awareness of careers in science

INTERMEDIATE OUTCOMES

Increase in students taking STEM

courses in high school

Increase in students studying STEM fields in college

LONG-TERM OUTCOMES/

IMPACTSIncreased number of

underrepresented groups in STEM

careers

MEASUREMENT COSTS

$ $$ $$$ $$$$

OUTPUTS-IMPACT CHAIN

OUTPUTSNumber of K-12 students from

underrepresented groups attending the

Science Festival

SHORT-TERM OUTCOMES

Increased interest in and awareness of careers in science

INTERMEDIATE OUTCOMES

Increase in students taking STEM

courses in high school

Increase in students studying STEM fields in college

LONG-TERM OUTCOMES/

IMPACTSIncreased number of

underrepresented groups in STEM

careers

What you might actually measure

EVALUATION PLAN

OUTCOMEOR IMPACT

INFORMATON NEEDED

SAMPLE DESIGN DATACOLLECTION METHOD

DATAANALYSIS

REPORTING• Timing• Format• Audience

Increased interest in and awareness of careers in science among K-12 students who are members of groups under-represented in STEM careers.

Self-reported interest in and awareness of careers in science among K-12 Science Festival attendees

Demographic characteristics of Science Festival attendees (gender, race, ethnicity, etc.)

Systematic sample (every 5th attendee who crosses an intercept point)

Single-group,posttest only

OR

Retrospective pretest-posttest

iPad survey Descriptive statistics

NSF

PublicAudiences

Identify population

Can entire population be included in the

evaluation?

Yes Census

No Sample

Sample vs. Census

Is random (probability) sampling possible?

Yes

Simple Random Sample

Stratified Random Sample

No

•Convenience sampling•Snowball sampling

•Systematic sampling•Purposive sampling

•Quota sampling

Sampling Options

Evaluation Designs

• Designs specify:• Whether there will be an intervention (observational vs. experimental)

• Timing and frequency of measurement (cross-sectional vs. longitudinal)

• Number of groups (treatment vs. control)

• Experimental designs are the gold standard for causal inference, but they are not always feasible or appropriate.

True Experimental Design:Pretest - Posttest Control Group Design

R Treatment O X OR Control O O

Strengths: Controls for everything except pretest effect, including selection bias

R = Random Assignment (NR = Non-Random Assignment)O = ObservationX = Intervention

True Experimental Design:Posttest Only Control Group Design

Strengths:• Controls selection bias• No testing effectWeakness• Cannot measure change over time

R Treatment X OR Control O

No Pretest

Experimental Designs Control Threats to Internal Validity

• Maturation – people change over time• History – Things happen• Mortality – participants drop out• Regression effect – outcomes move toward mean• Selection bias – participant characteristics differ by group• Reactivity – outcomes can be caused by observation• Testing effect – outcomes affected by measurement

Quasi-experimental designs can eliminate some, but not all threats to internal validity

Limitations• No comparison with non-intervention (No counterfactual)• No random assignment• Does not control for any threats to internal validity

Quasi-Experimental DesignOne Group Pretest - Posttest Design

NR Treatment O X O

Quasi-Experimental DesignPretest - Posttest Control Group Design

Strengths:• Controls for several threats to IV (maturation, history, testing)Limitations:• Does not control for pretest effect• No random assignment• Does not control for selection bias

NR Treatment O X ONR Control O O

Data Collection Options

• Surveys • Paper• Phone• Online – computer, tablet, phone

• Interviews• Focus groups• Observations• Secondary data• Document review

http://www.socialbrite.org/author/jd-lasica/

Types of Question Items

• Likert-type – measure degree of agreement with statement

• Multiple choice – select correct or preferred answer

• True-False – assumes an assertion is either true of false

• Rank–order – ranked based on characteristic(s)

• Rating scale – attitude direction and intensity

• Open-ended – no specific response options

Online Resources:Types of questions: http://www.socialresearchmethods.net/kb/questype.php

Likert scale options: https://www.extension.iastate.edu/Documents/ANR/LikertScaleExamplesforSurveys.pdf

Descriptive Statistics• Central tendency

• Mode• Median• Mean

• Dispersion• Range• Percentiles• Quartiles• Variance• Standard deviation

Online Resource: https://www.betterevaluation.org/en/rainbow_framework/describe/analyse_data

https://stats.stackexchange.com/questions/423/what-is-your-favorite-data-analysis-cartoon

Basic Statistical Tests and Hypothesis Testing• Examine variance in dependent variable based on group characteristics

• Independent samples t test

• Paired samples t test

• Analysis of variance

• Factorial analysis of variance

• Examine – associations among variables

• Pearson correlation

• Spearman correlation

• Chi-square and other nonparametric statistics

Online resource: https://www.betterevaluation.org/en/rainbow_framework/describe/analyse_data

EVALUATION PLAN

OUTCOMEOR IMPACT

INFORMATON NEEDED

SAMPLE DESIGN DATACOLLECTION METHOD

DATAANALYSIS

REPORTING• Timing• Format• Audience

Increased interest in and awareness of careers in science among K-12 students who are members of groups under-represented in STEM careers.

Self-reported interest in and awareness of careers in science among K-12 Science Festival attendees

Demographic characteristics of Science Festival attendees (gender, race, ethnicity, etc.)

Systematic sample (every 5th attendee who crosses an intercept point)

Single-group,posttest only

OR

Retrospective pretest-posttest

iPad survey Descriptive statistics

NSF

EVALUATION PLANOUTCOMEOR IMPACT

INFORMATON NEEDED

SAMPLE DESIGN DATACOLLECTION METHOD

DATAANALYSIS

REPORTING• Timing• Format• Audience

The MSU IRB

The Human Research Protection Programhttps://hrpp.msu.edu/

Evaluations are often exempt because they are not considered research, but investigators should always apply for rather than assume exempt status.

Your BI Evaluation Consultants

• Robert Griffore• Griffore@msu.edu

• Miles McNall• mcnall@msu.edu