Data Collection, Publishing, and Dissemination of Results

Eric Grodsky, University of MinnesotaChandra Muller, University of Texas at Austin

Motivation & Overview

The impact of the STEP program and your STEP Type 1 project depend on dissemination of knowledge for application to other settings.

Based on our early experiences working with STEP Type 1 projects

Steps to designing a study, collecting data, publishing and disseminating results: an overview and then example

Study Design

A good study design makes it easier to publish and disseminate results

Begin the design phase early, alongside the program design

Consider collecting baseline (and control group) data before beginning program implementation

Control or Comparison Group

What is the impact of your program on production of STEM majors? To answer this you must be able to estimate: What would have happened without the program? And ideally, what the program did to have the effect, and

for whom? Consider collecting a control group sample while you

are planning the program, before it is implemented Target students for comparison who are similar to those

you will target for the program

IRB: Institutional Review Board

Work with the IRB at your university to gain approval for your study

It is likely that your study will be considered “Exempt”

Example from University of Texas:

Data Collection & Analysis

Study design and response rates are crucial A randomized control trial—where the only

difference between your control and treatment groups is due to random assignment—produces the simplest analysis strategy to show program effects

Dissemination of Findings Begin analyzing your data and publishing early—pay a

writer to write up your results if you and your staff lack time

Early phase results might include: Description of challenges faced by control group Early evidence of climate change in college

As project matures (even at the end of second year phase) results might: Compare control and treatment group progress in STEM courses,

attitudes about STEM, other indicators relevant to your project? Estimate effects on longer range goals of project and the

possible mechanisms through which program works

Where do you publish and disseminate your findings? Publications of field specific education journals (the

education arm of professional organizations, have journals and newsletters, e.g., Journal of Engineering Education) and education journals like Journal of Higher Education, Education and Evaluation Policy Analysis

Early findings and news briefs might go into professional newsletters

Work with your office of public affairs to prepare press releases

Consider an end-of-project book Get ideas from other STEP projects

Designing the study

What do you want to learn? How/why do you believe that the project succeed?

Who will you serve? What is your target population?

How do you recruit sample members and collect data?

Designing the study

What exactly do you want to ask?

What will you do with all of those data once you have them?

What do you want to learn?

Project will implement intervention you and NSF believe will work What are the explicit objectives? What are the interventions? How/why will they work? (theory)

Example: Professor Yu

What do you want to learn?

Study groups

Chemistry degree

?

theory

Study groups

Chemistry degree

Knowledge

Self-confidence

Enjoyment

The target population

Potential chemistry and physics students Latino and African American students1. Recruitment: High school students 2. Tutoring: In selected classes3. Study groups: In selected classes4. Faculty mentor: in selected classes

Who is in the control condition?

Recruitment: Those not recruited? In comparable non-participating schools? Comparable non-participating classes?

Any way you cut it, must include kids in high school

The target population

Latino and African American students1. Recruitment: At selected schools2. Tutoring: In selected classes3. Study groups: In selected classes4. Faculty mentor: in selected classes

Peer tutoring

Target population: Students enrolled in gateway courses

Control group?

Study groups Target population: those enrolling in

gateway courses in chemistry and physics

Study groups

Control: Entrants to same classes prior to project Entrants to gateway courses in biology and

computer science

Random assignment of sections to study groups or not

Designing data collection

Mode of collection

Sampling procedures

Protocol for recruitment

Timing

The IRB

Mode of collection

Survey Interview (phone or in person) Paper and pencil

Mail In class

Online

Focus group

Sampling

Identify target population

Locate/create sampling frame Make sure you have contact data you need

Structure sample Stratification Matching Blocking

Sampling: The Yu study

Target population: Students enrolled in calculus 101, chemistry 101, biology 101 Each course has 2-4 sections and each

section 6-8 lab sections

Sampling: The Yu study

Randomly assign lab sections to have study groups or not starting in spring of 2013 Sample size of 600

200 students in fall of 2012 from all students enrolled in target courses

100 treatment and 200 control students in spring of 2013

Sampling: The Yu study

Threats to valid inference

Recruiting the sample

Sample nonresponse undercuts validity Shoot for at least 70%

Getting and maintaining sample is often the hardest part of the study

Without a good sample the rest of your evaluation efforts are wasted

Recruiting the sample

Incentives for participation There is actual research on this Cash is king, other incentives (Amazon gift

certificate, MP3 downloads, etc.) not very effective

Some evidence for effectiveness of lotteries, but weak

Recruiting the sample

Initial contact There is actual research on this Letter on letterhead in stamped envelope

if possible In Yu’s case, possibly hand-delivered in

lab

Recruiting the sample

Non-response follow-up Email Snail mail Phone

Data collection timing

Consider last date you can collect data and work backwards One week prior to exams

Snail mail incentive letter 2 email follow ups Snail mail follow up One more email After another week, phone calls

Data collection timing

And of course the IRB…

Instrument design

Goes back to what you want to know Mechanisms by which intervention has

effect Variation in effects of intervention and

what might account for that variation

Instrument design

Do not make up your own items if you can avoid it Consider extant instruments

Ask simple questions respondents can answerNo ‘double barrel’ questions

Instrument design

Survey is a ‘conversation with a purpose’ Script the conversation so it flows Use skip patterns to avoid asking

irrelevant questions Keep is as short as possible

theory

Study groups

Chemistry degree

knowledge

Self-confidence

Enjoyment

Self-confidence: Identity

How much do you agree or disagree with the following statements?

You see yourself as a science person

Others see you as a science person

1=Strongly agree

2=Agree

3=Disagree

4=Strongly disagree

Self-confidence: Self-efficacy

You are confident that you can do an excellent job on tests in this course

You are certain you can understand the most difficult material presented in the textbook used in this course

You are certain you can master the skills being taught in this course

Analyzing Data

Depending on design RCT: difference in means Other designs:

Covariate adjustment Difference in means for certain

subgroups

Analyzing Data

Our example: Does program impact self-confidence? Mean change in self-confidence:

Control students in fall 2012 Control students in spring 2013 Treatment students in spring 2013

Covariate adjustment

List of journals for possible publication of STEP Type 1 results (list generated by audience)

• Advances in Engineering Education - http://advances.asee.org/• CBE Life Sciences Education - http://www.lifescied.org/• Chemical Educator - http://chemeducator.org/• Community College Journal of Research and Practice - http://www.tandf.co.uk/journals/authors/ucjcauth.asp• CUR Quarterly - http://www.cur.org/publications/quarterlies.html• Journal of Applications and Practices in Engineering Education - http://japee.net/• Journal of Chemical Education - http://pubs.acs.org/journal/jceda8• Journal of College Science Teaching - http://www.nsta.org/college/• Journal of College Student Retention -http://www.cscsr.org/retention_journal.htm• Journal of Research in Science Teaching - http://www.narst.org/publications/jrst.cfm • Journal of STEM Education: Innovations and Research

http://ojs.jstem.org/index.php?journal=JSTEM&page=index• Journal of Women and Minorities in Science and Engineering -

http://www.begellhouse.com/journals/00551c876cc2f027 • Physics Teacher - http://tpt.aapt.org/• Physics Today - http://www.physicstoday.org/• Portland International Conference on Management of Engineering and Technology (PICMET) -

http://www.picmet.org/main/• STEPcentral.net A Community Forum for NSF STEP Projects - http://stepcentral.net/• Student Affairs in Higher Education - http://www.naspa.org/