OER Degee Initiative Kickoff | Data & Evaluation Services

© 2015 SRI International© 2015 SRI International

OER DEGREE INITIATIVE RESEARCH & EVALUATION

Being a Research Partner

OER Degree Kickoff InstituteSan Francisco, June 13-15


Agenda

• Meet each other• What do we want to learn?• Our approach• Cost analysis• Roles & responsibilities, permissions, & timeline• Questions / concerns?


Introductions

• Meet the SRI team

• Meet the rpkGROUP team

• Meet each other

3


Hypotheses to test• Some students do not purchase required textbooks due to high

cost and do poorly or withdraw from courses as a result

• Some students take fewer courses than they otherwise would due to cost of textbooks

• If this cost is removed, students will progress more quickly and successfully towards degree attainment

• OER Degrees are a cost effective way to improve student outcomes and reduce financial burden on students

• Redesigning programs around OER will trigger additional improvements in pedagogy and institutional culture


Research Questions for the Evaluation – Academic impacts and implementation

RQ 1: Academic outcomes: Do students who take multiple OER degree classes make more progress to degree than similar students who take traditional classes? Are OER degrees more or less beneficial to particular subgroups of students? What are the key moderators of effects on student outcomes?

RQ 2: What are best practices, facilitators, and barriers associated with implementation of OER degree programs?

RQ 3: What impacts do OER degrees have on key stakeholders’ experiences and on institutional culture?

(We’ll get to economic impacts in a minute)


Our Approach to the Evaluation

Aggregate pass/completion data will enable us to evaluate the overall impact of the OER Degree Initiative - how many OER course sections are offered, how many students are involved

Quasi-experimental studies will enable us to estimate the impact of OER Degrees on key student outcomes

Combine with cost analysis to measure cost effectiveness and sustainability

Implementation research will enable us to provide formative feedback to grantees and broader community


To put this another way…

Enrollment/completion/ cost reporting; instructor survey – all 34 colleges

Quasi experimental studies – 10-12 Research Partners

Implementation data collection (phone

interviews, student surveys, site visits)Cost deep

dives – 5 colleges


Research Partner Focus

• Most research on OER effectiveness has focused on course-level outcomes. This project will focus primarily on student progress to degree.

• Work with SRI to identify comparison population of similar students, ideally with no OER course exposure

• Provide agreed data for all students who enroll in OER degrees and comparison groups

8

Why so important?

What are we asking?


But aren’t course-level outcomes important too?

YES!

• SRI can also provide technical support for selected course-level studies, such as large enrollment gateway courses.

• These should be included in college study plans.

9


Research Questions for the Cost Evaluation RQ1: How do OER degrees impact costs for students and institutions?

RQ2: What is the “return” on investment (ROI) for students, colleges, and other stakeholders, where “return” refers to financial savings from reduced cost of attendance and time to completion, or increased revenue to colleges from greater persistence and attainment?

RQ3: What does it cost an institution to make the transition to OER degrees, including both start-up costs and lost revenue streams?

RQ4: What is the cost effectiveness of this model in terms of cost per desired unit of impact on student outcomes?

RQ5: Is this model self-sustainable (i.e., can OER degree programs be established and maintained without philanthropy)?


Approach to the Cost Evaluation Cost evaluation will address the financial impacts that OER Degrees have

on students and institutions. Capture impact of OER on student degree costs, institutional pricing strategies, and institutional revenues from lost

textbook sales and improved student outcomes.

Analyze start-up and ongoing costs associated with developing and delivering OER degrees.

Financial data combined with information on academic outcomes will inform the cost effectiveness of OER degree programs.

Collect information via interviews, SRI instructor surveys, data collection and analysis templates. Capture top level impacts on student costs and institutional revenues from all 34 colleges using a data collection and

analysis template.

Collect additional information from the five cost analysis partners on cost of implementing OER degree courses and programs (activity based cost model).

Provide technical assistance to help grantees report the information requested and to make appropriate data assumptions when needed.


Cost Evaluation Activities

1. Webinar on cost evaluation approach and data collection

2. Data collected via SRI phone interviews

3. Data collected via 2 SRI instructor surveys

4. Cost partner interviews

5. Data collection template

6. Modeling of economic impacts on students and institutions

7. Formative and summative reporting


Research Activity

SRI Responsibility Grantee Responsibility

Quasi-experimental studies

• Provide study planning tool

• Draft college study plan

• Complete study planning tool• Provide feedback and sign off on

college-specific study plan

Instructor surveys

• Develop survey instrument

• Conduct online survey

• Provide instructor contact information

• Encourage participation

Student survey

• Develop survey instrument

• Distribute packets of consent forms and surveys

• Conduct online survey for online classes

• Administer gift cards

• Provide student email addresses (for online classes)

• Coordinate distribution of survey packets to instructors (for F2F classes).

• Instructors conduct paper surveys during class time.

Roles & Responsibilities


Research Activity

SRI Responsibility Grantee Responsibility

Site visits • Coordinate with grantee liaisons

• Conduct visits

• Coordinate with SRI to schedule interviews, focus groups, class observations

Section-level data reporting

• Provide data template• Manage data collection

across sites• Provide secure file server

• Coordinate campus data collection

• Provide section-level data at 4 intervals for all OER courses and, where possible, comparable non-OER courses

Student-level data reporting

• Provide data template• Comparison group

matching & data analysis

• Provide student-level data at 4 or 5 intervals (depending on prior agreement)

Roles & Responsibilities Continued


Near Term Activities

• SRI liaisons will be assigned to each grantee

• Liaisons will work with grantees to − Sign letters of commitment− Create college study plan, starting with study planning

tool − Determine whether local research permission is

required− If yes, provide inputs to research application− Put in place data sharing agreements

15


Project Timeline


Research Challenges

• Contamination from existing OER courses

• Definition of “treatment” cohort for QEDs. Is four OER courses the right threshold?

• Student attrition

• Variation in academic calendars, data availability, data definitions

• Others?


Questions or Concerns?Thank you!


What data is required for quasi-experimental studies?• Essential elements. For each

student in the study population:− Baseline indicators (placement

test scores, GPA)− Demographic variables (DOB,

gender, race/ethnicity, Pell status)

− Whether the courses they enroll in are OER/non-OER

− Course level-outcomes (withdraw, course grade)

− Retention / progression− Credits earned for each course − Cumulative credits earned− Credential attainment

• Desirable elements− Whether student enters

program of study− Additional background data

(HS GPA, SAT/ACT score, year of HS graduation, ELL status, first generation status)


Recommended Campus Teams• Data person• CFO / finance• IT • Librarian?• Counselors / registrar


When What Who

Summer 2016 Detailed research plan Design college-level quasi-experimental studies

Grantees / SRI

Fall 2016 (Oct-Nov)

SRI conducts first instructor surveySRI conducts phone interviews

SRI, with assistance from grantees

Winter 2017 SRI collects first batch of student-level data (test run) Research partners with OER courses starting fall 2016

Spring 2017 Formative implementation reportFirst set of site visits

SRI, with assistance from grantees

Summer 2017, Winter 2018, Summer 2018, Winter 2019

All grantees submit section level data on OER coursesResearch Partners submit student-level data for spring 2017

All grantees

Fall 2017 Student survey SRI, with assistance from grantees

Throughout grant period

SRI provides technical assistance to grantees for design and implementation of studies, data collection

SRI

Fall 2018 Second instructor survey SRI

Summer/fall 2019 Final reporting SRI

Major Evaluation Components

Education

OER Degee Initiative Kickoff | Data & Evaluation Services