Measuring the Effectiveness of Online/Blended Programs (262191723)

8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)

http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 1/77

Measuring the Effectiveness of Online andBlended Programs

An ELI and Quality Matters webinar

April 8, 2015



What is evidence of impact?

Why do we need it?



Providing evidence to:

persuade

plan & set goals

scale

develop consensus

know your work makes a difference



“Does OL/BL work?”

Do the students learn “better”?

Do the faculty teach “better”?

Are the students more engaged?

Are the student evaluations “better”?

Is it “efficient”?



Brown, Diaz, “Seeking Evidence of Impact: Opportunities and Needs, EDUCAUSEReview, Sept/Oct 2011, pp. 42-54



how do we get to “what works?”



Answers via methodsAnswers and methods



how do we get there?

Research question1

Research plan2

Execute3

Communication to stakeholders4

here’s what works



tools

methodologies

rubrics

frameworks

data types



Challenges

afterthought syndrome

recalcitrant data

lack of expertise

poverty





ELI/QM: Measuring the Effectiveness of Online/Blended Programs

Kay Shattuck April 8, 2015



A continuous improvement model for

assuring the quality of online courses andonline components of blended courses

through a collegial review process.



• QM tools and processes for online learning quality

assurance

• Evidence of QM impact on learner success

•Research tools for providing meaningful evidence

Today’s Takeaways



How much do you know about QM and the continuousimprovement processes for effective online/blendedprograms?

• I am very knowledgeable and/or actively engaged in

them

• I’ve investigated and understand the basics about QMtools and processes, but do not actively use them

• I know little or nothing

Poll



A validated RubricTM of quality standards and a

process of continuous improvement at course,program, institutional levels.

QM Rubric is central; however, QM is more than “a Rubric”

About continuous, rigorous application of improvement based

on impact evidence

QM: Rubric and Processes



• Eight general standards (https://www.qualitymatters.org/rubric)

Course Overview and Introduction

Learning Objectives (Competencies)

Assessment and Measurement

Instructional Materials

Course Activities and Learner Interaction

Course Technology

Learner Support

Accessibility and Usability

• 43 specific standards Alignment of critical components to ensure students achieve desired

learning outcome

• Each specific standard detailed for application in annotations

Validated QM Standards of Quality

https://www.qualitymatters.org/rubric






QM Rubric Designed for Application

PeerFeedback

QM Rubric sets a threshold

Student

Learning

Course Design



Application: Impact Starts with a Course Review…

QM Peer

Reviewer

QM Peer

Reviewer

QM Peer

Reviewer



Student Impact: All of the Instructor’s Courses

123



Student Impact: Peer Reviewers’ Courses (carry over impact)



Whole is Greater Than Sum of

Parts

QM Rubric

ProfessionalDevelopment

Peer CourseReviews



Design-Based Programs & Research

• Specific application of educational theory &research

• QM Standards of quality

•QM Process of peer review as a component of aninstitution’s quality assurance efforts

• Evidence seeking from results of application

for continuous improvement

• QM focused-research



What is the impact on X of a course redesigned to meet QM Standards?

√ What exactly is X? What evidence is already available about X’s impact?

√ What is the context? Which review format is to be used? Official QM review?Informal review?

√ How is the sample to be gathered? What is the study size? How representative

are the results?

√ What exactly is the “unit” counted/observed/documented? How is it counted,etc.? What is the time frame of the study?

√ Which and how are details of the specific sample (for example, students, course-

types, and instructors) obtained and analyzed in relationship to their possibleinfluence?

√ Pre-and post-measures allow you to gauge the extent and significance of courseimprovement (QM review scores not a meaningful measure without pre/post )

Improved Course Impact Example



Consider the Impact of the Whole

Dot Potter

Sometimes we can’t see the forest for the trees



PollPlease identify the two most important shareholders for

evidence of effectiveness/impact of your online/blendedprograms?

• Yourself

• Your colleagues

• Your department

• Your institution’s academic head

• Your students

• Members of a governing board

• Members of an accrediting body• The institution’s marketing department for community

awareness



Think About/Make Notes

What evidence do those

shareholders want?

(We’ll return to this)



QM-related research from the field

What do we know now?



Positive Impact of Rubric

• Providing easy-to-use guide or self-assessment tool by designers &instructors

• Establishing shared language/concepts by teams (Greenberg, 2011)

• Linking discussion, development, and implementation of moreeffective online learning for new online instructors (Ward, 2011)

• QM standards have a high correlation with TPACK (TechnologicalPedagogical Content Knowledge) (Ward, 2011)

• High correlation with technological affordances (Bose, 2012)

• Actions, abilities, and possibilities offered within online learning environmentswhen using technology (p. 193). Refers to e-learning tools properties andinteraction of learner

http://www.tpack.org/

http://www.tpack.org/



Learner Satisfaction Studies

•Less confused, increased satisfaction

(Finley, 2005)

• Satisfaction ↑ in QM peer-reviewed courses/institutions(Aman, 2009)

• Students (Knowles & Kalata, 2010)

• Students still like lectures; even when there is more teacherinteraction online (Knapp & Paull, 2013)

• Student perceptions are not significantly impacted whencourse design is guided by QM standards (Parscal, Frey, & Lucas, 2011)



Learner Voice

• Students valued design elements identified in Rubric (Iyengar,

2006) and missed them when absent (Mott, 2006)

• Courses should be designed to meet needs of all learners,including those who might need assistive technologies (Bowen &

Bartoletti, 2009)

• Learners correlate quality design with QM standards• Quantitative, survey research

• N=3,160; 22 states; 31 institutions (Ralston-Berg, 2011, 2014)

• Research on the Impact of QM for Students projects (DropThought& Ruffalo Noel Levitz)



Student Learning

++ Engagement when activities met QM (Runyon, 2006)

↑ Higher grades on discussion board activities (Hall 2010)

↑↑Major assignment/final exam scores, as well as

overall course grades statistically significant (Swan & colleagues,

2010, 2011, 2014)

A-D course grades ↑ F course grades ↓ over 5 yrs.(Harkness, 2014)



Students’ Motivation & Self -Efficacy

Findability matters! (Simunich, Robins, & Kelly, 2012)

Motivation & self-efficacy positively correlated

Controlled study

• Eye-tracking & talk aloud (cognitive overload)

• Motivation & self-efficacy instrument

Readiness factors matter, however (Geiger, Morris, & Subocz, 2013)

Which SmarterMeasuresTM readiness factors correlate with learning

Controlled study

• QMed courses

• QMed, expert instructor• Strong LMS support

Only typing speed/accuracy and reading rate/recall had a statistically significantcorrelation to student course retention & course grade.

Question raised: What was the instructor’s role in mediating issues?



Faculty Culture

• Faculty beliefs regarding QM participation: Study it, don’t just

rely on hearsay & myths (Altman, Schwegler, & Bunkowski, 2014)

Heard faculty concerns about voluntary, unofficial QM PRprocess

Theoretical framework: Theory of planned behavior

Quantitative analysis → Qualitative lens (mixed)

Surveyed participants and non-participants

• Both: positive attitudes; likely promotion help; not likely toinfringe on academic freedom

• Nonparticipants more likely to think process would be effortful &time consuming



Organizational Impact

• Carryover effect (diffusion of treatment) to non-reviewed courses(Aman, 2009, p. 107)

• Informal sharing among faculty (Strickland & Alarcon, 2010)

• Impacting institutional course development (Parscal, Frey, & Lucas, 2011;

Harkness, 2014)

• Valuable recognition: “Expecting QM certification to solve allstudent success alone is unrealistic” (Miner, 2014, p. 106)

• Development & dissemination of accessibility policy template (Frey,

Kearns, & King, 2011)



Back to Your Shareholders? What Evidence Do They Want?

Improved grades?

Improved retention?

Improved course persistence/completion?

Improved satisfaction?

“Other” (post specifics in chat)?



Persistence/Completion/RetentionComplications of quick study

• Without controls:• No apparent difference in completion rate; however, positive

comments (Loser & Trabandt, 2006)

• Increased satisfaction, but could find no relationship with coursecompletion in one semester study via instructors’ self -report (no

evidence data tracking) (Aman, 2009)

*****************************************

• Taking a step at controls:

• 2 QMed courses completion/same instructors• Consistently higher completion rates (95.5; 95%) than average

rate for other online courses over multiple (6; 11) semesters (Dietz-

Uhler, Fisher, & Han, 2007)



Persistence/Completion/Retention: Results of Longitudinal Study

University of D.C. mappedProfessional development participation, QM courses

Applications into LMS technologies

QMed courses

Findings: Student outcomes pre/post QM

2007-2012N =1,570

Withdrawals ↓23.53 Pass class grades A-D ↑19.74

Failing class grades F ↓66.66



Research Issues and Realities• Too late to ignore possible QM influence in the “before” courses

Must detail the before for evidence of afterNot just recognizing the variables, but including in data analyses

• Definitions, definitions, definitionsQM implementation levels; persistence/retention

Exactly which input and which output variable

• Sample size, sample size, sample size & needed Time!Challenge to get participants, initially & completion

• QM has no direct access to institutional implementation dataAssist in designing

Inter-institutional study

• Expanding theoretical frameworks



•QM Research Pages

• https://www.qualitymatters.org/research

• https://www.qualitymatters.org/node/1866/download/What%20we%27re

%20learning%20paper_FINAL_May%2018,%202012_Dec2012ks.pdf

• QM Research Library• https://www.qmprogram.org/qmresources/research/

• Toolkit for Designing QM-focused Research

• https://softchalkcloud.com/lesson/serve/irsu5Mk1P8WFR4/html

Resources

https://www.qualitymatters.org/research

https://www.qualitymatters.org/node/1866/download/What%20we're%20learning%20paper_FINAL_May%2018,%202012_Dec2012ks.pdf


https://www.qmprogram.org/qmresources/research/

https://softchalkcloud.com/lesson/serve/irsu5Mk1P8WFR4/html

https://softchalkcloud.com/lesson/serve/irsu5Mk1P8WFR4/html










©2015 MarylandOnline, Inc.



POLL QUESTION

Please respond "yes" or "no" to each of the following

1. We have the mandate to conduct measurement of impact (e.g.,

tasked by campus or organizational leadership)

2. We have the resources to conduct measurement of impact (e.g.,human resources, technical capacity, workload, etc.)

3. We have an audience that is receptive to our reports on the

measurement of impact



• Join ELI as we Seek Evidence of Impact

–Catalyze new collaborations to advance evidence-basedapplications of learning innovation

– Encourage the teaching and learning community to explore

new ways of gathering evidence

– Initiate ongoing dialogues, projects, and collaborations toenable institutions to effectively gather and share evidence

of impact

http://www.educause.edu/eli/programs/seeking-evidence-impact




POLL QUESTION

Select the most difficult part of determining evidence of

impact of technology-based innovations in T&L.

1. Knowing where to begin to measure the impact of technologybased

innovations in T&L

2. Knowing which measurement and evaluation techniques are most

appropriate

3. Conducting the process of gathering evidence

4. Knowing the most effective way to analyze our evidence

5. Communicating to stakeholders the results of our analysis



0 20 40 60 80

Knowingwheretobegintomeasuretheimpactoftechnology-‐basedinnovationsinT&L

Knowingwhichmeasurementandevaluationtechniquesaremostappropriate

Knowingthemosteffectivewaytoanalyzeourevidence

Conductingtheprocessofgatheringevidence

Communicatingtostakeholderstheresultsofouranalysis

Please rank each of the following processes of determining evidence of impact of

technology-based innovations in T L issues in order of difficulty.

Select the 3 most important indicators you use to measure the



POLL QUESTION

Select the 3 most important indicators you use to measure theevidence of impact of technology-based innovations in T&L?

1. Final grades

2. Reports and testimonies from faculty and students3. Measures of student mastery; learning outcomes

4. Drop/withdrawal/incomplete rates

5. Student performance in subsequent classes

6. Graduation and retention rates

7. Measures of student and faculty engagement8. Measures of student and faculty satisfaction

9. Cost or other efficiency measures

10. Changes in faculty teaching practice





SEI Cases: http://tinyurl.com/seicases

• Probing the Inverted Classroom: A Study of

Teaching and Learning Outcomes in Engineering

and Mathematics, 2015

• Better Understanding through Data:

Completion, Motivation, and Learning inMinnesota MOOCs, 2014

• Considering Classroom Seating for Students and

Faculty in the 21st Century, 2014

• Degree Compass Course Recommendation

System, 2013

• Scaling Learning Analytics across Institutions of

Higher Education, 2013

http://tinyurl.com/seicases




SEI Cases: http://tinyurl.com/seicases

• Adoption, Pedagogy, and Analytics: Exploring TwoYears of Institutional Blog Data, 2013

• Using Laptops in the Classroom: The University ofMichigan, 2013

• Role-Differentiated Responses to Active LearningClassrooms: The University of Minnesota, 2012

• Developing a Context-Specific Measure ofTechnological Expertise: The University of

Washington, 2012• Increasing Student Success: Evaluating the

Effectiveness of U-Pace Instruction at UWM, 2012





Full SEI case available here.

http://www.educause.edu/library/resources/better-understanding-through-data-completion-motivation-and-learning-minnesota-moocs


































SEI CASES: 2 GOALS

1. Too put a spotlight/demystify the methodology used to

uncover the impact of technological/pedagogical

innovations

2. To share outcomes of impact studies/innovation work in

teaching and learning



• Go to http://tinyurl.com/SEIguide to

review the SEI case guide and template

• What are 2-3 key research questions for

your teaching and learning work?• Name 2-4 things you’d most like to know

their effectiveness to support teachingand learning?

InChat

http://tinyurl.com/SEIguide




• Go to http://tinyurl.com/SEIguide to

review the SEI case guide and template

• What kind of data will you need?• How will you collect that data?

InChat






Malcolm Brown, PhD

Director, EDUCAUSE Learning Initiative

[email protected]

Veronica Diaz, PhD

Director, Online Programs, EDUCAUSE

Associate Director, EDUCAUSE Learning Initiative

[email protected]


mailto:[email protected]











National Research Center for

Distance Education andTechnological Advancements

(DETA)

Tanya Joosten

Co-Director and PI

National Research Center for Distance Education and

Technological Advancement

Director, eLearning Research and Development

University of Wisconsin-Milwaukee

@tjoosten, [email protected]



How do we ensure all

students have access

to a quality highereducation?



Pathway

todegree

Online

BlendedCBE

Access through distance education



Ensuring quality Flickr vaxzine



F2F gold standard Flickr xenonbpics



Ensuring quality?

F2FOnline



Identify practices (instructional and institutional)

that impact those outcomes



Conduct rigorous,interdisciplinary, and

standardized research

Flicker cc

astronomyblog



Desired Outcomes

Access

Learning Effectiveness

Instructional Effectiveness

Satisfaction

http://uwm.edu/deta/desired-

outcomes/



Research questions

What are the definitions of success from students’33



Research questions

f f f

perspective?33

What are the different design components (content,

interactivity, assessments) that impact student learning?

29

What patterns of behaviors lead to increased student

learning for different populations?26

How can we define and measure student success beyond

traditional outcomes?

25

What support structures are critical to providing quality

access to online instruction?22

What is the currency of student learning beyond the existing

credit hours?22

What are the key components that promote a sustainable

and an effective teaching and learning ecosystem?21

POLL



POLLWhich three (3) research questions do you find most important toincrease access and success for students in blended and online learning,including CBE?

• What are the definitions of success from students’ perspective?

• What are the different design components (content, interactivity,assessments) that impact student learning?

• What patterns of behaviors lead to increased student learning for

different population?• How can we define and measure student success beyond traditional

outcomes?

• What support structures are critical to providing quality access toonline instruction?

• What is the currency of student learning beyond the existing credithours?

• What are the key components that promote a sustainable and aneffective teaching and learning ecosystems.



Framework of inquiry



Framework of inquiry

RQs - What are the



Research toolkits

Shared measures

– Student

performance is

based on numerical

representation of

grade converted to a4.0 scale received in

the course on

assessments and as

an overall grade.

different design

components (content,

interactivity, assessments)

that impact student

learning?

Go to: http://tinyurl com/whatisss

http://tinyurl.com/whatisss






What is student success?

Go to: http://tinyurl.com/whatisss

Spend 10 minutes sharing yourresponse to one of the following…

What is student success for you as aneducator or researcher?

What is student success for your

institution?

Thank you!!







y

Tanya Joosten, [email protected]

Twitter | @tjoosten

http://www.uwm.edu/deta

http://www.slideshare.edu/tjoosten





Documents

Measuring the Effectiveness of Online/Blended Programs (262191723)