Measuring Your Success in Course Redesign

The Dirty Words in Education

1. Success

2. Efficacy

3. Assessment

4. Evaluations

Our Topics for Today

1. How will you measure success?

2. Will you use historical data (common exams, etc.) to support the efficacy of your assessments (MyLab & Mastering)?

Mastering platform in Sciences similar to MyLab platform in other disciplines

3. MyLab & Mastering assignments will contribute what percentage to a students final course grade? = assessment

4. What do you do with the information evaluated?

Today’s Plan

My course structure

Implementation of a few simple techniquesQuick ways to measure successHow to use MyLab / Mastering to maximize student learning

How do other data support MyLab / Mastering use?

How much is too much (or too little!) for a grade item?

How to use MyLab / Mastering for Course Assessment

My Course

General Biology (year-long sequence)

Typical # of students: 250-340 (taught in two sections)This year 408

Typical # of different majors: 55 from all RIT Colleges

~ 50/50 science/non-science

Freshmen -> grad students

Current grade calculation (2013 – 2014):

65-70% exams; 15% Mastering Bio HW; 10% Mastering Bio quizzes; 5-10% in class participation

Before you measure success…

Are Students Prepared to Succeed?

Preparation (lack of!) comes in many formsReading / writing comprehensionEnglish as a Second Language (or 3rd, 4th, ….)Learning disabilities Fundamental knowledge in sciencesStudy skillsDedication to the courseDedication to the degree / collegeEtc……

Preparing Students for Success

Homework & QuizAssign immediately on Day 1Provide timely feedback on the assignment

“Suggestions” from instructor do hold water!

Require regular assignments

In Class Response ActivitiesEvery day! (“mock quiz” type)Immediate or timely feedback critical

Preparing Students for Success

Be clear about the goals of pre-course activitiesE.g. pretests, diagnostic tests, etc.Benefit to students

Not enrolled in a class in which they cannot succeed

Decreases DWF rates

Increase retention – intangibles

Benefit to you / courseLess time on “catch up”, More time on application

Less stresses associated with DWF, office hours, etc.

So you have “prepared” students

… Now what?

Measuring Success – How? Of what?

Retention (D/W/F)

Common exams?

Shared assessments across sections?

Final exam scores

Final course grades

MyLab / Mastering

Alternative learning initiatives

Measuring Success – How? Of what?

What works for …YouYour studentsYour discipline

Your assessment office?Your supervisor?

Measuring Success – How? Of what?

Let’s focus on MyLab / Mastering as a success tool …

We can add in the other things later …

MyLab / Mastering = X %

2008-2009: Not much – trial only; assignments optional

3 attempts per question; unlimited time

2009-2010: Not much – trial only; assignments optional3 attempts per question; unlimited time

2010-2011: 5% of grade; ~20% never did any; nearly all students who completed the assignments had a 100% avg

3 attempts per question; unlimited time

2011-2012: 15% of grade; all complete at least most; avg scores 88-93%

2 attempts per question; unlimited time

MyLab / Mastering = X %

2012: 20% of grade HW and 5% on quizzes; all students complete nearly all (> 97% completion rate)

HW = 2 attempts per question; unlimited timeQuiz = 1 attempt per question; 60 min time limit

2012 ExampleMitosis/Meiosis HW: 96 questions; avg time spent = 44 minutes; avg score = 85.9%Homework due before lecture

Result? No time in class on process – instead, theory!

MyLab / Mastering = X %

2013 – 2014: 15% of grade HW and 10% on quizzes; all students complete nearly all (> 98% completion rate)

HW = 2 attempts per question; unlimited time; weeklyQuiz = 1 attempt per question; 60 min time limit; weekly

Food for thought … is this % too much?Will students cheat, copy, etc.?

Everything has challenges …

What is your biggest challenge?

Time is the untamed obstacle for everyone ..

Take 5 minutes Sit down! (or better yet – while you are shampooing!)What information do you really want your students to retain?

WHY?

What do you want your students to leave your class knowing?

WHY?

Challenges to Assessment

Things that I have considered and toyed with …

How much do I want the students to do outside of class on MyLab/Mastering?

Limited to “Time Estimated” = 1 hourLess if possible! I want their attention!

Challenges to Assessment

Things that I have considered and toyed with …

What is the value of a long assignment if the students don’t do it, copy it from another student, just use all of the hints, etc.?

For students who are failing exams, but have 100%+ on the homeworks, I look at their time spent in MyLab / Mastering

Discuss issue with them in person and make note in the Institute Reporting

Challenges to Assessment

Things that I have considered and toyed with …

Are the students just memorizing answers, or are they applying the information and retaining it?

Have stopped assigning reading / test bank questions

Implement more low stakes quizzes

Forced repeated recall of information in class

You have data … Now what?

Student Data Assessment

Why assess?

My general interestAre the students learning what I think they are learning?

How can I improve this course?

University call to assess Gen Ed courses (Accreditation)

Assessment needs?

My interest Exam and homework scores before and after MyLab / Mastering implementation

University Guidelines: 1 – 4 scale of comprehension

Student Data Assessment

What was assessed?

Kept it simple to start!Exam averages

Add as I have interest / time!Participation in MyLab / Mastering

Scores on MyLab / Mastering

Time spent on MyLab / Mastering

Student Data Assessment

Exam grades (presented herein)

All stored in ExcelTM since 2007

Mastering (not presented)

Export grades & time spent from Mastering to ExcelTM

Processing, Compilation, and Statistical Analyses

ExcelTM

Exam Score Trends

Err = St ErrAvg n = 244

Talk to me data!

Higher exam averages every year – but leveling out

Significant differences only in plants – but trends are obvious

Too many students from too many majors to see huge jumps in scores despite all efforts

Tightening of exam scores

Standard deviations and ranges decreasing

Increasing amount of “average time spent” on MyLab / Mastering assignments

Assessment Challenges

Why do exam grades change?

Things I have consideredWhat changes from year to year other than MyLab / Mastering?

Students?

Accessibility of tutors?

Me?

Order of topics covered?

Lab instructors?

Assessment Challenges

Why do exam grades change?

Things that I want to consider

Students Time spent on studying (other than MyLab / Mastering)?

Major?

College entrance exams? AP Bio?

DWF rates by major and biology “experience”

MeLectures improved?

Lecture structure change with online assignments?

“Napkin Notes”

General observations:

Increased time spent on assignments after 1st exam of term

Increased # questions in office hours on assignments

Increased time spent “off topic” in lecture – not a bad thing!

Changes implemented to date:

Lecture time is more interactiveParticipation activities with the students (worksheets, group work, think-pair-share, flipped lectures, etc.)

Modifications to assignments and quizzes to “challenge” the students in preparation for class and exams

Advice for Instructors Looking to Assess Success

Don’t assess while you are teaching the course!

Assessment can be done any time! Store the data, save it for later

Even if you have time … subconsciously not good for your teaching!

Don’t try to assess everything at once!

Pick one exam in your course, and compare the average ± standard deviation (standard error) between years

Even simpler?

Compare exam averages between class sections. TA DA! You just did an assessment!

Advice for Instructors Looking to Assess Success

Don’t expect MyLab / Mastering to make your students in to geniuses!

These are tools! And how the students implement the tools (accept the tools!) will be highly variable.

You may not see immediate changes in exam grades

Accept the “intangible” as a result!

You can not grade student enthusiasm (which means you can’t assess it – sorry!)

Student enthusiasm is contagious – not just to other students – but also to you!

Advice for Instructors Looking to Assess Success

What can you do with your data?

FLAUNT IT!White papers

Peer-reviewed papers

Teaching portfolio

Promotion dossier

Assessment office

Final thoughts …

What else is assessable?

I got my class back!

Not spending class time on giving quizzes, answering questions about homework format, etc.

What do we do then? Talk about Biology!

AND … most of my assessment is built in to MyLab / Mastering and my course management system!

Don’t reinvent the wheel!

Download grades, assign your assessment strategy, .. TA DA!

What else is assessable?

HUGE time savings in my classes

I have “recovered” 35 minutes per week in class (of 150 min) with no quizzes and homework things in class

Prepared students means < 45 min lecture!

> 105 min biology exploration!

Time saving is so huge … how huge is it?

Enough that I am completely “flipping” one section

DATA! DATA! DATA! DATA! DATA! DATA! DATA! DATA!

But .. The Most Important Thing to

Me?

What the Students Have to Say!

"Mastering Bio helps a lot! The more you have us do, the more I learn!”

"I wish that our exams were as interactive as Mastering! I love learning that way!"

"(Favorite part about Mastering) aspects of the assignments helped me to better grasp the materials through the videos. I would not have done as well on those parts of the test without Mastering.”

"More Mastering - 'nuf said!"

Let’s Recap

1. How will YOU measure success?

2. Will YOU use historical data (common exams, etc.) to support the efficacy of your assessments (MyLab & Mastering)?

3. MyLab & Mastering assignments will contribute what percentage to YOUR students final course grade? = assessment

4. What do YOU do with the information evaluated?

1. Success

2. Efficacy

3. Assessment

4. Evaluations

THOUGHTS!?