Evaluating and Implementing the Right One (for you)teach.ufl.edu/wp-content/uploads/2017/02/UF... · Whatare the major Systemcomponents? Student UX Lesson Reporting Instructor UX

1

Adaptive Learning SystemsEvaluating and Implementing the Right One (for you)

University of Florida 2017

Dale P. JohnsonAdaptive Program Manager

2

What have we learned (and why it matters)?

Adaptive software is necessary but not sufficient to ensure student success.

Cengage (Learning Objects) – psychologyCerego - medical school coursesCogBooks – biology and US historyKhan Academy – developmental mathKnewton – developmental mathMcGraw Hill ALEKS - college algebraMcGraw Hill LearnSmart Connect - chemistryPearson MyMathLab with Knewton - college algebraPearson Mastering with Knewton - physicsSmartSparrow – custom science course

3

Your needs and interests today?

• Who’s already working with adaptive learning systems?

• Who’s trying to decide whether to use them?

• Who’s evaluating results from a completed project?

• Who’s in the wrong seminar?

• Get the MOST out of this session:

ACTIVITIES:What do you want to accomplish today?What does your institution want to accomplish?

4

Why is this discussion important?

• Biology 100 – The Living World (300 non-majors)

Persistence - Performance - Satisfaction

Financials


Persistence - 10% withdraw rate Performance - 80% grade C or better Satisfaction

Low student evaluations for large lecture sectionsFrequent faculty turnover

FinancialsWe lose money when students repeat the course.We spend money training and retaining faculty.

5

Getting Started with Adaptive Learning

Understand adaptive systems & terminology

Building and Maintaining Adaptive Learning

Identify challenges and opportunities with adaptive systems Analyze the costs and benefits of adaptive systems

Session Learning Objectives

6

What’s the best way to help?

What do they need help with?

Who needs help?

Students ON-Track

What is the promise of adaptivity?


From mass production to mass PERSONALIZATION.

7

How are the AL systems different?

Personalized

LMS ADAPTIVE

- Lesson Plan Fixed Variable

- Content Common

Individual- Presentation Group


8

How do the systems help students?

Personalize

Respect their prior knowledgeRespond to their learning needsReduce gaps in their understanding


9

How do the systems help faculty?

Tracking

Monitor which students need assistanceMeasure curriculum performanceMaximize course outcomes


10

February 21, 2017

Learning Benefits from these Pedagogical Techniques*

Formative Evaluation (d=.90) Acceleration (.88) Effective Feedback (.73) Meta-cognition (.69) Mastery Based Learning (.58) Concept Mapping (.57) Interactive content (.52)

*Source: John Hattie’s Visible Learning (2009)

15 year, 800+ meta analysis on achievement

Cohen’s d = effect size which is an indication of whether something is practically significant.

10


What should we expect to gain?

When d is > .40 there were excellent learning gains.

11

Where are we in the technology cycle?

WE ARE ALL HERE!

Adaptive learning systems are still experimental.


12

Lesson sequence Content selection

Algorithm (analytics) – recommendations Assessment – rapid remediation Assignment – decision tree Agency – student chooses


What is adapting to the student?

What is guiding the adaptation?

13

(or Competency, Skill, Topic, Unit, Module, etc.)

Components form an “Ed Molecule”


What makes up a measurable Lesson?

Objective

ContentAssessment

14


How are the Lessons sequenced?

Basic Math Algebra Calculus

Course Configuration

Lesson 1

L-3

L-5

L-4

L-6

L-7 L-x

L-2

Lesson Chart (or Knowledge Map, Skill Graph, etc.)

15

AlgorithmAssessmentAssignmentAgency

Lesson Library

LessonInstructional

MaterialsPractice

ProblemsAssessment Instruments

LessonInstructional

MaterialsPractice

ProblemsAssessment Instruments

LessonObjective

Assessments

Content

How do students experience Lesson adaptivity?


StudentLessonObjective

Assessments

ContentContent

Content

16

What is an example of Lesson adaptivity?

Khan Academy

ACTIVITY:Go to https://www.khanacademy.org/exercisedashboard


http://www.khanacademy.org/exercisedashboard

http://www.khanacademy.org/exercisedashboard

17

How is Content adapting to the student?

Learning Loop

Lesson #1 Lesson X

AlgorithmAssessmentAssignment

Agency


(or Resource, Material, Text, Video, Exercise, Problem, etc.)

Pre-Assessment

Content Content


Agency


Agency

18

Vendors often use a mix of all four models:


What is an example of Content adaptivity?

Algorithm

Assessment

Assignment

Agency

19

Progress can depend on performance! NOT JUST THE SCHEDULE


What does all this mean to the student?

ASU MAT142August to December 2012Concepts completed by each student Median

20

Get the MOST out of this session:

ACTIVITIES:Complete Khan Academy initial analysis

Think of questions to ask tomorrow

21

Getting Started with Adaptive Learning


Building and Maintaining Adaptive Learning

Identify challenges and opportunities with adaptive systems Analyze the costs and benefits of adaptive systems

Seminar Learning Objectives

22

What are the major challenges and decisions?

Pedagogy System Course Development Faculty Facilities Evaluation

Identify challenges and opportunities with adaptive systems

ACTIVITY:Build your plan as we go…

23

How can the system support your Pedagogy?


Lecture

Flipped

Online

Other?

- homework and assessment

- content delivery and assessment

- content, homework and assessment

24

Student learning

What other pedagogical decisions matter?

Self-paced or Synced

Complex tracking Lesson alignment

Flexible exam dates Fixed exam dates

Independent Interactive

Competency Completion

Faculty need to make these decisions first.


25

Options we’ve tried:

Online and Self-paced

Flipped and Self-paced

Flipped and Synced

Our best results have been with “Flipped and Synced”

What pedagogical strategy works for us?

Bloom’s Taxonomy

Adaptive Learningbefore class

Active Learning in class


26

ADAPTIVE SYSTEM

ACTIVE CLASS

4. ASSIMILATEWrite essay, solve problems, take quiz, etc.

Prove

ParticipateProve

Prepare

1. ACQUIRE INFORead textbook, watch video, do simulation, etc.

2. ANALYZEDo practice problems, take quiz before class

3. APPLYSolve an applied problem (case study) with classmates.

How does this pedagogy work in practice?


27


What does a course structure look like?

Adaptive and Active learning are symbiotic in the educational ecosystem.

28

Adaptive

What are the major System components?

Student UX

ReportingLesson

Instructor UX

GroupingReporting

Content Curation

AssociateEdit

InformationArchitecture

ContentDatabase

Course Creation

SequenceEdit

Analytics


29

Construct or Configure

Time consuming Quick

Flexible Constrained

Costly Cheap

Riskier Safer

Consider the commitment before committing.

What are some key system selection issues?


Course Creation

30

How are different systems implemented?

Knewton Knowledge Graph

The market is moving toward modularity.


Course Creation – Construct

Lesson 1

L-3

L-5

L-4

L-6

L-7 L-x

L-2

Configure

31


How does the system handle different subjects?

Course Creation – Subject Matter

Math Model Assess, learn, assess Predicated on 12 years of prior math education

Other Disciplines More traditional presentation Learn, assess, remediate, assess

Adaptive logic may not be the same for all subjects.

32


How can you manage content in the system?

Content Curation

Vendor Resources Prebuilt courses Library of materials

Other Resources Faculty produced content Open Educational Resources (OER)

Will the vendor allow outside content?

33

How can you manage Course Development?


Adaptive tech is a team sport, so plan accordingly.

Faculty Technologists VendorPersonnel

Instructional Designers







GraphicDesigners

Librarians Others?Video Producers

GraphicDesigners


GraphicDesigners


GraphicDesigners

Librarians LeadersVideo Producers

34

Strong provost and departmental

support (and $)

Faculty leadership for each course

Instructor peer mentoring for training

“Guide on the Side” is not for everyone

Be patient, it’s a learning process

The technology success depends on the teacher!

What can help Faculty succeed with AL?


35

Think the project all the way through.

What types of classroom Facilities will you need?

How many rooms?

How big do they need to be?

How long will they take to build?

How much will they cost?

Who is going to pay for them?

How will you support them?


36

Macro indicators• Persistence (lower student withdraw rate)• Performance (higher student pass rate)• Satisfaction

• Student• Instructor• Administrator

• Financials• Money saved or spent

Micro indicators• Assessment results (lesson or exam level)


What factors could you Evaluate?

37


How does ASU evaluate our GenEd efforts?

MISSION: Enable Student Success

OBJECTIVES:

Improve critical thinking & problem solving

Increase student subject mastery

Increase student retention

Improve instructor insight

Achieve 90% freshman retention

Do Applied Concept Exercise every class

Help 90% of students get C or better

Re-enroll 90% of students next semester

Identify struggling students by week 2

38


Persistence Performance Satisfaction

Financials

How has it worked at ASU?


Persistence – withdraws fell to 1.5 % Performance – C or better increased to 94% Satisfaction

Student course evaluations improvedProfessors said, “I’m never lecturing again.”

FinancialsWe saved money from not having repeat students.We spent money to put more instructors in the classroom.

Analyze the costs and benefits of adaptive systems

39

What do the students think?


40

• Rio Salado College student survey results…•

98% of the students liked the adaptive course

96% felt the system adapted to their needs

89% stated the adaptive course was more helpful than traditional textbooks

**N = 138 students enrolled in ENG 081 and 091 (69% response rate)

What do the students think?


41

What is this going to cost?

Implementation• Construct

• Configure

Operation

Implementation• Construct ~ $50,000

• Faculty time ~ 9 months • Staff time ~ 12 months• Systems integration ~ 1 month• Content development and licensing

• Configure ~ $5,000• Faculty time ~ 1 month• Staff time ~ 3 months

OperationStudent license $35 - $100Faculty training – every semester


42

• Dale Johnson

• [email protected]

• 480-884-1927

Thank You for participating!

mailto:[email protected]

Documents

Evaluating and Implementing the Right One (for you)teach.ufl.edu/wp-content/uploads/2017/02/UF... · Whatare the major Systemcomponents? Student UX Lesson Reporting Instructor UX