Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)

8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)

http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 1/39

Rob AbelChief Executive

IMS Global Learning [email protected] , @LearningImpact

Simplifying Learning

Analytics via theCaliper AnalyticsFramework

Vince KellenSenior Vice Provost for Analytics and Technologies

University of [email protected] , @vkellen

mailto:[email protected]







© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved

Agenda1. Brief Overview of Learning Analytics

2. Progress of IMS Caliper Analytics3. Predictions4. Challenges and Steps to the Future





2. Progress of IMS Caliper Analytics3. Predictions for Next Year Progress4. Challenges and Steps to the Future



Source: CETIS: Adam Cooper; A brief history of analytics

http://publications.cetis.ac.uk/wp-content/uploads/2012/12/Analytics-Brief-History-Vol-1-No9.pdf


















Source: EDUCAUSEwww.cdn.educause.edu/visuals/shared/er/ERM1151/ERM1151_table.jpg

PersonalizedLearningandStudentSuccess

FacultyandInstitutionalPerformance

http://www.cdn.educause.edu/visuals/shared/er/ERM1151/ERM1151_table.jpg





Learning Analytics : Does student appear to be participating and/orunderstanding a key concept?

Academic Analytics : Is the performance of a cohort of students athistorical norms?



Source: EDUCAUSEwww.cdn.educause.edu/visuals/shared/er/ERM1151/ERM1151_table.jpg

IMSFocus










“Analytics” is one of the hottest buzzwords ineducation. For ed-tech companies, it is also aselling point. By using vendors’ suites andsolutions, colleges will gain access to dataabout why some students succeed and whereand when others stumble -- or so the pitchgoes.

But there’s a catch, said Linda Feng, senior

product manager for analytics and SISintegration at Instructure . “The whole premiseis that all that data has to be in their world,”she said.

“All these systems have their own languages

that are written for their own needs,” saidJohn Whitmer, director for platform analyticsand research at Blackboard. “In order to talk acommon language, we have to agree to acommon vocabulary .”




The Future of the “LMS” aka The EdTech Ecosystem

Easier/BetterFunctionality

Easier/BetterInsight



© 2015 IMS Global Learning Consortium, Inc. All Rights ReservedPage 10

Advancing

TechnologyUsage

EducationalProcess + ResultsImprovement

Teacher/Student/InstitutionalProductivity Gain

Learning ImpactEdTech MaturityModel

E d u c a

t i o n a

l V a

l u e

Evolving Expectations









Caliper Facilitated by IMS Global

12

• World’

s primary K-20 educational technology standards memberconsortium: 20 year history, 330+ member organizations (institutions& suppliers), accelerating growth & adoption

• 25+ IMS standards enable IT/academic technology flexibility,

efficiency and agile innovation in a time of rapid change• IMS works closely with our community on adoption of new

technologies and building awareness through Learning ImpactLeadership Institute conference and awards program (May)

imsglobal.org @LearningImpact

0

50

100

150

200

250

300

350IMS Global Learning Consortium Growth

IMS MemberOrganizationsEnd of Year




EDU

OPERATIONSPLATFORMS

MOBILE EDU APPS

PUBLISHER /COURSEWARE SOLUTIONS

LEARNINGPLATFORMS

S

EDU TOOLS /SERVICES

S

S

S

S

InstructorsStudents

Adaptive Personalize

CurriculumLearningResearch

S = Sensor

EP = Sensor EndPoint

IMS Caliper EnabledServices

● Caliper Sensor Endpoint(s)● Compliant Data Mgmnt Services

DataManagement

Services

EPData

Stores

Admins

Caliper Learning Analytics Ecosystem




Why Caliper?Standardization : encourage a common, extensible yet structuredapproach to describing, collecting and transporting learner interactiondata for later consumption by researchers, educators, platforms, appsand services.

Innovation: provide a extensible data model, controlled vocabularies andan API that enables new uses of learner interaction data.

Interoperability : promote data exchange, sharing, mashups between

systems, institutions and people: learning analytics ecosystem

Stewardship : evolve an EDU-optimized technical specification under theauspices of IMS Global and its member institutions and organizations.



“Caliper opens up access to meaningful data”

“Caliper is built around the IMS Learning Sensor API to define basic eventsand standardize metrics across different learning environments, and alsouses IMS LTI standards to integrate standardized measurement with toolsinteroperability. Put together, Caliper is less about analytics, and moreabout the data standardization that will eventually drive learning analyticsefforts .”



Caliper v1.0 Certified Suppliers!




Caliper event

action

actor

object

target

generated

edApp

group

membership

startedAtTime

endedAtTime

duration

Learning Context

Activity Context

Time (ISO 8601)

Base statement

fedSession




IMS Caliper: Click Stream Data forLearning Analytics




Caliper fundamentalsInformation model : standardized set of learning activityprofiles & controlled vocabularies that extend set offoundational metrics.

API: governs the interactions between Caliper, apps,platforms, services, event store(s).Sensor : code library designed to simplify instrumentation ofplatforms/apps/services. Multiple bindings: Java, JS, Ruby,Python, PHP, .Net.

Conformance: test suite ensuring implementationcompliance.Dev community: docs, presentations, bootcamps, referenceimplementations.




Scope: Caliper

describe

collecttransport store

analyzevisualizereportsharepredictinterveneadapt

instrument persist use

feedback




Scope: Caliper 1.0

describe

collecttransport store

analyzevisualizereportsharepredictinterveneadapt

instrument persist use




Caliper code repos (Github)

sensorscaliper-javacaliper-jscaliper-ruby

caliper-pythoncaliper-php (includesexample)caliper-net

supportcaliper-contextscaliper-conformancecaliper-common-fixtures

future?caliper-model/profilescaliper-docs

sample codecaliper-java-examplecaliper-net-example



Unizin is a Major Caliper Initiative





“There’s a danger that the vendor communitycan say, ‘This is our data, not yours, and youhave to pay us to learn how you teach

students,’” Kellen said. “I disagree with that.The academy needs to have the line of sight tothe student in digital form.”



Caliper Institutional Leadership Council(Caliper Real-time Analytics Messaging*)

*Caliper RAM = Processing thousands of events per second and responding withintervention messages within one second



Real-TimePersonalized Messaging

Tai lo r e d C o m m u n i c a t io n sf o r S t u d en t S u c c es s

Vince Kellen, Ph.D., Senior Vice Provost, Analytics and TechnologiesPatsy Carruthers, Director of Academic TechnologyChris Rice, Enterprise Architect, Academic Technologies



Caliper RAM Purpose:Integrate data needed to personalized learning and analyze

student success analytics in one real-time architecture• Real-time personalized interactions

• Tailor and target reminder services, avoid over messaging, enable timing of message deliverybased on user temporal proclivities, mix and match messages across learning, support andprogression areas

• Enrich the university’s ability to analyze and research both student success and student learning • Deliver targeted, on- demand peer tutoring based on the student’s profile • Target micro-surveys and assessments to capture additional information needed to improve

personalization• Give students ‘academic health’ recommendations on how they can improve in study,

engagement, support, etc.• Potentially let students opt their parents, friends in to this information so the student’s network

can help

• Allow for open personalized learning

• How content gets matched to students is psychologically complex• Several theories of how humans learn give many insights• Students differ in the following abilities and attributes: visual-object, visual-spatial, reasoning,

cognitive reflection, need for sensation, need for cognition, various verbal abilities, confidence,persistence, prospective memory, etc.

• We need an open architecture to promote rapid experimentation, testing and sharing of whatworks and what doesn’t

• We need real- time messaging so that learning interventions can ‘fit’ within the ‘window’ of short -term memory University of Kentucky



Confusion Detector I A student performs poorly on homework questions

aligned to a Basic Chemistry Learning Outcome,and misses 3 out of 5 in-class audience-responsequestions in class the following day.

Real-time analytics detects that the student has performedsimilarly poorly on questions and homework assignments alignedwith that learning outcome.

The real-time analytics service lets the student know that theyappear to be having trouble with the Learning Outcome inquestion (link to syllabus in Canvas/LMS).

Student is informed that a peer tutor is available and will becalling shortly.

Student receives supplementary information and links toadditional exam prep questions related to the Learning Outcome.

Peer tutor discusses the Learning Outcome and relatedconcepts with the student. Follow-up meetings can be arranged.Real-time analytics service places an entry into the students

advising log and notifies the faculty.If the student underperforms on questions aligned to that

learning outcome in the online examination software, repeatsteps 2-6 immediately following the exam, including a reminder

that (if exam is cumulative) that they will see this material againon the final exam.

Real-timeevent stream

“Uber Tutor” Person-to-person

Confused student

Real-time analytics Peer tutor



Confusion Detector IIIn the middle of working on an onlinehomework assignment, a General Chemistrystudent searches for [key chemistry concept]in the online course content, replays theassociated segment in the video lecture andhas gotten 2 out of 3 in-class audienceresponse questions incorrect in the priorlab/lecture related to this topic.

1.Through NLP content analysis, real-time analytics detectthe confusion regardless of the type of content andnotifies an Uber (peer) Tutor.

2.The real-time analytics service lets the student knowthat a peer tutor is available and will be calling

3.The peer-tutor and the student discuss the [keychemistry concept] and arrange to meet later to explain

4.Homework system displays follow-up questionstriggered by student check-in at the Uber Tutor meeting.

5.The real-time analytics service places a entry in thestudents advising log and notifies the faculty


“Uber Tutor” Person-to-person

Confused student

Real-time analytics Peer tutor



Early Alert

Several freshmen Chemistry students arestruggling with the material in the first few days.Each one scores low or fails the first formativeassessment.

1. Real-time analytics detects an LMS quiz or in-classaudience response system score below an instructor-

defined value and alerts each student and theinstructor of those grades.

2. The real-time analytics service sends an alert to thestudent and instructor of early difficulty withintroductory material.

3. Real-time analytics message to instructor triggersmessage from instructor to student featuring remedialmaterial and referral to The Study.

4. The real-time analytics service places a entry in thestudents advising log for follow-up


Mobile feedmessage

re or failure on early assessment

Real-time analytics LMS Mobile Feed



Class engagementSome students in a large-enrollment course

have delayed logging in to access readingsand video, placing them in danger of losingmomentum as the course begins. Introductoryposts on the discussion board are very brief.

1. The real-time analytics service detects low

recency and frequency of clickstream activitycombined with few or short, disengaged discussionresponses

2. The real-time analytics service sends apersonalized message in the learning systemstudent home page and messages the student

mobile feed3. Student accesses instructor’s recommendationsand content links contained in message andconsiders how to improve his engagement

4. The real-time analytics service places a entry inthe students advising log and notifies the faculty,

with an optional entry in the student academic alertsystem


Remindersand resources

Confused student

eal-time analytics



Abrupt class disengagement behavior

A student who has been engaged in class

experiences a family issue that causes a loss ofpersonal or financial support, and as a resultmust turn attention to those problems.

1. Through content analysis, real-time analytics detecta sudden drop-off of clickstream in the LMS, audience

response systems, use of self-service tools and classattendance

2. Real-time analytics notifies faculty and sendsmessages of concern and offer of assistance ispushed to student learning system homepage,student mobile feed and, optionally, to the student

mobile phone

3. The student finds multiple messages of concern andcontacts instructor or adviser to discuss

4. The real-time analytics service places a entry in thestudents advising log and sends an alert to the

student academic alert system


Messageof concern

Worried student

Real-time analytics Mobile message









Next 12 Months• Caliper will be adopted by a majority of the global market-leading

learning platforms• Caliper will be used by several major publishers to collect millions of

events per week to analyze and improve quality of learning materials• Several universities will implement the first ever real-time analyticsmessaging at scale based on Caliper RAM

• Caliper profile to measure adoption/use of e-text and popular learningtools will be in production use by a first wave of leading U.S. universities

and their suppliers• Synergies between learning analytics and digital credentialing/badges will

emerge in the context of enabling student success









Learning Analytics Challenges

• Need to start with good data

• Need to have a clear goal

• Need organizational culture and capacity to enact action• To achieve any of the above in education, motivation for

improvement must be compelling

• “Good judgment” is still required – as much an art as a science




How to Take Advantage of Caliper?

38

Collaborate with IMS communityInstitutional analytics leadership group

Supplier technical development and supportContact Rob [email protected]

Ask for Products IMS Caliper CertifiedRequire IMS Caliper conformance certification (play your

part in making sure the ecosystem is interoperable)imscert.org


http://imscert.org/

http://imscert.org/




Thank You!

[email protected] @LearningImpact

[email protected] , @vkellen





Documents

Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)