Upload
steve-lonn
View
2.827
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Learning Analytics is an emerging topic of interest throughout all levels of education focusing on how to harness the power of data mining, interpretation, and modeling. However, there are several similar terms (academic analytics, predictive analytics, business intelligence, etc.) that can confuse educators and administrators alike. In this session, we will unpack this new area of interest and discuss how institutions can begin to leverage available products and open source communities to utilize analytics to improve understandings of teaching and learning and to tailor education more effectively. We will briefly present an overview of the learning analytics field, drawing from popular examples such as the Signals project at Purdue U. and the Check My Activity tool at U. Maryland, Baltimore County. We will also review the structure of Sakai CLE and OAE user-level metrics and briefly discuss projects to design and implement tools to utilize these metrics in meaningful ways.
Citation preview
June 10-15, 2012
Growing Community; Growing Possibilities
Learning Analytics 101
Steve Lonn, University of MichiganJosh Baron, Marist College
2012 Jasig Sakai Conference 2
1. What is Learning Analytics (LA)?
2. Current LA work in Higher Education
3. Data available in Sakai CLE & OAE
4. Big Questions to Ponder
5. Q & A
Slides Available: slideshare.net/stevelonn/
Agenda
2012 Jasig Sakai Conference 3
BIG Data
“...datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze.”
Manyika et al. (2011)
2012 Jasig Sakai Conference 4
Big Data in Higher Education
2012 Jasig Sakai Conference 5
Big Data in Higher Education
Analytics:
An overarching concept that is defined as data-driven
decision making
van Barneveld, Arnold, & Campbell, 2012adapted from Ravishanker
2012 Jasig Sakai Conference 6
Analytics at Your Institution RIGHT NOW
2012 Jasig Sakai Conference 7
Analytics at Your Institution RIGHT NOW
Business / Academic Analytics:
A process for providing higher education institutions with
the data necessary to support operational and financial
decision making
van Barneveld, Arnold, & Campbell, 2012adapted from Goldstein and Katz
8
evidenceframework.org/big-data/
Educational Data Mining
Learning Analytics
Bienkowski, Feng, & Means, 2012◦ SRI International
2012 Jasig Sakai Conference
Dept. of Education Issue Brief
2012 Jasig Sakai Conference 9
Generally emphasizes reduction into small, easily analyzable components◦ Can be then adapted to student by software
◦ Siemens and Baker, 2012
Predicting future learning behavior
Domain models for content / sequences
Software-provided pedagogical supports
Computational models that incorporate student, domain, and pedagogy
Educational Data Mining
2012 Jasig Sakai Conference 10
Example: Cognitive Tutors Pittsburgh Advanced Cognitive Tutor Center Carnegie
MellonUniversity
Educational Data Mining
http://ctat.pact.cs.cmu.edu
2012 Jasig Sakai Conference 11
Educational Data Mining:
A process for analyzing data collected during
teaching and learning to test learning theories and
inform educational practice
Bienkowski, Feng, & Means, 2012
2012 Jasig Sakai Conference 12
Understand entire systems and support human decision making
Applies known methods & models◦ answer questions about learning and
organizational learning systems
Tailored responses◦ adapted instructional content, specific
interventions, providing specific feedback
Learning Analytics
2012 Jasig Sakai Conference 13
Learning Analytics:
The use of analytic techniques to help target instructional, curricular, and support resources to support
the achievement of specific learning goals through
applications that directly influence educational practice
van Barneveld, Arnold, & Campbell, 2012adapted from Bach
2012 Jasig Sakai Conference 14
Predictive Analytics◦ uncover relationships and patterns◦ can be used to predict behavior and events
Visual Data Analytics◦ discovering and understanding patterns in large
datasets via visual interpretation
Additional Terms
2012 Jasig Sakai Conference 15
Terms are not ExclusiveTerm Definition Level of
Focus
Analytics An overarching concept that is defined as data-driven decision making
All levels
Academic Analytics
A process for providing higher education institutions with the data necessary to support operational and financial decision making
Institution
Educational Data Mining
A process for analyzing data collected during teaching and learning to test learning theories and inform educational practice
Department / Instructor / Learner
Learning Analytics
The use of analytic techniques to help target instructional, curricular, and support resources to support the achievement of specific learning goals through applications that directly influence educational practice
Department / Instructor / Learner
2012 Jasig Sakai Conference 16
Examples of LA ProjectsWho’s been working in this space in Higher Education?
17
Purdue University’s Course Signals
◦ College-wide learning analytics approach
University of Michigan’s E2Coach
◦ Course-specific learning analytics approach
UMBC’s “Check My Activity” Tool
◦ Student-centered learning analytics approach
2012 Jasig Sakai Conference
Three Different Approaches…
18
Built predictive model using data from…◦ LMS – Events (login, content, discuss.) & gradebook
◦ SIS – Aptitude (SAT/ACT, GPA) & demographic data
Leverage model to create Early-alert
system◦ Identify students at risk to not complete the course
◦ Deploy intervention to increase chances of success
Systems automates intervention process◦ Students get “traffic light” alert in LMS
◦ Messages are posted to student that
suggest corrective action (practice tests)
2012 Jasig Sakai Conference
Purdue University’s Course Signals
19
Impact on course grades and retention◦ Students in courses using Course Signals…
scored up to 26% more A or B grades up to 12% fewer C's; up to 17% fewer D's and F‘s
Ellucian product that integrates w/Blackboard
Open Academic Analytics Initiative (OAAI)◦ Creating a similar Sakai-based OS solution2012 Jasig Sakai Conference
Purdue University’s Course Signals
Arnold & Pistilli, 2012 - LAK
20
Focused specifically on introductory Physics Uses data from…
◦ Pre-course survey: academic info,
learner’s goals, psycho-social factors
◦ Performance: Exams, Web HW, Sakai
Michigan Tailoring System (MTS)◦ OS tool designed for highly customized messaging
◦ Used in health sciences for behavior change
◦ Messaging based on input from many sources
2012 Jasig Sakai Conference
University of Michigan’s E2Coach
“…to say to each what we would say if we could sit down with them for a
personal chat.”
2012 Jasig Sakai Conference 21
Example MTS Message
2012 Jasig Sakai Conference 22
UMBC found that students earning D/F’s use Bb 39% lessthen higher grade achievers◦ Not suggesting cause and effect◦ Goal is to model higher achiever
behavior Provides data directly student
◦ Compare LMS use to class averages◦ Can also compare averages usage
data to grade outcomes Feedback has been positive
UMBC’s Check My Activity Tool
2012 Jasig Sakai Conference 23
Student Success Plan – Sinclair CC◦ Holistic case-management system◦ Connects faculty, advisors, counselors, & students◦ Jasig Incubation Project
STAR Academic Journey – U of Hawaii◦ Online advising and degree attainment system
SNAPP – UBC/Wollongong◦ Visualize networks of
interaction resulting fromdiscussion forum posts andreplies
A Few Others Projects…
24
Papers and Articles on Purdue’s Course Signals http://www.itap.purdue.edu/learning/research/
Michigan’s Expert Electronic Coaching http://
sitemaker.umich.edu/ecoach/home
UMBC’s Check My Activity Tool http://
www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/Vi
deoDemoofUMBCsCheckMyActivit/219113
Student Success Plan http://
www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolu
m/TheStudentSuccessPlanCaseManag/242785
STAR Academy Journey http://
net.educause.edu/ir/library/pdf/pub7203cs7.pdf
SNAPP http://research.uow.edu.au/learningnetworks/seeing/snapp 2012 Jasig Sakai Conference
More information at…
2012 Jasig Sakai Conference 25
Data Available in SakaiWhat can we know in CLE and OAE products?
2012 Jasig Sakai Conference 26
User-level data stored as “events”
List of events available on Confluence◦ Search for “event table description”
CLE Data Overview
sakai_session SESSION_ID SESSION_USER SESSION_IP SESSION_USER_AGENT SESSION_START SESSION_END SESSION_SERVER SESSION_ACTIVE SESSION_HOSTNAME
sakai_eventEVENT_IDEVENT_DATEEVENTREFSESSION_ID
EVENT_CODECONTEXT
2012 Jasig Sakai Conference 27
Site-level data stored in separate tables
CLE Data Overview
sakai_siteCUSTOM_PAGE_ORDEREDSITE_IDTITLETYPESHORT_DESCDESCRIPTIONICON_URLINFO_URLSKINPUBLISHEDJOINABLEPUBVIEWJOIN_ROLECREATEDBYMODIFIEDBYCREATEDONMODIFIEDONIS_SPECIALIS_USER
sakai_realmREALM_KEYREALM_IDPROVIDER_IDMAINTAIN_ROLECREATEDBYMODIFIEDBYCREATEDONMODIFIEDON
realm_id like '/site/' || site_id
2012 Jasig Sakai Conference 28
F04
W05 F0
5W
06 F06
W07 F0
7W
08 F08
W09 F0
9W
10 F10
W11 F1
1W
120
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
Project Sites Logarithmic (Project Sites)Course Sites Logarithmic (Course Sites)Max Users Logarithmic (Max Users)Logarithmic (Max Users)
Thanks to John Leasia
Site Creation & Concurrent Users
2012 Jasig Sakai Conference 29
Overall Tool Usage
43%
25%
18%
3%
3% 2
%1% 1% 1%
Presence Web Content ResourcesAttachments Test Center AssignmentsSyllabus Forums GradebookDrop Box Evaluations MessagesPreferences Basic LTI Site Info / SetupRealms Announcements ChatCalendar Digest ChecklistiTunes U Help ModulesEmail Archive Course Eval Help PodcastsWiki Polls NewsSign Up Library Materials OSPDiscussion Engin Honor Code UbookLibrary Help Super User DB HelpSearch Page Order Global Alert
2012 Jasig Sakai Conference 30
Instructors Using / Not Using Sakai
2012 Jasig Sakai Conference 31
Dental School
2012 Jasig Sakai Conference 32
Subjects Using the Wiki Tool
Sum of RevisionsBITENGRNURSIOESIENGLISHRCHUMSAAASEECSRCLANGCOMPPSYCH
Count of Course Sites
SIENGLISHBITEECSPSYCHCOMPMODGREEKNURSNRELING
2012 Jasig Sakai Conference 33
Summary information about site visits, tool activity, and resource activity
Site Stats Tool
2012 Jasig Sakai Conference 34
Site Stats: Activity Detail
35
User-level data available via “activity feeds”◦ follows a “push and publication” model rather than
a “store and query” model (CLE is store & query)
◦ Activity is both highly specific: individual interactions between users, content, contexts…
◦ …and more general: user interaction everywhere rather than only within a single course context.
What new questions will we ask?◦ Interesting activity can happen with external
capabilities: CLE tools, LTI tools, widgets. How will we ensure this data is captured?
Analytics in OAE
2012 Jasig Sakai Conference
Many thanks to Nate Angell for OAE slides
2012 Jasig Sakai Conference 36
OAE activity stream design: feeds
FYI: Designs are still in draft form.
37
OAE activity stream design: user
2012 Jasig Sakai Conference
FYI: Designs are still in draft form.
38
OAE activity stream design: content
2012 Jasig Sakai Conference
FYI: Designs are still in draft form.
2012 Jasig Sakai Conference 39
Activity (OAE) & Grades (CLE): Week 1
Student success in OAE & CLE
Developed by the Kaleidoscope Project in collaboration with rSmart.
2012 Jasig Sakai Conference 40
Activity (OAE) & Grades (CLE): Week 7
Student success in OAE & CLE
Developed by the Kaleidoscope Project in collaboration with rSmart.
2012 Jasig Sakai Conference 41
Activity (OAE) & Grades (CLE): Animation
Student success in OAE & CLE
Developed by the Kaleidoscope Project in collaboration with rSmart.
2012 Jasig Sakai Conference 42
Tools / services to support analytics initiatives◦ Ways to connect different silos of data◦ Methods to connect back to CLE / OAE
LTI? Web services? Others?
OAE improvements over CLE approach to user data◦ What data is most relevant for analytics?◦ What displays and/or data are most useful to help
learners?
Developer Questions to Ponder
2012 Jasig Sakai Conference 43
Some Big LA Questions: Research & Ethics
Josh Baron, Marist College
2012 Jasig Sakai Conference 44
Data Mining vs. Learning Science Approaches◦ Do we build predictive models from large data sets or
from our understanding of learning sciences?◦ Is both the right answer? How does that work?
Challenges of Scaling LA Across Higher Ed◦ Does each institution have to build its own model?
How “portable” are predictive models?
◦ Do we need an open standard for LA? Could LIS and LTI play a role?
How can LA be used to assist ALL students?◦ Michigan’s E2Coach system is a good example
Big LA Research Questions
2012 Jasig Sakai Conference 45
“The obligation of knowing” – John Campbell◦ If we have the data and tools to improve student
success, are we obligated to use them? Consider This > If a student has a 13% chance of
passing a course, should they be dropped? 3%? Who owns the data, the student?
Institution?◦ Should students be allowed to “opt out”?
Consider This > Is it fair to the other students if by opting out the predictive model’s power drops?
What do we reveal to students? Instructors?
Consider This > If we tell a student in week three they have a 9% chance of passing, what will they do? Will instructors begin to “profile” students?
Big LA Ethical Questions
2012 Jasig Sakai Conference 46
Resources & Conference Sessions
Connect with Learning Analytics communities
2012 Jasig Sakai Conference 47
http://www.solaresearch.org/
Learning Analytics & Knowledge Conferences (LAK)
STORM – initiative to help fund research projects
FLARE – regional practitioner conference ◦ Purdue University, Oct 1-3, 2012
2012 Jasig Sakai Conference 48
Symposium on Learning Analytics at Michigan
http://sitemaker.umich.edu/slam/
15 speakers (12 UM, 3 external)
Videos & slides available from all speakers
2012 Jasig Sakai Conference 49
Analytics in Higher Education: Establishing a Common Language◦ Van Barneveld, Arnold, Campbell, 2012◦ http://www.educause.edu/Resources/AnalyticsinHigherEducationEsta/245405
Analytics to Literacies: Emergent Learning Analytics to evaluate new literacies◦ Dawson, 2011- http://blogs.ubc.ca/newliteracies/files/2011/12/Dawson.pdf
Learning Analytics: Definitions, Process Potential◦ Elias, 2011◦ http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf
The State of Learning Analytics in 2012: A Review and Future Challenges◦ Ferguson, 2012 - http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf
Academic analytics: A new tool for a new era. ◦ Campbell, Deblois, & Oblinger (2007). Educause Review, 42(4), 40-57. ◦ http://net.educause.edu/ir/library/pdf/ERM0742.pdf
Mining LMS data to develop an "early warning system" for educators: A proof of concept. ◦ Macfadyen & Dawson (2010) - Computers & Education, 54(2), 588-599.
Classroom walls that talk: Using online course activity data of successful students to raise self- awareness of underperforming peers. ◦ Fritz, 2011 - Internet and Higher Education, 14(2), 89-97.
Publications
50
Wednesday, 13 June◦ Learning Analytics: A Panel Debate on the Merits,
Methodologies, and Related Issues (1:15pm)
◦ Learning Analytics at Michigan: Designing Displays for Advisors, Instructors, and Students (2:30pm)
◦ BOF for Learning Analytics: Current and Planned Projects and Tools (3:45pm)
Thursday, 14 June◦ Creating an Open Ecosystem for Learner Analytics
(10:15am) Open Academic Analytics Initiative (OAAI) https://confluence.sakaiproject.org/x/8aWCB
2012 Jasig Sakai Conference
Related Conference Sessions
2012 Jasig Sakai Conference 51
Steve Lonn◦ [email protected] @stevelonn
Josh Baron◦ [email protected] @joshbaron
Questions?