Upload
alywise
View
245
Download
0
Embed Size (px)
Citation preview
DATA ARCHEOLOGY
Image Credit: Pedro Szekely via Flickr (CC BY 2.0), adapted
A T H E O R Y - I N F O R M E D A P P R O A C H T O A N A L Y Z I N G D A T A T R A C E S O F S O C I A L I N T E R A C T I O N I N L A R G E S C A L E L E A R N I N G E N V I R O N M E N T S
A LY S S A W I S ES I M O N F R A S E R U N I V E R S I T Y
OVERVIEW
DATA ARCHEOLOGY – THE BIG IDEA
APPLICATIONT H E R O L E O F “ L I S T E N I N G ” I N O N L I N E D I S C U S S I O N S
S O C I A L I N T E R A C T I O N I N L A R G E S C A L E L E A R N I N G E N V I R O N M E N T S
CONCLUSION
DATA ARCHEOLOGY
Image Credit: U.S. Army Corps of Engineers Europe District via Flickr (CC BY 2.0), adapted
( W I S E , 2 0 1 3 , 2 0 1 4 )
DATA ARCHEOLOGY
Image Credit: Pedro Szekely via Flickr (CC BY 2.0), adapted
( W I S E , 2 0 1 3 , 2 0 1 4 )
DATA ARCHEOLOGY
THEORETICALLY-INFORMED EFFORTS TO MAKE SENSE
OF THE DIGITAL ARTIFACTS LEFT BEHIND BY A PRIOR
LEARNING “CIVILIZATION”
MOVING BEYOND “MORE IS BETTER”
AS A LEARNING MODEL TO PROBE WHAT KINDS OF
THINGS ARE BETTER FOR WHAT PURPOSES AND WHY
THEORETICALLY-INFORMED
ATTENDING TO THE PEDAGOGICAL CONTEXT
AS A CRITICAL FRAME FOR INTERPRETING THE
PAST ACTIVITY THAT OCCURRED
LEARNING “CIVILIZATION”
THE COMPLETE AGGREGATED DATA
RECORD AVAILABLE AT THE END DOESN’T
REFLECT THE DYNAMIC ENVIRONMENT IN WHICH THE ACTIVITY OCCURRED
TEMPORALITY & TRAJECTORIES
W H AT TO O L S A R E T H E R E I N T H E O N L I N E E N V I R O N M E N T ?
(RE)FRAMING QUESTIONS
W H AT I S T H E P U R P O S E O F T H E L E A R N I N G A C T I V I T I E S C O N D U C T E D I N T H E TO O L S ?
H O W M U C H D O S T U D E N T S U S E T H E M ?
W H AT A R E T H E O R E T I C A L LY D E S I R A B L E PAT T E R N S O F PA R T I C I PAT I O N ?
H O W C A N T H E S E B E S T B E P R OX I E D BY T H E AVA I L A B L E D ATA?
F R OM
TO
F O R M O R E O N C O N N E C T I N G L E A R N I N G A N A L Y T I C S + L E A R N I N G D E S I G N S E E L O C K Y E R , H E A T H C O T E & D A W S O N [ 2 0 1 3 ]
AN ONLINE DISCUSSION FORUM IS A TOOL
IT ’S EDUCATIONAL PURPOSE CAN CHANGE
Q & A
Peer Review
Dialogue
Reading Response
Team Decision Making
Argumen-tation
ONLINE DISCUSSION LEARNING PURPOSE
Externalizing one’s ideas by contributing
posts to an online discussion
Taking in the externalizations of others by accessing
existing posts
• Social constructivist perspective - online discussions as a forum for learning through dialogue
• Learning occurs as students articulate their ideas, are exposed to the ideas of others, and negotiate differences in perspective
• Focus on how students contribute comments (“speak”), attend to other’s messages (“listen”), and the cxns bet them
UNDERLYING THEORY OF ONLINE “L ISTENING”
Listening not LurkingLurker• Specific person who participates
passively • Accesses existing comments but
does not contribute • Negative connotation
Listening • Active process conducted by anyone in
online discussion• Activity interrelated with contributing.• Productive element of discussion
participation
Listening • Specific term (online discussions)• Dynamic text, distinct sub-units • Multi-authored • Generating a response often involved
Reading• Generic term (all written text)• Static, cohesive text• Single author• Does not require response
Listening not Reading
Speaking
Mechanism for sharing ideas
Value in speaking that is Relevant to the topic at hand Rationaled with evidence Recurring and distributed Moderately portioned Responsive to the
conversation
Listening
Mechanism for becoming aware of ideas
Value in listening that is Broad (to consider a diversity of ideas) Deep (to consider ideas in earnest) Recursive (to provide context for
discussion flow) Integrated (attending to connected
rather than scattered comments)
ONLINE DISCUSSIONLEARNING MODEL
ONLINE DISCUSSION PEDAGOGICAL CONTEXT
• Group and Timing– Small group discussions (~8-12 students)
– Random assignment (would be better with differing perspectives)
– Discussions run on a weekly schedule with course
• Task– Contested real-world challenges (business, edu psychology)
– Given two viable contrasting perspectives, come to consensus
– Share decision with rationale with whole class
• Expectations– Given criteria / guidelines for speaking and listening
– Assessment varies (individual/group, student/instructor driven)
Listening
Mechanism for becoming aware of ideas
Value in listening that is Broad (to consider a diversity of ideas) Deep (to consider ideas in earnest) Recursive (to provide context for
discussion flow) Integrated (attending to connected
rather than scattered comments)
ONLINE DISCUSSIONLEARNING MODEL
Criteria Metric Definition
Breadth
% posts viewed Number of unique posts that a student viewed divided by the total number of posts in the discussion
% posts read Number of unique posts that a student read divided by the total number of posts in the discussion
Depth
% (real) reads Number of times a student viewed other’s posts at < 6.5 wps, divided by the total number of views
Av length of real reads (min)
Total time spent reading posts, divided by the number of reads (after scans removed )
Recursiveness # of reviews of others’ posts
Number of times a student revisited posts that they had viewed previously in the discussion
Integration Posts read connected, not scattered
Concentration of posts viewed by a student in the discussion space* [thread-density, network metrics..]
ONLINE DISCUSSIONLISTENING ANALYTICS
Speaking
Mechanism for sharing ideas
Value in speaking that is Relevant to the topic at hand Rationaled with evidence Recurring and distributed Moderately portioned Responsive to the
conversation
ONLINE DISCUSSIONLEARNING MODEL
Criteria Metric Definition
Recurring
Number of posts Total number of posts a student contributed to the discussion
Percent of sessions with posts
Number of sessions in which a student made a post, divided by their total of number sessions
Moderately Portioned Average post length Total number of words posted by a student divided by
the number of posts they made to the discussion
Responsive Depth of response to existing conversation
0 None1 Acknowledging2 Responding to an idea3 Responding to multiple ideas
Rationaled Degree of argumentation
0 No argumentation1 Unsupported argumentation (Position only)2 Simple argumentation (Position + Reasoning/Evidence)3 Complex argumentation (Position + Reasoning /Evidence+ Qualifier/Rebuttal)
ONLINE DISCUSSIONSPEAKING ANALYTICS
SOME RESULTS[ W I S E , H S I A O E T A L . 2 0 1 2 ][ W I S E , P E R E R A E T A L . , 2 0 1 2 ][ W I S E , S P E E R E T A L . 2 0 1 3 ]
DepthBreadth (% of posts viewed)
Low High
Low Disregardful Coverage
High Focused ThoroughUn-engaged Engaged
SOME RESULTS
Depth (% of real reads)
Breadth (% of posts viewed)
Low High
Low Disregardful Coverage
High Focused Thorough
Un-engaged
Engaged
SOME MORE RESULTS[ W I S E , H A U S K N E C H T & Z H A O , 2 0 1 4 ]
Greater Listening Depth (% of real reads)
Listening Recursiveness(# reviews of others’ posts)
Associated with More Rationaled Speaking
More Responsive Speaking
Listening Breadth not associated with any speaking qualities in the study. Less important for current pedagogical design?
FLESHING OUT TYPOLOGIES
Pattern Characteristic Behaviors
Disregardful Minimal attention to others’ posts (few posts viewed; short time viewing). Brief and relatively infrequent sessions of activity in discussions.
CoverageViews a large proportion of others’ posts, but spends little time attending to them (often only scanning the contents). Short but frequent sessions of activity, focusing primarily on new posts. *May be socially-oriented or content-driven.
FocusedViews a limited number of others’ posts, but spends substantial time attending to them. Few extended sessions of activity in discussions.
ThoroughViews a large proportion of other’s posts; spends substantial time attending to many of them. Long overall time spent listening; considerable revisitiation of posts already read.
GROUND TRUTHING VIA TEMPORAL MICROANALYTIC CASE STUDIESDate Time Session Action Duration
(min)Length(words)
Message #
6/3/2011 23:46 1 Read 44.43 413 447
6/3/2011 23:52 1 Read 1.73 60 455
6/4/2011 00:08 1 Scan 0.23 117 459
6/4/2011 00:09 1 Read 12.51 413 460
6/4/2011 23:49 2 Post 3.18 120 477
TAKEAWAYATTENDING TO THE
PEDAGOGICAL CONTEXT OF DISCUSSION FORUM USE AND
CRAFTING THEORETICALLY INFORMED METRICS LET US EXTRACT EXPLANATORY AND ACTIONABLE INFORMATION
FROM THE CLICKSTREAM DATA
SOCIAL INTERACTION
IN LARGE SCALE
LEARNING
ENVIRONMENTS
W I T H T H A N K S TO M Y R E S E A R C H
A S S I S TA N T
Y I C U I
CHALLENGES WE SET OURSELVES FOR LOOKING AT THE MOOC DATALO O K AT S O C I A L I N T E R A C T I O N
A D D R ES S I S S U E S O F S C A L E
E M P LOY N AT U R A L L A N G UA G E P R O C E S S I N G *
AT T E N D TO P E DA G O G I C A L CO N T E X T
WO R K I N A T H EO RY- I N F O R M E D WAY
SOCIAL INTERACTION IN MOOC FORUMS
S T R O N G P R E D I C T O R O F P E R S I S T E N C E B U T T H I S M AY B E B E C A U S E I T I N D E X E S ( R AT H E R T H A N C A U S E S ) E N G A G E M E N T – W H AT A B O U T L E A R N I N G ?
C L A I M E D T O P R O V I D E C R I T I C A L S O C I A L L E A R N I N G S U P P O R T B U T W I T H O U T T I E S T O T H E A C A D E M I C C O N T E N T, S O C I A B I L I T Y M AY N O T I M PA C T L E A R N I N G[ K U H , 2 0 0 2 ; W I S E , D E L V A L L E , C H A N G & D U F F Y , 2 0 0 4 ]
O N LY A S M A L L % PA R T I C I PAT E B U T T H I S I S N O T S U R P R I S I N G I F I T I S N O T D E S I G N E D I N T O A C O U R S E . H O W P E O P L E PA R T I C I PAT E I S A S I M P O R TA N T A S I F T H E Y D O S O .
WHY FOCUS ON LEARNING NOT AT TRIT ION?
S T R O N G N E E D T O R E C O N C E P T U A L I S E P E R S I S T E N C E A N D AT T R I T I O N I N M O O C S G I V E N T H E N U M B E R O F P E O P L E W H O R E G I S T E R W / O “A N I N F O R M E D C O M M I T M E N T T O C O M P L E T E T H E C O U R S E ” [ D E B O E R , H O , S T U M P & B R E S L O W , 2 0 1 4 ]
G R E AT V A R I E T Y I N I N T E N T I O N S , W O R K I N G PAT T E R S , R E S O U R C E S U S E D , S E Q U E N C E A N D F R E Q U E N C Y O F U S E [ D E B O E R E T A L . , 2 0 1 4 ; K I Z I L C E C , P I E C H & S C H N E I D E R , 2 0 1 3 ]
J U S T I N D E X I N G L E V E L O F E N G A G E M E N T T O P R E D I C T W H O W I L L S T O P PA R T I C I PAT I N G D O E S N ’ T T E L L U S W H Y O R H O W T O I N T E R V E N E
L E A R N I N G M AY B E O C C U R R I N G E V E N F O R T H O S E W H O D O N ’ T E V E N T U A L LY C O M P L E T E
WHY FOCUS ON LEARNING NOT AT TRIT ION?
S T R O N G N E E D T O R E C O N C E P T U A L I S E P E R S I S T E N C E A N D AT T R I T I O N I N M O O C S G I V E N T H E N U M B E R O F P E O P L E W H O R E G I S T E R W / O “A N I N F O R M E D C O M M I T M E N T T O C O M P L E T E T H E C O U R S E ” [ D E B O E R , H O , S T U M P & B R E S L O W , 2 0 1 4 ]
G R E AT V A R I E T Y I N I N T E N T I O N S , W O R K I N G PAT T E R S , R E S O U R C E S U S E D , S E Q U E N C E A N D F R E Q U E N C Y O F U S E [ D E B O E R E T A L . , 2 0 1 4 ; K I Z I L C E C , P I E C H & S C H N E I D E R , 2 0 1 3 ]
J U S T I N D E X I N G L E V E L O F E N G A G E M E N T T O P R E D I C T W H O W I L L S T O P PA R T I C I PAT I N G D O E S N ’ T T E L L U S W H Y O R H O W T O I N T E R V E N E
L E A R N I N G M AY B E O C C U R R I N G E V E N F O R T H O S E W H O D O N ’ T E V E N T U A L LY C O M P L E T E
“I'm very happy to be in this course. I [couldn’t] finish it on
time, but I think I have learnt a lot. Thank you Prof X, you
are a great teacher, very [professional], excellent in many
ways. I will miss you!”
SOCIAL INTERACTION IN MOOC FORUMS
S T R O N G P R E D I C T O R O F P E R S I S T E N C E B U T T H I S M AY B E B E C A U S E I T I N D E X E S ( R AT H E R T H A N C A U S E S ) E N G A G E M E N T – W H AT A B O U T L E A R N I N G ?
C L A I M E D T O P R O V I D E C R I T I C A L S O C I A L L E A R N I N G S U P P O R T B U T W I T H O U T T I E S T O T H E A C A D E M I C C O N T E N T, S O C I A B I L I T Y M AY N O T I M PA C T L E A R N I N G[ K U H , 2 0 0 2 ; W I S E , D E L V A L L E , C H A N G & D U F F Y , 2 0 0 4 ]
O N LY A S M A L L % PA R T I C I PAT E B U T T H I S I S N O T S U R P R I S I N G I F I T I S N O T D E S I G N E D I N T O A C O U R S E . H O W P E O P L E PA R T I C I PAT E I S A S I M P O R TA N T A S I F T H E Y D O S O .
FRAMING QUESTIONS
W H AT WA S T H E P E D A G O G I C A L P U R P O S E / D E S I G N O F T H E D I S C U S S I O N F O R U M S I N T H E P S YC H M O O C ?
B A S E D O N T H I S , W H AT W E R E T H E O R E T I C A L LY D E S I R A B L E PAT T E R N S O F PA R T I C I PAT I O N ?
H O W C A N T H E S E B E S T B E P R OX I E D BY T H E AVA I L A B L E D ATA?
H O W C O U L D T H E D E S I R E D PAT T E R N S B E B E T T E R S U P P O R T E D ?
MOOCPEDAGOGICAL CONTEXT
• Course Topic– Introductory Psychology
• Level and Expected Background– Designed for college freshmen– Equivalent of high school education expected– No specific prior knowledge indicated
• Course Design– Video lectures (8-15 min long)– Readings from OLI (Open Learning Initiative) online textbook – Weekly timed multiple choice quiz– Final exam at the end of the course
WHAT ABOUT THE DISCUSSION FORUMS?
• Optional part of the course, main pedagogical design a Q&A forum to ask and answer questions about course material
Communication“There will be a Q&A forum where you can post your questions about the course. Students will have the opportunity to "vote up" questions they want answered, and the questions with the most votes will be answered either in a forum post or a video.”
….
Expectations“Participants are expected to seek help if needed from your fellow students by using the forums”
RECREATED STUDENT FORUM VIEW
ForumsWelcome to the course discussion forums.
Sub-forum Activity
General DiscussionDiscuss general aspects of the course. Q&A Ask and answer questions about course material.
Assignments Discuss details of the course assignments.
Technical Issues Post any issues with, or questions about, technical aspects of the course website (trouble with video playback, broken links, etc.).
OLI Textbook Questions Post any issues with, or questions about, technical aspects of the OLI Textbook.
Student BiosIntroduce yourself and learn about other students.
RECREATED STUDENT FORUM VIEW
Notes: [1] Counts only include non-deleted posts/threads [2] Counts taken prior to data cleaning, may include duplicate or nonsense posts
ForumsWelcome to the course discussion forums.
Sub-forum ActivityThreads (Posts + Comments)
General DiscussionDiscuss general aspects of the course. 289 (1341+804)
Q&A Ask and answer questions about course material. 158 (525+204)
Assignments Discuss details of the course assignments. 147 (827+775)
Technical Issues Post any issues with, or questions about, technical aspects of the course website (trouble with video playback, broken links, etc.).
108 (318+79)
OLI Textbook Questions Post any issues with, or questions about, technical aspects of the OLI Textbook. 99 (347+106)
Student BiosIntroduce yourself and learn about other students. 662 (1614+354)
PROCESS CHECKD O E S I T M A K E S E N S E T O U S E L I S T E N I N G A N D
S P E A K I N G T H E O R Y I N T H I S C O N T E X T ?
• P E D A G O G I C A L C H A L L E N G E– M O S T O F T H E D I S C U S S I O N I S N ’ T C O N T E N T
R E L AT E D , L I S T E N I N G I S N ’ T E X P E C T E D T O R E L AT E T O L E A R N I N G
• T E C H N I C A L C H A L L E N G E– L O W G R A N U L A R I T Y D ATA ( “ V I E W. F O R U M ” &
“ V I E W.T H R E A D ” V S . “ V I E W. P O S T ”, T H O U G H “ V O T E . U P ” N O W A V A I L A B L E )
• P R A C T I C A L C H A L L E N G E– M A N Y T H R E A D S I N Q & A F O R U M N O T A C T U A L LY
C O N T E N T
CHANGING TRACKSA W H O L E B U N C H O F Q U E S T I O N S W E
T H O U G H T W E W E R E G O I N G T O A S K W E N T O U T T H E W I N D O W …
• N E W ( V E R Y B A S I C ) F O C U S O N I F T H E PAT T E R N S O F F O R U M U S E M AT C H E D T H O S E D E S I R E D F O R T H E I N T E N D E D P U R P O S E– D I D S T U D E N T S U S E T H E Q & A F O R U M T O A S K
Q U E S T I O N S A B O U T T H E C O U R S E M AT E R I A L ?– D I D T H E I N S T R U C T O R S R E P LY T O ( T H E
H I G H E S T V O T E D ) Q U E S T I O N S A B O U T T H E C O U R S E M AT E R I A L ?
DID STUDENTS USE THE Q&A FORUM TO ASK QUESTIONS ABOUT THE COURSE MATERIAL?
• After preliminary inspection we decided to code both the Q&A and General Discussion (GD) forums b/c no clear fxnl difference was seen
• Two raters coded the starting post in each thread as either– Content [C] (Asking questions about course material, expanding on
course content; discussing a resource shared)– Non-Content [X] (Including logistics, social, study group formation and
link sharing)
• 439 of 447 total threads coded – 8 removed for foreign language or complete nonsense contents– 92% agreement (k=0.81), All difference reconciled, rule of leniency
Image: So Many MOOCs by mksmith23, CC by 2.0 license
DID STUDENTS USE THE Q&A FORUM TO ASK QUESTIONS ABOUT THE COURSE MATERIAL?
Content Threads Non-Content Threads
General Discussion 55 226
Q&A 68 90
Total 123 (28%) 316 (72%)
Image: So Many MOOCs by mksmith23, CC by 2.0 license
D I D T H E I N S T RU C TO R S R E P LY TO ( T H E H I G H E S T VOT E D ) Q U E S T I O N S A B O U T T H E C O U R S E M AT E R I A L ?
• First approach: “Instructor Replied” label [problematic]
• 2 “official” Instructor IDs (threads automatically labelled)– Course Professor [2XXXXX4]
– Course TA [5XXXX1]
• 1 “unofficial” Instructor ID (threads not automatically labelled)– Course Professor [2XXXXX0]
“[Yes,] I really am NAME2XXXXX0 (XXXX is my first name) and am the instructor for the course. I've been at UNIVERSITY for 43 years and love teaching. This course was a challenge because there was no feedback from students when the modules were being taped. The lack of student interaction is the real challenge of a MOOC. Just looking at a camera is a very different context than looking at a classroom of bright students. NAME2XXXXX0, Instructor”
Image: So Many MOOCs by mksmith23, CC by 2.0 license
Forum ThreadsInstructor Replied
(All 3 IDs)Content Threads
Replied by instructor
% of Instructor Replies
Directed at Content
General Discussion 289 31 (11%) 55 5 (9%) 16%
Q&A 158 31 (20%) 68 17 (25%) 55%
Total 447 62 (14%) 123 22 (18%) 35%
D I D T H E I N S T RU C TO R S R E P LY TO ( T H E H I G H E S T VOT E D ) Q U E S T I O N S A B O U T T H E C O U R S E M AT E R I A L ?Image: So Many MOOCs by mksmith23, CC by 2.0 license
Instructor Replied (62 threads)
Non-Replied (377 threads)
Average # (range) of votes 2.6 (0 to 30) 1.7 (-13 to 45)
Av # (range) of posts+comments 8.6 (2-110) 6.2 (1-92)
Av # (range) views 110 (25-1185) 75 (5-1143)
Content Threads Non-Content Threads
Av # votes 1.2 2.1Av # posts+comments 4.1 7.5
Av # views 51 91
D I D T H E I N S T RU C TO R S R E P LY TO ( T H E H I G H E S T VOT E D ) Q U E S T I O N S A B O U T T H E C O U R S E M AT E R I A L ?Image: So Many MOOCs by mksmith23, CC by 2.0 license
A CORE CHALLENGE FOR SOCIAL INTERAC TION AT SCALE
• Too much quantity, not enough quality
• Students get lost / overwhelmed in the abundance of communication
• Instructors too, challenging to find where their input is needed
• A need to separate “the wheat from the chaff”
Image: So Many MOOCs by mksmith23, CC by 2.0 license
CAN NATURAL LANGUAGE PROCESSING HELP?
• Goal to support the instructor in finding content threads more efficiently in the forums to be able to respond and facilitate learning
• A modest attempt to build a proof-of-concept model– Feature extraction performed with basic bag-of-words feature set
(inc. bigrams, trigrams and parts-of-speech tagging), rare threshold of 5
– Unigrams and bigrams alone most useful to characterize and model posts
– Total of 1573 features extracted
Image: So Many MOOCs by mksmith23, CC by 2.0 license
CHARAC TERIST IC FEATURES
Feature Kappabut 0.25
more 0.24
by 0.24
in_the 0.24
why 0.24
as 0.23
what 0.22
is 0.22
that 0.21
or 0.20
then 0.20
in 0.19
when 0.18
of_the 0.17
of 0.17
Feature Kappaand_the 0.16
question 0.16
between 0.16
age 0.16
correct 0.16
than 0.16
were 0.15
by_the 0.15
an 0.15
answer 0.15
does 0.14
mental 0.14
research 0.14
to_the 0.14
their 0.14
Feature Kappacourse 0.12
i 0.09
my 0.08
this_course 0.07
final 0.06
BOL_i 0.06
quiz 0.06
the_course 0.06
exam 0.05
thanks 0.05
i_am 0.05
videos 0.04
grade 0.04
certificate 0.04
BOL_hi 0.04
Feature KappaBOL_hello 0.04
hello 0.04
everyone 0.04
will 0.04
final_exam 0.04
i_just 0.04
hi 0.04
i_can 0.04
find 0.04
coursera 0.03
courses 0.03
i_have 0.03
grades 0.03
material 0.03
quizzes 0.03
Content Threads Non-Content ThreadsImage: So Many MOOCs by mksmith23, CC by 2.0 license
PREDIC TING CONTENT POSTS
• Procedure– Algorithm: Support Vector Machines– Setting for Nominal Class Values: LibLINEAR– Cross-validation, 10 randomly generated folds
• Results– Best Model Accuracy/Kappa = 0.86/0.64– Recall = 0.71 (False Neg. rate = 0.29)– Precision = 0.76
Image: So Many MOOCs by mksmith23, CC by 2.0 license
STANDARD FORUM INDICATORS DON’ T HELP IDENTIFY CONTENT
Accuracy Kappa Recall Precision
Base Model 0.86 0.64 0.71 0.76
Addition of Standard Forum Indicators
# votes 0.84 0.60 0.68 0.74
# posts 0.85 0.62 0.69 0.75
# views 0.85 0.62 0.69 0.76
Image: So Many MOOCs by mksmith23, CC by 2.0 license
INSTRUC TOR PERSPEC TIVE
W/o Content Model(Default)
With Content Model
Total Number of Potential Content Threads to Read
439 (37/wk on av)
114 (10/wk on av)
Percent of Threads Actually About Course Content 28% 76%
Percent of Content Threads With Instructor Replies 18% >18%?
Percent of Instructor Replies Addressing Content 35% >35%?
Image: So Many MOOCs by mksmith23, CC by 2.0 license
STRENGTHS, L IMITATIONS & FUTURE OPPORTUNIT IES
• Design of forums can improve, but unexpected use will still happen. For instructors to facilitating learning, the first step is to locate where learning opportunities are happening – content modeling can help.– Aligns well with Coursera’s development of content / logistics TAs.
• Model is simple but useful, more sophisticated modelling can improve these results.
• Model built with only 439 starting posts, including all the posts could lead to both better prediction of if a post is content-related and more nuanced assessment of threads (e.g. “This thread is estimated to have 87% content-related posts)
• Model seemed not to draw heavily on domain-specific vocabulary but may rely on domain-specific discourse types (extensibility to other social sciences but perhaps not humanities / hard sciences)
Image: So Many MOOCs by mksmith23, CC by 2.0 license
TAKEAWAYAT TENDING TO THE PEDAGOGICAL CONTEXT OF D ISCUSS ION FORUM USE AND GET TING CLOSE TO THE DATA LET US DEVELOP A S IMPLE YET APPROPRIATE AND USEFUL
MODEL - SUPPORTING CONTENT RELATED LEARNING DISCUSS ION
MAY BE PRE-REQUIS ITE TO STUDYING MORE COMPLEX FACETS
OF INTERACTION LARGE SCALE LEARNING ENVIRONMENTS
A DATA ARCHEOLOGY APPROACH THAT PAYS
ATTENTION TO THE LEARNING “CIVIL IZATION” THAT CREATED THE DATA AND POSITS THEORY-
INFORMED PATTERNS OF BEHAVIOR CAN HELP US
BETTER UNDERSTAND AND SUPPORT SOCIAL INTERACTION
IN LARGE SCALE LEARNING ENVIRONMENTS
CONCLUSION
DATA ARCHEOLOGY
Image Credit: Pedro Szekely via Flickr (CC BY 2.0), adapted
A T H E O R Y - I N F O R M E D A P P R O A C H T O A N A L Y Z I N G D A T A T R A C E S O F S O C I A L I N T E R A C T I O N I N L A R G E S C A L E L E A R N I N G E N V I R O N M E N T S
A LY S S A W I S ES I M O N F R A S E R U N I V E R S I T Y