Embed Size (px)
Citation preview
Making Sense of Online Learning: Frames, Rubrics, Tools & Coding Systems for Analyzing
Asynchronous Online Discourse
Theresa FlynnPepperdine [email protected]
Linda PolinPepperdine [email protected]
Co-construction of Knowledge & Asynchronous Online Discourse
ASSUMPTIONS:
collaborative learning is happening
online
Asynchronous discourse is a rich source of data for
analysis
Research Questions
• Can we capture evidence of learning in asynchronous online settings?
• If so, how?
• Are there indicators that “deep learning” is
occurring in asynchronous learning settings?
Study Context
• Pepperdine University’s 90% Online Masters in Educational Technology Program—2002/2003
• 5th year of the program
• 3 trimesters July 2002—July 2003
• 23 students—across the USA and abroad
• Only 3 face-to-face meetings in the 13 month program
Rubric Development
• Audience– To whom are messages being addressed in
newsgroups?
• Discourse Function– What roles do different “types” of posts serve
in the learning process?
Audience Rubric
• Self
• Specific Person(s)
• Instructor
• Group
• Other
Discourse Functions Rubric
• Rumination
• Storytelling
• Disagreement/Argumentation
• Social Interaction
• Procedural/Logistical
Discourse Functions Rubric
• Acknowledgement
• Reference/Resource
• Inquire
• Other
Thread Selection
• 2 threads from 1st trimester
• 2 threads from 2nd trimester
• Length of threads—looked for active posting
• Topic:• 2 threads about course readings (chapters)• 2 threads about class process (asynchronous &
synchronous classes)
Results of Coding: Audience
Chapter Threads
Chapter 4: Workplace Settings
Group
38%
Specific Person(s)
62%
Chapter 3: Schooling
Group
50%
Specific Person(s)
50%
Process-Oriented ThreadsSynchronous Chats
Group
40%
Specific Person(s)
60%
Posting Protocol
Specific Person(s)
43%
Instructor
4%
Group
53%
Results of Coding: Discourse Functions
Instructor: A
Trimester 1
# of messages: 29
Thread Length: 20 days
# of Unique Participants: 15
Total # of class members: 23
Chapter 4: Workplace Settings
Rumination
24%
Storytelling
44%
Social Interaction
3%
Acknowledgement
5%
Reference/Resource
3%
Inquiry
21%
Rumination
Storytelling
Argumentation
Social Interaction
Procedural/Logistical
Acknowledgement
Reference/Resource
Inquiry
Other
Instructor: B
Trimester 2
# of messages: 28
Thread Length: 26 days
# of Unique Participants: 17
Total # of class members: 23
Chapter 3: Schooling
Rumination34%
Storytelling32%
Social Bonding4%
Acknowledgement4%
Reference/Resource13%
Inquiry11%
Other2%
Rumination
Storytelling
Argumentation
Social Bonding
Procedural/Logistical
Acknowledgement
Reference/Resource
Inquiry
Other
Instructor: A
Trimester 1
# of messages: 16
Thread Length: 8 days
# of Unique Participants: 11
Total # of class members: 23
Synchronous Chats
Rumination
14%
Storytelling
5%
Procedural/Logistical
24%
Acknowledgement
28%
Reference/Resource
5%
Inquiry
10%
Other
0%
Social Interaction
14%
Rumination
Storytelling
Argumentation
Social Bonding
Procedural/Logistical
Acknowledgement
Reference/Resource
Inquiry
Other
Instructor: B
Trimester: 2
# of messages: 29
Thread Length: 11 days
# of Unique Participants: 22
Total # of class members: 23
Posting Protocol
Procedural/Logistical
33%
Acknowledgement
13%
Reference/Resource
0%
Inquiry
10%
Social Interaction
38%
Rumination
3% Storytelling
3%
Rumination
Storytelling
Argumentation
Social Bonding
Procedural/Logistical
Acknowledgement
Reference/Resource
Inquiry
Other
Validity: Interviews
• 8 students interviewed
• All agreed with the categories presented in both rubrics
• “The rubric covers it—You’ve done a fantastic job framing all of this. I never thought it had structure”
Validity: Interviews• “I think [the self category] is right on track—these are the
posts that usually trigger the best discussions—but also sometimes the hardest to share”
• “Very rarely do I feel that I am writing for the benefit of the professor”
• “When I ruminate it’s because I’ve reflected on something. When I’m storytelling it’s because I’m reflecting on a previous experience and how it’s linked to the current learning experience. When I’m disagreeing it’s because I’m reflecting on how and what you’re saying jives or doesn’t jive with a previous belief I have. In other words, everything that I’m saying in newgroups is a form of reflection”
Reliability
• 2 coders
• Inter-rater agreement
• 89% agreement in Discourse Functions category
• 94.5% agreement in Audience category
Disagreement/Argumentation
• No messages coded as disagreement/argumentation in any thread
• Closer-examination revealed we had missed several instances of disagreement
• Dissent was subtle
• Only apparent when examining the entire context of the newsgroup thread
Disagreement/Argumentation: A Closer Look at the Text
• Tension between “Old-timers” & “Newcomers”
• Sometimes “newcomers” leave or “jump ship”
• “Do students have the option of jumping ship?”
• MLK reference
• “But what do you do as the professional stagnating in a place unwilling to change?”
Conclusion
• Don’t sacrifice validity for reliability
• Large degree of inference involved in coding messages—makes coding more difficult
• The message alone as a unit of analysis is insufficient
• Must consider the context of the discourse
Student Insight
The most important part of the newsgroup process is not the reading or the posting—it’s the part in between where I’ve read it, and then I reflect on it before I post a reply. It’s that little, in that processing place”.
Context
AND
Content
