Upload
rebecca-ferguson
View
1.391
Download
6
Tags:
Embed Size (px)
DESCRIPTION
Social learning analytics are concerned with the process of knowledge construction as learners build knowledge together in their social and cultural environments. One of the most important tools employed during this process is language. In this presentation we take exploratory dialogue, a joint form of co-reasoning, to be an external indicator that learning is taking place. Using techniques developed within the field of computational linguistics, we build on previous work using cue phrases to identify exploratory dialogue within online discussion.
Citation preview
An Evaluation of Learning Analytics To Identify Exploratory Dialogue in Online Discussions
Rebecca Ferguson, The Open University, UK
Zhongyu Wei, The Chinese University of Hong Kong
Yulan He, Aston University, UK
Simon Buckingham Shum, The Open University, UK
Discourse analytics
The ways in which learners engage in dialogue indicate how they engage with the ideas of others, how they relate those ideas to their understanding and how they explain their own point of view.
• Disputational dialogue• Cumulative dialogue• Exploratory dialogue
Exploratory dialogueCategory Indicator
Challenge But if, have to respond, my view
Critique However, I’m not sure, maybe
Discussion of resources Have you read, more links
Evaluation Good example, good point
Explanation Means that, our goals
Explicit reasoning Next step, relates to, that’s why
Justification I mean, we learned, we observed
Reflections of perspectives of others
Agree, here is another, makes the point, take your point, your view
Pilot study: LAK 2011Time Contribution
2:42 PM I hate talking. :-P My question was whether "gadgets" were just basically widgets and we could embed them in various web sites, like Netvibes, Google Desktop, etc.
2:42 PM Thanks, that's great! I am sure I understood everything, but looks inspiring!
2:43 PM Yes why OU tools not generic tools?
2:43 PM Issues of interoperability
2:43 PM The "new" SocialLearn site looks a lot like a corkboard where you can add various widgets, similar to those existing web start pages.
2:43 PM What if we end up with as many apps/gadgets as we have social networks and then we need a recommender for the apps!
2:43 PM My question was on the definition of the crowd in the wisdom of crowds we acsess in the service model?
2:43 PM there are various different flavours of widget e.g. Google gadgets, W3C widgets etc. SocialLearn has gone for Google gadgets
Computational linguistics
Interdisciplinary field thatdeals with statistical and rule-based modelling of natural language from a computational perspective
Zhongyu Wei Yulan He
Three challenges
1. The annotated dataset is limited
2. Text classification problems are typically topic driven – this is not
3. Nevertheless, both dialogue features and topical features need to be taken into account
Self-training from labelled instances – a problem
Pseudo-label
Exploratory
Pseudo-label
Exploratory
Pseudo-label
Exploratory
Pseudo-label
Exploratory
✓
✓
✓
✗Including this
instance would degrade the
classifier
• For each turn in the dialogue, consider each unigram (word), bigram (2 words) and trigram (3 words)
• Exploratory or non-exploratory?• Take into account word-association probabilities
averaged over many pseudo-labelled examples
Self-training from labelled features
Pseudo-labelNon-
exploratory
✓Focusing on
features gives a more reliable classification
Bigrams
To improve labelling, take into account the classification of a number (k) of the nearest neighbours of that turn in the dialogue
Taking context into account
Pseudo-label
Non-
exploratory
Unlabelled turn in the dialogue, p1
Pseudo label for that turn, l1
Confidence value for that label, c1
0.272727271
Nearest neighbour pni1
Pseudo label lni1
Confidence level cni1
Nearest neighbour pni3
Pseudo label lni3
Confidence level cni3
Nearest neighbour pni2
Pseudo label lni2
Confidence level cni2
Let k = 3(look at 3 nearest
neighbours)
Pseudo-label
Non-
exploratory
Pseudo-label based on features is considered correct if support value (s)
based on context is high enough
Checking against context
Support value is calculated by taking into account the pseudo labels and confidence values of k nearest neigbours
?
Checking the pseudo-labels
Pseudo-label
Exploratory
Nearest neighbour 1Confidence level 0.333Pseudo-label
Non-
exploratory
Nearest neighbour 2Confidence level 0.999
Nearest neighbour 3Confidence level 0.666
s = 0.333 + 0 + 0.666
3
Because s < Rthis turn in the dialogueshould not be labelled ‘non-exploratory’
If the support valuefor this pseudo labelis greater than R then this turn in the dialogue can be labelled ‘non-exploratory’
Let R = 0.5
Pseudo-label
Non-
exploratory
Pseudo-label
Non-
exploratory
s = 0.333
?
Cue phrases from pilotAgreeAlsoAlthoughAlternativeAny researchAre weBecauseBut ifChallengeClaimDebateDefinitelyDependsDifficultDiscussionDo we haveDo youDoes that mean
Does this suggestDraftEvidenceExampleExceptMisunderstandingGood exampleGood pointGood thing aboutHave weHave you looked atHave you readHere is anotherHow areHow can[...]WhyYour view
94 cue phrases•Precise but•Low recall
Used to improve accuracy when classifying unannotated dataset
Dataset
Annotated•Elluminate text chat•Two-day conference•2,636 dialogue turns•Mean word tokens per turn: 10.14
Unannotated•Elluminate text chat•Three MOOCs•10,568 dialogue turns•Mean word tokensper turn: 9.24
Time Contribution
2:43 PM
Issues of interoperability
2:43 PM
The "new" SocialLearn site looks a lot like a corkboard where you can add various widgets, similar to those existing web start pages.
2:43 PM
What if we end up with as many apps/gadgets as we have social networks and then we need a recommender for the apps!
2:43 PM
My question was on the definition of the crowd in the wisdom of crowds we acsess in the service model?
Manual coding of data subset
Category Description Examples include
Challenge A challenge identifies something that may be wrong and in need of correction
Calling into questionContradictingProposing revision
Evaluation An evaluation has a descriptive quality
AppraisingAssessingJudging
Extension An extension builds on, or provides resources that support, discussion
Applying idea to a new areaIncreasing range of an ideaProviding related resources
Reasoning Reasoning is the process of thinking an idea through
ExplainingJustifying your positionReaching a conclusion
Combining methods
• Train initial classifier on annotated dataset• Apply trained classifier to un-annotated data• Use self-learned features to find exploratory dialogue• Use cue-phrase matching to improve accuracy• Take context into account using k-nearest neighbours• Add selected instances to the training dataset
• Repeat for five iterations or until less than 0.5% of labels are changed
Evaluation criteriaOn a scale of 0 to 1…
Accuracy How many decisions were correct?Pilot 0.5389 SF+CP+KNN = 0.7924
PrecisionHow many ‘exploratory’ turns were actually exploratory?Pilot 0.9523 SF+CP+KNN = 0.8083
Recall How many exploratory turns were classified as exploratory?Pilot 0.4241 SF+CP+KNN = 0.8688
F1
Weighted average of precision and recallPilot 0.5865 SF+CP+KNN = 0.8331
Varying the value of k
k Accuracy Precision Recall F1
1 0.7868 0.8007 0.8666 0.8282
3 0.7924 0.8083 0.8688 0.8331
5 0.7881 0.8005 0.8685 0.8292
7 0.7586 0.7505 0.8640 0.8001
Looking at three nearest neighbours gives best results
Making use of the classifier
Each colour block represents 10 turns in the dialogueRed blocks are primarily exploratory, blue blocks primarily non-exploratory
Making use of the classifier
Total turns in the dialogue
Exp
lora
tory
tur
ns in
the
dia
logu
e
The line here is set to highlight anyone who had more than5/6 of their turns classified as exploratory
Analytics like these could be used to provide focused support to learners
Issues
Total turns in the dialogue
Exp
lora
tory
tur
ns in
the
dia
logu
e
Visual literacyHow can we share the maximum amount of information while making these analytics easy to use?
Assessment for learningHow can we use these analyticsto motivate and guide, rather than to discourage?
Participatory design How can we involve learners and teachers in learning discussions around these analytics?
Working in the middle space
Conclusion
• We proposed and tested a self-training framework
• Found it out-performs alternative methods of detecting exploratory dialogue
• Developed an annotated corpus for the development of automatic exploratory dialogue detection
• Identified areas for future research
• Identified ways of applying this work to support learners and educators
Yulan HeSenior Lecturer at the School of Engineering and Appied Science, Aston University, UK
SoLAR Storm webinarbit.ly/YSEVHG