Methodology for Authoring Dialogues

Click to edit the title text formatMethodology for Authoring Dialogues

Pamela Jordan

University of Pittsburgh

Learning Research and Development Center

Agenda

Methodology for authoring dialogues Some lessons learned on authoring

computer-mediated dialogues Next steps for projects & discussion

Authoring preparation methodologies

Corpus-based Theory-based Corpus-inspired Incremental refinement

Corpus-based authoring

Collect corpus of humans interacting on task Computer mediated Non-interruptible turns

Analyze for goals/topics & adjust for learning objectives Analyze goals/topics identified for student responses, look for

answer categories of: Partially correct/incomplete Partially incorrect Overly vague Overly specific Correct but premature

Identify tutor tactics for each answer category Analyze student language

Tutoring tactics in ProPl

Form tactics

Pump: can you say more about X? Hint & reask: fill in a possible missing piece then

try again Socratic: lead through line of reasoning Simulation: lead through an example & abstract For additional ones, see chapters 7 & 8 of Evens

& Michael (2006), One-on-One Tutoring by Humans and Computers

Applying tactics in ProPl

ProPL student language analysis

Use to define response concepts Strategy: pick a minimal set of key words that will

distinguish between responses

Authoring preparation methodologies

Corpus-based Theory-based Corpus-inspired Incremental refinement

Theory-based authoring

Based on theories about domain/task & learning

Examples of theoretical conceptual tactics:Definitions & applications of concepts (e.g.

distinguish technical & lay senses of terms)Conceptual variant of a domain principle (e.g.

boundary conditions)Variant of problem

Authoring preparation methodologies

Corpus-based Theory-based Corpus-inspired Incremental refinement

Corpus-inspired authoring

Combination of corpus-based & theory-based Locate related corpus Identify theoretical goals Search for some of those goals within a corpus

& refine relative to what can find Identify theoretically expected student responses Refine relative to those response instances can

find in corpus

Authoring preparation methodologies

Corpus-based Theory-based Corpus-inspired Incremental refinement

Incremental refinement

Author main-path dialogues w/ correct answers

Refine according to answer categories Author responses to answer categories Pilot dialogues Analyze logs & refine authored dialogues

Agenda

Methodology for authoring dialogues Some lessons learned on authoring

computer-mediated dialogues Next steps for projects & discussion

Recognizing student responses Language recognizer uses simple technique of minimum edit

distance Minimum edit distance is smallest number of “edits”

(insertions/deletions) needed for student response to match a response phrase

For each set of alternative NL phrases (concept) for all responses for the current question, find the minimum edit distance

Select concept with smallest minimum edit distance If that edit distance is within threshold (default of < .5) then select

that concept as the response Else the student response is not recognized, so follow the

unanticipated response path Beginnings & endings of unmatched parts of responses are not

penalized Stop words (e.g. of, the) are not penalized Strategy for authoring a response: pick a minimal set of key

words that will distinguish between responses for a question

Advice on computer-mediated dialogues

Students prone to refusal to answer e.g., “I don’t know”, “who cares” Don’t always bottom out Prod student to try (e.g. “Make your best guess”)

Avoid interrogation: remember to address coherency; include short recaps, turn and topic transitions, make some abstractions, meta-information explicit e.g., “Let’s break it down some

more”, “First, we’ll identify the givens”. Assess understanding:

Avoid explicit “do you understand?” Use trick questions; after success check strength of assertion

“Are you sure?” “What other forces are there?” (when answer is no more)

Don’t be interactive just for sake of being interactive, instead use it to adapt to individual Interact in order to diagnose what the student needs Dialogue slow if cover everything; figure out what can be skipped

Agenda

Methodology for authoring dialogues Some lessons learned on authoring

computer-mediated dialogues Next steps for projects & discussion

Next steps for projects

Look at dialogue samples/corpus for yur project and identify goals to cover in dialogueavailable corpora:

http://andes3.lrdc.pitt.edu/TuTalk/corpora/ For each goal author main path with only

correct responses and unanticipated response follow-ups

Discussion & questions

Describe your projects What help/advice do you anticipate

needing?