Embed Size (px)
Citation preview
Spoken Dialog Management for an Astronaut’s Procedure Assistant
Presented by: Dan Bohus ([email protected])Collaborators: Gregory Aist, RIALIST Group
NASA Ames/RIACS Summer Student Research ProgramAugust 2002
S.D.M.A.P.A.
10-10-02 Dialog Management for an Intelligent Procedure Assistant 2
RIALIST Group @ NASA AmesLocation: NASA Ames / Moffett Field / Mountain
View, CA (CMU West)People
Jim Hieronymous John Dowding Beth-Ann Hockey Greg Aist + 7 summer students
Projects Personal Satellite Assistant Mobile Robots Intelligent Procedure Assistant
10-10-02 Dialog Management for an Intelligent Procedure Assistant 3
Motivation
Crew members on International Space Station (ISS) have to perform a wide variety of procedural tasks:
ISS maintenance and repair Operations (i.e. EVA Suit Checkout) Setup and running scientific
experiments Medical Exams
10-10-02 Dialog Management for an Intelligent Procedure Assistant 4
Typical setup
Paper documentation2 or more astronauts involved: one reads, one
executes the procedure
10-10-02 Dialog Management for an Intelligent Procedure Assistant 5
Proposed alternative
Build a spoken dialog system for procedural task guidance and support
Create more crew-time! Used also for ground
training Potential for increased
safety
10-10-02 Dialog Management for an Intelligent Procedure Assistant 6
Outline
MotivationThe Intelligent Procedure AssistantA Dialog Management Component for IPADemoFuture work
10-10-02 Dialog Management for an Intelligent Procedure Assistant 7
The Intelligent Procedure Assistant
Multi-modal spoken dialog system providing crew support and training for the execution of procedural tasks
Guide astronauts through procedural tasks Provide information access (diagrams, docs) Act as a training tool Facilitate task delegation Provide support for voice annotations, intercom
10-10-02 Dialog Management for an Intelligent Procedure Assistant 8
IPA Version 0: June 2002
RIALIST IPA Version 0 Load procedure Next Previous Drill-down Describe remaining steps Increase volume Decrease volume Say again
10-10-02 Dialog Management for an Intelligent Procedure Assistant 9
Outline
MotivationThe Intelligent Procedure AssistantA Dialog Management Component for the IPADemoFuture work
10-10-02 Dialog Management for an Intelligent Procedure Assistant 10
SpeechRecognition
DialogManagement
LanguageGeneration
Synthesis
U D SI
SONLS
Semanticizer(parsing)
Resou
rces/A
gen
ts
Contributions: Dialog Management & Resources
10-10-02 Dialog Management for an Intelligent Procedure Assistant 11
Nuance[recognition]
DialogManagement
Gemini[generation]
Festival
U D SI
SONLS
Gemini[parsing]
Resou
rces/A
gen
tsNuance[recognition]
DialogManagement
Gemini[generation]
Festival
U D SI
SONLS
Gemini[parsing]
Resou
rces/A
gen
ts
Audio Dialog Agent
RavenClawDM framework
(CMU)
Contributions: Dialog Management & Resources
1
2
3CORE/Conversational Skills
Dialog TaskSpecification
The actual domain task (IPA domain)
Handles dialog flow, initiative, focusshifts, universal dialog mechanisms
10-10-02 Dialog Management for an Intelligent Procedure Assistant 12
1. Resources
Procedures represented in XML (DTD, XSLT) Title, Description. Steps: Basic Steps, Conditional Steps, Messages,
Images Messages: Notes, Cautions Images: Sub-images
Water Sampling Procedure 28 steps, 150 sub-steps, 18 conditional steps,
1 image with 5 sub-images HTML version (using XSLT)
10-10-02 Dialog Management for an Intelligent Procedure Assistant 13
2. Dialog Task DevelopmentIn RavenClaw, a dialog task is implemented as a
hierarchy of dialog agents (C++ classes)
IPA
Welcome LoadProcedure Do Procedures
DoProcedure1
Intro Closing
DoProcedure2
Step1 Step28 Image1
SubStep1 SubStep4
Help Suspend/Resume
10-10-02 Dialog Management for an Intelligent Procedure Assistant 14
2. Dialog Task Development (cont)
52 dialog task agents (types) were implemented: Loading procedures by name Guidance
Step-by-step execution Drill-down Branching Presenting notes / cautions / images
Information access Talking on/about images, objects
Navigational behavior History / Trace through procedure Next, Previous step/substep Start over
10-10-02 Dialog Management for an Intelligent Procedure Assistant 15
3. Conversational Skills Enhancements
Timeouts System reacts after a timeout period
User Barge-inRequests for repeatRequests for help
generic “help!” “what can I say ?”
Context establishment “where are we ?”
10-10-02 Dialog Management for an Intelligent Procedure Assistant 16
Outline
MotivationThe Intelligent Procedure AssistantA Dialog Management Component for IPADemoFuture work
10-10-02 Dialog Management for an Intelligent Procedure Assistant 17
Demo
Guidance
Volume control
Information Access
10-10-02 Dialog Management for an Intelligent Procedure Assistant 18
Outline
MotivationThe Intelligent Procedure AssistantA Dialog Management Component for IPADemoFuture work
IPA System (RIALIST Group) Advances in dialog management
10-10-02 Dialog Management for an Intelligent Procedure Assistant 19
Future work for IPA system (RIALIST Group)
IPA system functionality goals Guide astronauts through procedural tasks Provide information access Act as a training tool Facilitate task delegation Provide support for voice annotations, intercom
Iterative improvement process Other features, including those suggested by JSC
training personnel and astronauts
10-10-02 Dialog Management for an Intelligent Procedure Assistant 20
Timeline …
Reliable prototype by the end of September Guidance/Information Access
First fieldable system by March 2003
10-10-02 Dialog Management for an Intelligent Procedure Assistant 21
Summer @ NASA is cool !
10-10-02 Dialog Management for an Intelligent Procedure Assistant 22
Being back is also cool ! 2 new systems to be developed based on the
Communicator SDS setup / RavenClaw DMRoomLine system
Provide conference room reservation services within SCS
Ridy LieBusLine system
Provide access to bus line information Project “Let’s go!” (Maxine/Alan)? Brian Langner, Antoine Raux
Hope you can call them sometime before Christmas !!!
