Upload
clifford-stone
View
75
Download
0
Tags:
Embed Size (px)
Citation preview
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
A novel interface system for seamlessly integrating human-robot
cooperative activities in space
Nilanjan SarkarMechanical Engineering
Electrical Engineering and Computer Science
Craig A. SmithPsychology and Human Development
Vanderbilt UniversityNashville, TN
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Inspiration“Exploration of space and solar system will be most effective if human capabilities are synergistically combined with those of robots. Such a human-robot system, developed correctly, will reduce exploration risks, improve efficiency, and achieve overall mission goals faster and in a better manner.”[Objective of the ICASE/USRA/LaRC Workshop on Revolutionary Aerospace Systems Concepts for Human/Robotic Exploration of the Solar System, Nov. 2001]
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Major ObstacleNatural interaction between human and robot
Is implicit communication possible?
- e.g., robot brings the right tool to the astronaut without being explicitly commanded
Can the robot understand the psychological states of the human?
- e.g., robot rushes to help the human if the robot senses “panic”
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Novel Approachaffect detection and recognition
brainwave monitoring and characterization
design of control architecture for implicit communication
integrates research in signal processing, wearable computing, experimental psychology and control theory
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
HypothesesA robot will:
implicitly understand a task command from an astronaut
sense the psychological state of the astronaut and take necessary actions
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Rationale
Initially, the research will be:person specificcontext specific
Afterwards, with enough understanding and data, an affect recognizer for a class of people will be attempted
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Phase I TasksThe following tasks were proposed:
A. Develop Affect Recognizer1. Design human subject experiments to elicit target
affective states (e.g., engagement, anxiety, fatigue, etc.)
2. Assess physiological indices that are important for the target affective states
3. Conduct pilot studies 4. Data analysis and signal processing for affect detection
B. Investigate the currently available brainwave monitoring technologies
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Phase I Tasks
The following additional tasks were performed:
Developed a functional human-robot system that is affect-sensitive, and
Conducted preliminary brainwave monitoring experiments with EEG
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Development of
Affect Recognizer
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Experimental TasksThree Problem-Solving Tasks (Each lasting ~1 hour)
Anagrams
Math Word Problems
Sound Discrimination
Two Sequences for Each Task1. Easy: Fairly Easy --> Trivially Easy
2. Difficult: Fairly Easy --> Virtually Impossible
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Sample Anagrams
Easy Condition Difficult Condition
AWADR IYTED
DEITYAWARD
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Easy Math Problem
An astronaut requires 2 pounds of oxygen per day while in space. How many pounds of oxygen are needed for a team of 3 astronauts who are going to spend 5 days in space?
30Answer:
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Difficult Math Problem
Tammy has $9.70 in nickels, dimes, and quarters. The number of nickels is 4 more than 3 times the number of dimes, and the number of quarters is 5 fewer than 2 times the number of nickels. How many nickels does Tammy have?
19Answer:
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Sound TaskSequence of three tonesJudge whether first and third tones are the same or differentDifficulty manipulated by varying tone length and frequency difference between tones
Easy Trials Difficult Trials
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Physiological Data Collection
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Physiological MeasuresEKG
interbeat interval (IBI)meanvariability
sympathetic power *parasympathetic power *
Finger Pulse Amplitudemean variability
Pulse Transit Time (PTT)Digit Skin Temperature
meanslope of change
Skin ConductanceTonic
meanslope of change
Phasicresponse rate *average amplitude *maximum amplitude
Facial Muscle Activity (EMG)Brow (Corrugator)
meanvariability *
Jaw (Masseter)mean *variability
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Self Report MeasuresSampled:
Two minutes into each taskEvery seven minutes thereafter
Assessed:Task DifficultyPerceived AbilityAffective States
Key Affective Parameters:
Anxiety IndexAnxietyOverload
Engagement Index
Task ImportanceHopeChallengeInterest(R) Resignation(R) Apathy
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Correlations of Anxiety with Physiology
-0.5
-0.3
-0.1
0.1
0.3
0.5
Symp Paras SCR-Rate
SCR-Amp
Corr-Var
Mass
Physiological Parameter
Subject 2Subject 4
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Correlations with Physiology:Anxiety vs. Engagement (Subject 2)
-0.5
-0.3
-0.1
0.1
0.3
0.5
0.7
Symp Paras SCR-Rate
SCR-Amp
Corr-Var
Mass
Physiological Parameter
AnxietyEngagement
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Schematic of Fuzzy LogicAnalyzer
SympatheticPower
ParasympatheticPower
Skin ConductanceResponse Rate
Skin ConductanceResponse Amplitude
CorrugatorVariability
MasseterAverage
Fuzzy LogicAnalysis
AffectDiagnostic
Output
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Development of a
Functional Human-Robot
Interactive System
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Our Robot
Operates in three modes:1. Explore -- Wanders
about environment2. Survival -- Uses sonar to
detect and avoid obstacles in environment
3. Affective -- Responds to affective signals from human operator
If robot senses high anxiety, it uses light detection algorithm to go to operator
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Robot Control ArchitectureA hybrid subsumption control paradigm
Emergency Stop
Reverse
Affect Layer
Obstacle avoidance
Wall Follow
Wandering
S
S
S
S
S
Sensor Data
Survival Mode
Deliberative Mode
Reactive Mode
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Human-Robot Interaction Scenario
Human operator works at computer task while the robot explores the environment and monitors
operator’s affective state
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
The Functioning System
QuickTime™ and a DV/DVCPRO - NTSC decompressor are needed to see this picture.
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Physiological Signals & Fuzzy Logic Output for Moderate Anxiety Trigger
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Timing Diagram from Robot Experiment
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Investigation of Brain Wave
Monitoring Technologies
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Technologies to Assess Brain ActivityEEG - Measures electric currents generated by the brain at the scalp
MEG - Measures magnetic fields generated by the brain
PET - Measures emissions from radioactively labeled chemicals, monitors metabolic rates and blood flow
fMRI - Measures oxygen concentration of blood, correlates to blood flow
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Modality Comparison
EEG MEG fMRI PET
Spatial Resolution 7-16 mm 3-11 mm 1 mm 5 mm
Temporal Resolution Milliseconds Milliseconds 1 second to
minutes 45 seconds to
minutes
Cost $15-120K
$500k (37 channel)
$300-400k Shielding
Millions Millions
Wearability Yes No No No
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Example EEG and MEG Systems
http://www.xltek.com/xlwebv2/products/eeg/xlamb.htmhttp://www.vsmmedtech.com/MEG_Technical/MEG_Images_275_151.asphttp://www.biomag.helsinki.fi/meg.htmlhttp://www.elixa.com/mental/MindSet.htmhttp://www.emedicine.com/neuro/topic445.htm
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
EEG Experiment: Cognitive Load
0
5001000
1500200025003000
350040004500
Alp
ha P
ower
Eyes Closed Eyes Open
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
EEG Experiment: Math Problem-Solving
0
100
200
300
400
500
600
700
800
900
Alp
ha P
ower
Eyes Open Easy Moderate Difficult
Problem Difficulty
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Summaryperformed every proposed task for Phase I
performed two additional tasks beyond what was proposed
were successful in eliciting and detecting the target affective states under controlled situations
developed a functional human-robot system to demonstrate the feasibility of the central concept
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
ConclusionsPhase I work demonstrates that:
implicit human-robot communication is feasible
can detect human affect on-line and in the context of realistic taskscontrol system can be made affect-sensitive
brainwave monitoring will likely supplement peripheral physiology in affect detection and implicit communication
Vand
erbi
lt U
nive
rsity
Robotics and Autonomous Systems Lab (RASL)
Future WorkExpand range of tasks and contexts to which framework can be appliedIncrease the reliability and sophistication of affect recognition
Increase range of affects detected and discriminated beyond anxiety and engagement to include frustration, fatigue, boredom, etc.Advance analytical tools for extracting relevant information from physiological signals
Increase degree to which physiological recording is ambulatoryFurther explore utility of EEG recording to improve affect recognition and implicit communicationFormalization of affect-sensitive control system