Upload
others
View
14
Download
0
Embed Size (px)
Citation preview
Coherence in One-Shot Gesture Recognition for Human-Robot Interaction
Maru Cabrera
December 3rd 2018
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA1
Relevance of Gestures in HRI
vs
❖ Human have the unique ability to quickly adjust their context and learn from very few examples.
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA2
Motivation for One-Shot Gesture Recognition
❖ Lack of a comprehensive method that generalizes gesture recognition from few observations.
❖We focus on the process used to generate a given gesture:❖Cognition❖Learning and Generalization❖Physical execution
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA3
Motivation for Coherence in Gesture Recognition
❖ Explore gesture recognition and understanding when the roles between performer and listener are exchanged.
❖ Including the human aspect within the framework to artificially generate examples.
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA4
Overview of One-Shot Learning Framework
Skeleton Info Gesture example“Gist of
Gesture”Generated Dataset
Train Classifier
HMM
SVM
DTW
CRF
Artificial Generation Methods
Kinect Sensor
Performance Metrics
- Recognition accuracy- Efficiency- Coherency
Forward Method
Backward Method
1
2
3
4
5 6
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA5
Extracting the Gist of the Gesture
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA6
ൟ𝒙𝒒 ∈ {𝒙1 ∪ 𝐼𝑃𝐷 ∩ 𝐼𝑃𝐶 ∪ 𝒙𝑯, 𝑞 = 1,… , 𝑄
2 ≤ 𝑄 ≤ 𝐻
Given 𝑔1𝑖 = {(𝑥1, 𝑦1, 𝑧1 ), . . . , (𝑥𝐻, 𝑦𝐻 , 𝑧𝐻)}
Set of inflection points
𝒙𝒒 = (𝑥𝑞 , 𝑦𝑞 , 𝑧𝑞)
Working Hypothesis:
Compact amount of information stored during cognitive processes of
gesture perception
Validating Extracted Gist of the Gesture
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA7
17 Participants
EEG Power Dynamic wavelets
Motion EEG
Correlation
Validating Extracted Gist of the Gesture
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA8
Validating Extracted Gist of the Gesture
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA9
Overview of One-Shot Learning Framework
Skeleton Info Gesture example“Gist of
Gesture”Generated Dataset
Train Classifier
HMM
SVM
DTW
CRF
Artificial Generation Methods
Kinect Sensor
Performance Metrics
- Recognition accuracy- Efficiency- Coherency
Forward Method
Backward Method
1
2
3
4
5 6
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA
Artificial Gesture Generation –Forward Approach
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA11
Input:𝒈𝟏𝒔 – 3D hand trajectory of a gesture of class s𝒙𝒔 = (𝒙𝒔,𝒚𝒔,𝒛𝒔) – 3D position of the shoulder
K – Number of artificial trajectories to generate
Variance Estimation based on 𝒙𝒔 and Gaussian Mixture Model (GMM)
Output:
ൟ𝑮𝒔 = { ො𝑔1𝒔, ො𝑔2
𝒔, … ො𝑔𝐾𝒔 – Set of artificial trajectories
for gesture class s
ො𝑔𝑘𝑖 = 𝒜𝑡
෨𝐺𝑖 𝑘 = 1,… , 𝐾 ; 𝑖 = 1,… , 𝑁 ; 𝑡 = 1
ൟ𝑮𝑖 = { ො𝑔1𝑖 , ො𝑔2
𝑖 , … ො𝑔𝑘𝑖 , … , ො𝑔𝑁
𝑖
Artificial Gesture Generation –Backward Approach
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA12
Artificial Gesture Generation –Backward Approach
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA13
Jerk Minimization
Smooth changes in joint space usingthe third derivative of the joint angle
Least Energy Expenditure
Changes in joint space using torqueto calculate economic trajectories
𝑆𝐽𝑒𝑟𝑘 = min
𝑖=1
𝐾
න𝑡1
𝑡2ሸ𝜃𝑖 +⋯+min
𝑖=1
𝐾
න𝑡𝑄−1
𝑡𝑄ሸ𝜃𝑖 𝑆𝐸𝑛𝑒𝑟𝑔𝑦 = min
𝑖=1
𝐾
න𝑡1
𝑡2
𝜏𝑖 × ሶ𝜃𝑖 +⋯+
𝑖=1
𝐾
න𝑡𝑄−1
𝑡𝑄
𝜏𝑖 × ሶ𝜃𝑖
Begins in one IK solution for human arm (k= 7) for one IP 𝓈𝑞𝑣, ends in
solution for different IP 𝓈𝑞+1𝑤
𝓈𝑞𝑣 = 𝜃1
𝑣, 𝜃2𝑣, … , 𝜃𝐾
𝑣
𝓈𝑞+1𝑤 = 𝜃1
𝑤, 𝜃2𝑤, … , 𝜃𝐾
𝑤
Initial
Final
𝑉𝑞 solutions in IP 𝑞 of gesture trajectory
𝑊𝑞+1 solutions in IP 𝑞 + 1
Artificial Gesture Generation –Combined F+B Approach
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA14
Combination at gesture instance level.
Artificially generated observations from both approaches used as training data
Overview of One-Shot Learning Framework
Skeleton Info Gesture example“Gist of
Gesture”Generated Dataset
Train Classifier
HMM
SVM
DTW
CRF
Artificial Generation Methods
Kinect Sensor
Performance Metrics
- Recognition accuracy- Efficiency- Coherency
Forward Method
Backward Method
1
2
3
4
5 6
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA
Gesture LexiconMSRC-12
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA16
Contains sequences of human movements (Kinect skeleton)
12 iconic and metaphoric gestures
◦ Gaming commands and media player
Lexicon reduced to 8 gestures◦ Excluded gestures with leg
motions or whole upper body.
Fothergill, S., Mentis, H., Kohli, P., & Nowozin, S. (2012). Instructing people for training gestural interactive systems. In Proceedings of the SIGCHI Conference (pp. 1737-1746). ACM.
Classification Algorithms
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA17
Hidden Markov Models (HMM):
✓ One-vs-all scheme
✓ 5 states left-to-right
✓ Baum-Welch algorithm
Support Vector Machines (SVM):
✓ One-vs-all scheme
✓ Radial Basis Function (RBF) Kernel
✓ MATLAB® library
Conditional Random Fields (CRF):
✓ Multi-class scheme
✓ Samples encoded using BIO: Beginning, Inside, Outside
✓ CRF++ toolkit
Dynamic Time Warping (DTW):
✓ Multi-class scheme
✓ Gesture Recognition Toolkit (GRT)
Training and Features
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA18
1 gesture instance per class was
used to extract inflection points
200 gesture examples per class
generated for training
100 gesture instances per class
from each data set for testing
Classifiers trained and compared
in terms of accuracy (𝐴𝑐𝑐%), and
recognition coherence (𝛾)
Feature vector* 𝑔𝑖 for training:
𝑔𝑖 = {𝒙𝟏 = 𝑥1, 𝑦1, 𝑧1 , … , 𝒙𝑯}
𝑔𝑖 = ሶ𝒙𝟏 𝜷𝟏… ሶ𝒙𝒋 𝜷𝒋… ሶ𝒙𝑯−𝟏 𝜷𝑯−𝟏
ሶ𝒙𝒋 = 𝒙𝒋+𝟏 − 𝒙𝒋 𝜷𝒋 = tan−1𝑦𝑗
𝑥𝑗, tan−1
𝑧𝑗
𝑦𝑗, tan−1
𝑧𝑗
𝑥𝑗
* M. G. Jacob and J. P. Wachs, “Context-based hand gesture recognition for the
operating room,” Pattern Recognition Letters, vol. 36, pp. 196–203, Jan. 2014.
Overview of One-Shot Learning Framework
Skeleton Info Gesture example“Gist of
Gesture”Generated Dataset
Train Classifier
HMM
SVM
DTW
CRF
Artificial Generation Methods
Kinect Sensor
Performance Metrics
- Recognition accuracy- Efficiency- Coherency
Forward Method
Backward Method
1
2
3
4
5 6
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA
Performance Metrics
Coherence (g) is defined asthe intersection betweenthe agreement indices (AIx)for humans and machines,whether each agentcorrectly recognized eachgesture or not. 𝛾 =
𝐴𝐼𝑥𝑀 ∩ 𝐴𝐼𝑥𝐻
𝐴𝐼𝑥𝐻
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA20
𝐴% =𝑡𝑜𝑡𝑎𝑙𝑡𝑟𝑢𝑒−ℎ𝑖𝑡𝑠𝑡𝑜𝑡𝑎𝑙𝑠𝑎𝑚𝑝𝑙𝑒𝑠
Accuracy Results
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA21
Forward and Backward approach for One-Shot Gesture Learning compared.
K-fold cross validation scheme with k = 10.
10 𝐴𝑐𝑐% values for m and s for each approach.
Coherence Experiment Setting
❖ Two scenarios were explored with Baxter performing artificially generated gestures:❖ Scenario 1 (MH): Gestures are
recognized by 10 human participants.
❖ Scenario 2 (MM): Gestures are recognized by 4 classification algorithms.
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA22
Coherence Experiment Setting
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA23
Experimental Results
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA24
Takeaway Messages❖ In its application to one-shot learning, the proposed methodhighlights the use of context for gesture recognition from the wayhumans use their bodies.
❖ The obtained results show the performance of the method,demonstrating independence from the selected classification strategy.
❖ The robotic implementation opens a different route towardscoherence in human–robot interaction.
❖ Coherence can be related to gesture classification when humans andmachines interchange the roles of performing and recognizing agesture.
❖ The calculated coherence metric is our main indicator that thegenerated gestures capture human-like variations for all the gestureclasses.
12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -
MARU CABRERA25
Maru Cabrera
Coherence in One-Shot Gesture Recognition for Human-Robot Interaction