Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation

2 December 2005

Bruno Dumas1, Beat Signer1 and Denis Lalanne2

1 WISE Research Lab, Vrije Universiteit Brussel, Belgium2 DIVA Research Group, University of Fribourg, Switzerland

Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 2June 26, 2012

Outline• Fusion of multimodal input

• Multimodal interaction: from design to implementation• Algorithms for fusion of multimodal input

• A Hidden Markov Model-based fusion algorithm• Instantiation of the algorithm

• Evaluation of the algorithm• Qualitative test• Quantitative evaluation• Performance assessment

• Conclusion

Fusion of Multimodal Input

Challenges in multimodal interaction engineering: continous and probabilistic inputs, real-time…

Challenging/Ambiguous Fusion Cases

• “Put-that-there” Complementarity case

• “Play next track” Different meanings based on order and context

CARE Properties at the Design Level

Describe how modalities can be combined Complementarity

All modalities are necessary

Assignment Absence of choice

Redundancy Same expressive power between modalities

Equivalence Any modality is sufficient

HephaisTK Multimodal Framework

Description of Multimodal Dialogues

• Syntactic-level description

• For more details: check SMUIML language

Fusion Algorithms

• Frame-based algorithms & Unification-based algorithms• Symbolic approaches• Both are a de facto standard

• Symbolic-statistical algorithms

Symbolic-Statistical Approaches

Symbolic data Interpretation of low-level data

Statistical processing Machine learning algorithms

Our HMM-Based Fusion Algorithm

Why hidden Markov models? Very good modelling of time-related events Takes advantage of input data coming from

probabilistic input modalities On-the-fly adaptation of the model

- e.g. based on user feedback

Put-That-There Example

Instantiation of the Algorithm

Evaluation of the Algorithm

Evaluation on three levels Qualitative “gut check” test Benchmarks on real-world examples Performance assessment

Evaluation: Qualitative Test

• Three expert users were asked to assess the behaviour of the algorithm frames-based and HMM-based fusion 5 minutes per condition + interview

• Coherent behaviour between the algorithms

• No noticeable difference in responsiveness

Evaluation: Benchmark

Benchmark Results (1)

• CARE properties: equivalence and redundancy tests

• Sequential and non-sequential complementarity tests

• “Play next track” example

Evaluation: Performance

• Per fusion algorithm: • 5 runs of 40 input pieces• 5 x 20 expected fusion results

• Frame-based: 18.2 ms• Standard deviation: 12.7 ms

• HMM-based: 16.6 ms• Standard deviation: 11.6 ms

Conclusion

• A new symbolic-statistical multimodal fusion algorithm based on Hidden Markov Models

• Integrated into the HephaisTK framework

• Superior recognition rates compared to existing algorithms

• Handling of ambiguous cases

• Processing of real-time user feedback

http://wise.vub.ac.be/bruno-dumas

Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation

Technology

Multilayer and Multimodal Fusion of Deep Neural Networks ...jankautz.com/publications/MMFusion4Video_ACMM16.pdf · Multilayer and Multimodal Fusion of Deep Neural Networks ... proposed

Robust multimodal face and fingerprint fusion in the ...centaur.reading.ac.uk › 48392 › 1 › PRWildetal.pdf · Robust multimodal face and ngerprint fusion in the presence of

Multimodal Registration and Data Fusion for Cardiac

Tensor Fusion Network for Multimodal Sentiment Analysis

Coupled Feature Learning for Multimodal Medical Image Fusion

Registration/Fusion of multimodal images using higher order graphs

MULTIMODAL FUSION VIA A SERIES OF TRANSFERS FOR NOISE …

Tensor Analysis and Fusion of Multimodal Brain Images

FUSION AND COORDINATION FOR MULTIMODAL … · FUSION AND COORDINATION FOR MULTIMODAL INTERACTIVE INFORMATION PRESENTATION ... 1 Some slides are available at wahlster ... & Invisible

Deep Multimodal Fusion Networks for Semantic Segmentation

Multimodal Sentiment Analysis with Word-Level Fusion and ... · Multimodal Sentiment Analysis, Word-Level Fusion, Reinforcement Learning ICMI’17, November 13–17, 2017, Glasgow,

Hierarchical feature representation and multimodal fusion

Fusion in Multimodal Interactive Systems: An HMM-Based ......Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation Bruno Dumas WISE Lab Vrije

Fusion Engines for Input Multimodal Interfaces: a Survey

Multimodal Weighted Fusion of Transformers for Movie Genre

Multilayer and Multimodal Fusion of Deep Neural Networks ...research.nvidia.com/sites/default/files/pubs/2016-10_Multilayer... · Multilayer and Multimodal Fusion of Deep Neural Networks

Comparing Classical and Embodied Multimodal Fusion for ... · Comparing Classical and Embodied Multimodal Fusion for Human-Robot Interaction Manuel Giuliani Vollst andiger Abdruck

Choosing Smartly: Adaptive Multimodal Fusion for Object ...ais.informatik.uni-freiburg.de/publications/papers/mees16iros.pdf · Choosing Smartly: Adaptive Multimodal Fusion for Object

Fusion of Multimodal Partial Estimatesjxu323/publications/report_fusion_MPE.pdf · Fusion of Multimodal Partial Estimates Jiang Xu, ... 4.2 MPE Fusion in a Probabilistic View

Multimodal fusion of fingerprint and iris