Upload
vuonglien
View
220
Download
0
Embed Size (px)
Citation preview
Contributions to the Design of
Novel Hand-based Interaction Techniques
for Virtual Environments
Advisors Maud Marchal
Anatole Lécuyer
Merwan Achibet
1
Using our hands in virtual environments
[Catmull, 1972] [Fisher, 1986]
• The hand in the real world
– Action
• Strength or precision
• High flexibility
– Perception
• Different scales
• High sensitivity to touch
• The hand in virtual environments
2
Applications of hand-based interaction
3
[Dinsmore, 1997]
[Dai, 1996] [Seth, 2006]
[Corvaglia, 2004]
[Cutler, 1997]
Virtual prototyping
Virtual training
Dextrous interaction
[Veit, 2008]
Context of hand-based interaction in VR
Virtual reality (VR)
Immersive computer simulations that users can interact with and perceive
3D interaction techniques
Methods allowing users to accomplish a task via the user interface
Haptic feedback
Sensory information relating to the sense of touch 4
Challenges of hand-based interaction
5
Challenge 1
Providing convincing haptic sensations
Challenge 2
Handling the many DoF of the hand
Degrees of freedom (DoF)
Independent parameters of a system defining its mechanical configuration
Related work – The human hand
• Mechanics of the hand
– Highly articulated skeleton
– Extrinsic muscles
– 27 DoF in a small volume [ElKoura, 2003]
• Function of the hand
– Depends on object shape
– Depends on intended action
6
Flexion
Adduction
[Cutkosky, 1986]
Related work – Virtual hands
• Rigid proxies
Simple simulation
Limited to abstract interaction
• Articulated rigid hands
Enable grasping
Only simulate hard contacts
• Deformable models
Realistic interaction
Higher computational cost
7
[Boeck, 2004] [Craig, 2009]
[Jacobs, 2011] [Talvas, 2015]
[Borst, 2005] [Maciel, 2004]
Challenge 1
Handling the many DoF of the hand
Related work – Haptic hardware
(Geomagic) [Monroy, 2008]
(Haption) (CyberGlove Systems)
(CyberGlove Systems)
Whole-hand Multi-finger
8
Desktop
Mobile
Providing convincing haptic sensations
Challenge 2
Related work – Passive and pseudo-haptics
• Passive haptic feedback
– Real objects to provide haptic cues
– Elastic input devices
Less flexible than active haptics
• Pseudo-haptics [Lécuyer, 2000]
– Leverages the visual dominance
on the perception of physical properties
– For hand-based interaction:
alter the motion of the user’s hand
[Borst, 2005]
[Ban, 2012]
[Pusch, 2008] 9
[Kry, 2008]
Challenge 1
Handling the many DoF of the hand
Providing convincing haptic sensations
Challenge 2
DoF separation DoF reduction
Contributions
10
Axis I: Improving the control of articulated hands Axis II: Improving feedback with passive and pseudo-haptics
Active Passive
Pseudo
Objective: Improve hand-based interaction in virtual environments
DoF reduction DoF separation Whole-hand Grasping Multi-finger Level of detail
Axis I: Improving the control of articulated hands
11
Axis I: Improving the control of articulated hands
DoF reduction DoF separation
Contribution #1: THING
12
DoF reduction
Axis I: Improving the control of articulated hands
DoF reduction
• Approach – Reduce DoF of hand models to couple them to common input devices
• THING (in reference to the Addams Family character)
– Multi-touch control for animating hand models
– Fingers animated via gestures on the tablet
– Global position controlled via two variants, mobile and desktop
A multi-touch interaction technique for controlling 3D hand models
13
𝒅
𝒅𝒂
𝒅𝒇
Animating the fingers
14
𝑏1
𝑐1
𝑏2
𝑐2
Mu
lti-
tou
ch in
pu
t
For each fingertip, 𝒑𝒏 = 𝐵 𝑓𝑛 + 𝑎𝑛 ∗ 𝒄𝟏 − 𝒃𝟏 × 𝒄𝟐 − 𝒃𝟐
gain function
Δ𝑓 = 𝒅𝒇 𝑠𝑖𝑔𝑛 𝒅 ⋅ 𝒔 𝐺(𝒗𝒇)
Flexion
flexion/extension
𝑎 = 𝑠𝑖𝑔𝑛 𝒅 ⋅ 𝒔⊥ 𝒅𝒂
Adduction
adduction/abduction
Han
d m
od
el
B
𝒔
Moving the hand: two variants
15
DesktopTHING
• All DoF integrated on the tablet • Move and tilt controls
• Position DoF delegated to traditional mouse controls
MobileTHING
• Objective – Compare MobileTHING to traditional
techniques of computer animation
• Task – Reproduce predefined postures
User study #1
17
3 techniques × 4 poses × 5 repetitions × 12 participants
Computer mouse Data glove (mocap) MobileTHING
User study #1 • Performance
– Glove and MobileTHING outperform Mouse
• Subjective questionnaire
– No significant differences for MobileTHING in terms of ease, speed, comfort, fatigue
– Glove did not fit 4 participants out of 16
18
• Objective
– Compare MobileTHING vs. DesktopTHING
User study #2
19
3 techniques × 3 poses × 5 repetitions × 12 participants
Pro
toco
l R
esu
lts
• Performance
– Both THING variants outperform Mouse
– No differences between
DesktopTHING and MobileTHING
• Appreciation
– DesktopTHING for precision, fatigue
– Overall preference for DesktopTHING
Computer mouse MobileTHING DesktopTHING
1
7
Appreciation Ease Speed Precision Fatigue
4
2
3
6
5
Conclusion
THING: Multi-touch technique for manipulating 3D hand models
• Common low-cost input device
• The user’s gestures on the tablet are reflected on the hand models
• User studies
– Performance similar to data glove, better than traditional computer mouse
– Desktop version preferred over mobile variant, similar performance
20
Contribution #2: DesktopGlove
21
DoF reduction
Axis I: Improving the control of articulated hands
DoF separation
Separating the DoF of virtual hands for haptic manipulation
• Approach – Separate DoF between two distinct interfaces
• DesktopGlove – Distribute the DoF of one virtual hand between both user’s hands
– Global wrist control on one side, Local finger control on the other
22
Interface Modified DigiHaptic [Casiez, 2003]
Input Finger flexion Output Local grasping forces
System
23
Interface Common haptic arm Input Position and orientation Output Global forces (collisions, weight)
Global motion
Local motion
Example
24
• Pink arrows: forces sent to the fingers (left hand)
• Orange arrow: force sent to the wrist (right hand)
Conclusion
DesktopGlove: Separate the degrees of freedom of one virtual hand
between both user’s hands
• Exhaustive haptic feedback (local + global forces) with accessible interfaces
• User studies
– Separated control: no significant negative effect on performance, preferred by users
– Separated feedback: no significant negative effect
– User preference:
• global controls with dominant hand,
• full force feedback on both hands
25
Axis I: Improving the control of articulated hands
26
Axis I: Improving the control of articulated hands
DoF reduction DoF separation
Axis II: Improving feedback with passive and pseudo-haptics
27
Axis II: Improving feedback with passive and pseudo-haptics
Whole-hand Grasping Multi-finger Level of detail
Contribution #3: Elastic-Arm
28
Passive
Pseudo
Axis II: Improving feedback with passive and pseudo-haptics
Whole-hand Level of detail
Human-scale haptic feedback for augmenting 3D interaction
• Approach
– Leverage passive components to design lightweight and mobile haptic interfaces
• Elastic-arm
– Elastic armature providing egocentric force feedback to the hand
– Several interaction techniques based on a stretchable arm
29
System
• Two states – Relaxed (feedback on)
– Extended (feedback off)
• Main parameters – Reach 𝒓 = 𝒉 − 𝒔
– Rest length 𝑑𝑒
– Max. extension 𝑑𝑚
30
𝒉 𝒔 𝒓
𝒅𝒆 𝒅𝒎
Rate control
Position control
Navigation
• Stretchable arm to navigate towards out-of-reach objects – Go-Go: extension of the user’s arm [Poupyrev, 1996]
– Bubble: hybrid position/rate control [Dominjon, 2005]
32
• Simulate collisions with the Elastic-Arm – Redirected touching: warp virtual hand to simulate shapes [Kohli, 2010]
– Match the rubber band extension with virtual collisions
+ do
Exploration
33
𝑠
𝑑𝑜
ℎ𝑣
= 1 when the cable is extended
Virtual hand position
Shoulder position
Distance to facing obstacle
𝒉𝑣 = 𝑠 min (1,|𝒓|
𝑑𝑒)
• Simulating different levels of effort with the Elastic-Arm – Inspired by pseudo-haptic stiffness [Lécuyer, 2000]
– First, match the rubber band extension
– Then, scale the Control/Display ratio
Manipulation
34
Same as with redirected touching
Object-dependent Control/Display ratio
High k = low stiffness
Low k = high stiffness
Extension beyond rest length
𝒉𝑣 = 𝒔 + do min 1,𝒓
𝑑𝑒 +
𝒓 − 𝑑𝑒
𝑑𝑚 − 𝑑𝑒 𝒌
Alternative designs
35
• Rubber bands with different lengths
• Each band matches a different event
• Rubber bands attached to the waist
• Pseudo-haptic simulation of weight
Layers of stiffness
Weight
Conclusion
Elastic-Arm: Body-mounted armature providing egocentric force
feedback to the user’s hand
Proposed 3D interaction techniques:
• Navigation/selection by stretching the arm
• Perceptual effects: collisions and different levels of effort
• Alternative designs: layers of stiffness and weight
36
Contribution #4 – The Virtual Mitten
37
Passive
Pseudo
Axis II: Improving feedback with passive and pseudo-haptics
Grasping Level of detail
Haptic manipulation of virtual objects using grip force
• Approach
– Simulate grasping sensations with passive devices providing grip forces
• Virtual mitten
– Real hands represented by simplified mittens
– Elastic devices to control the mitten and deliver grip force
– Visual feedback and pseudo-haptic effect to vary perceptions
38
System
41
Object A
Objective: simulate different levels of effort for different virtual objects
Pseudo-haptics
Object B
Example
• Bimanual cooking scenario
• Stiffness of the fruits, weight of the glass, different interactive parts 42
User study #1 • Objective
– Evaluate if users can perceive levels of effort
– Compare Boolean and Progressive feedback
• Task
– Perform various manipulation tasks
with pairs of object
– « Which object required more effort? »
43
2 types of feedback × 4 tasks × 3 repetitions × 12 participants
[Bloomfield, 2003]
• With Progressive feedback
– Significantly more correct answers
– Compression applied closer to
the grasping threshold
Re
sult
s P
roto
col
Re
sult
s P
roto
col
User study #2
44
r 0 1 𝑟𝑓𝑜𝑙𝑑𝑖𝑛𝑔
𝒓𝒆𝒇𝒆𝒓𝒆𝒏𝒄𝒆 𝒕𝒉𝒓𝒆𝒔𝒉𝒐𝒍𝒅
+12% +24% +36% -36% -24% -12%
𝒄𝒐𝒎𝒑𝒂𝒓𝒊𝒔𝒐𝒏 𝒕𝒉𝒓𝒆𝒔𝒉𝒐𝒍𝒅𝒔
• Just Noticeable Difference:
minimum difference between
stimuli that can be discriminated
• Four different levels of effort
• Objective
– Determine the resolution of the effect
• Task
– Rotating the cylinder with the Progressive feedback
Multi-finger extension
45
Syst
em
P
seu
do
-hap
tics
• ElasticGlove – Modular passive exoskeleton
– Constrains each finger individually
• Pseudo-haptic simulation of heterogeneous materials – Finger-specific Control/Display ratio
– Haptic data embedded into mesh
Same C/D ratio Different C/D ratios
Conclusion Virtual Mitten: Interaction metaphor based on an elastic device
• Lightweight elastic input device
• Pseudo-haptic effect to simulate different levels of effort
• User studies
– Progressive visual feedback provided better results
– Can simulate up to 4 levels of effort
ElasticGlove: Extension to multi-finger interaction with passive exoskeleton
47
General conclusion
• Objective – Improve hand-based interaction in virtual environments
• Contributions – Axis I: Improving the control of articulated hand models
• DoF reduction and multi-touch input
• DoF separation between two haptic devices
– Axis II: Combining passive and pseudo-haptics
• Body-mounted elastic interface for whole-hand interaction
• Elastic grip force device for simulating grasping
• Passive exoskeleton for multi-finger manipulation
48
Future work
• Additional input to handle more DoF
– Force sensing for THING,
– Greater number of fingers for DesktopGlove
• Bimanual interaction
– Parallel hand animation for THING
– Bimanual interaction techniques for DesktopGlove
• Low-cost tracking
– Pressure sensing for Virtual Mitten, bend sensing for ElasticGlove
• Evaluate passive interfaces vs. active counterparts
– Elastic-Arm vs. Desktop interfaces
– ElasticGlove vs. active exoskeletons 49
Axis I
Axis II
Perspectives
• Software control of the missing DoF – Inspiration from robotics (grasp planning),
computer animation (shape matching, behavioral data)
• Pseudo-haptic feedback for the simulation of complex sensations – Multiple stimuli (multi-finger, bimanual)
– Complex environments
• Combining different haptic modalities – Design a passive exoskeleton from passive components
– Active/Passive/Crossmodal feedback combinations
50
Publications • International conferences
– M. Achibet, A. Girard, A. Talvas, M. Marchal, and A. Lécuyer. “Elastic-Arm: Human-scale Passive Haptic Feedback for Augmenting
Interaction and Perception in Virtual Environments”. In Proc. of Virtual Reality, pp. 63–68, 2015.
– M. Achibet, G. Casiez, A. Lécuyer, and M. Marchal. “THING: Introducing a Tablet-based Interaction Technique for Controlling 3D Hand
Models”. In Proc. of Conference on Human Factors in Computing Systems, pp. 317–326, 2015.
– M. Achibet, M. Marchal, F.Argelaguet, and A. Lécuyer. “The Virtual Mitten: A Novel Interaction Paradigm for Visuo-haptic
Manipulation of objects using Grip Force”. In Proc. of Symposium on 3D User Interfaces, pp. 59–66, 2014.
• National conferences
– M. Achibet, A.Girard, M. Marchal, F. Argelaguet, and A. Lécuyer. “Elastic-Arm: A Novel Human-Scale Passive Haptic Approach for
Augmenting Interaction and Perception in Virtual Environments”. In Annual Conference of the AFRV, 2015.
– M. Achibet, M. Marchal, F. Argelaguet, and A. Lécuyer. “Visuo-haptic Manipulation of Objects with the Virtual Mitten”. In Annual
Conference of the AFRV, 2014.
• Submitted
– M. Achibet, A. Girard, M. Marchal, and A. Lécuyer. “Leveraging Passive Haptic Feedback in Virtual Environments with the Elastic-Arm
Approach”. Submitted to Presence.
– M. Achibet, G. Casiez, and M. Marchal. “DesktopGlove: a Multi-fingered Force Feedback Interface Separating Degrees of Freedom
between Hands”. Submitted to IEEE Symposium on 3D User Interfaces.
– M. Achibet, M. Marchal, A. Lécuyer, and H. Kajimoto. “Elastic Exoskeleton: Multi-Finger Interaction Combining Passive Haptics and
Pseudo-Haptics”. Submitted to IEEE Transactions on Haptics.
• Patents
– A. Lécuyer, M. Achibet, M. Marchal, and F. Argelaguet. “Système d’interaction avec des objets virtuels”. French patent, 3016451-A1.
51