The Sound of One Hand:A Wrist-mounted Bio-acoustic
Fingertip Gesture Interface
Brian Amento, Will HillAT&T Labs – Research
Loren TerveenUniversity of Minnesota
Outline
• Motivation
• Gesture Interfaces
• Signal Classifiers
• Prototype Applications
• Future Work
Motivation• Small wearable digital devices increasingly
popular (Cellphones, PDAs, Rios, etc..)
• Nonlinear access to linear media will increase– Voicemail, Music, Video, Radio, Text– Controls: Device Select, Play, Stop, Scan forward,
Scan backward, Faster, Slower, Item Select, Exit
Current Interfaces to Mobile Devices
• Two-handed control mechanisms• Pressing device buttons• Writing/selecting with stylus
– Un-holstering a wearable is a pain (i.e., wristwatches beat pocket watches)
• Speech recognition– Noise or social setting may rule out voice control
• Our Goal: Invisible, weightless, un-tethered and cost-free
How about a gesture interface?
Body tracking
Teresa Martin 1997
Polhemus 2000
Datagloves
Image hand tracking
Cullen Jennings, 1999
Our Approach
“Natural” fingertip gestures
What’s “natural”• Small - max displacement of 5 cm
• Gentle, < 10% of pressing strength (e.g. no finger snap)
• Few gestures, little memory work
• Avoid ring and pinky finger
• Examples:– Thumb as anvil - index, middle as hammer– Thumbpad to fingerpad– Thumbpad to fingernail edge
Fingertip Gestures• Tap, double tap
• Finger and thumb pads rub
• Money gesture and reverse
• Finger and thumb pads press
• Soft Flick
Fingertip Gesture Interface• Wristband-mounted piezo-electric contact
microphones positioned on the styloid bones
• Sense bone conducted sounds produced by gentle fingertip gestures
Simple Classifier
• Allows real-time analysis and control
• 800 samples every 10th of a second
• Take max absolute, quantize to 10 levels
• Finite state machine outputs Taps and Rubs– Intermediate states filter background noise– Buffer states allow continuous gestures
• Surprisingly accurate: ~90%
Example Signals
More Sophisticated Classifier
• Noticeable differences in audio signals
• Hidden Markov Models
• Gesture and noise models trained with sampled data
• Confidence levels for each trained gesture
HMM Classifier Accuracy
• Using 3 subjects, collected 100 instances of gestures rub, tap and flick
• 80 used for training, 20 for testing
Accuracy
Tap 55/60 (92%)
Rub 59/60 (98%)
Flick 56/60 (96%)
Wrist Display Prototype
• Timex Internet Messenger watch
• Tap to cycle through messages
• Double-tap to rewind
Other Prototypes
• Cellphone dialing application– Rub scrolls list in one direction– Tap dials phone number
• Powerpoint slide control– Tap moves forward one slide– Double tap moves back
Future Work
• Miniaturization of device– Hitachi SH5 controller
• Improved gesture classifiers• Finger Identification
– Analyze signals from multiple microphone locations
• User Studies– Usefulness: Compare performance to current cellphone,
PDA and desktop control interfaces.– Social impact: Study how users exploit private control
techniques to mobile devices
Conclusion
• Fingertip gestures– sensed acoustically at the wrist – can be communicated wirelessly to nearby
devices– show promise as a control method.