Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
Input devices & sensingInput devices & sensing technologies
Andy WilsonAndy WilsonAdaptive Systems and Interaction
In the “future”…Sensing technology can enable a wide variety of new
interactions
As hardware approaches “free”, we can afford a diversity of form factors
already have: phones, tablet, TV, car, console gamewill have: walls, tables, rooms, ?
d ll b d d l!Not every device will be used to do email!
Devices can and should be pleasing to use, as well as useful.
TouchLight
an imaging touch screen with some unique capabilities
Two webcams + DNP Holoscreen + IR illuminant
video
DNP HoloScreen
Image Processing
Edge Maps
One camera view Product of both views
Potential sensing capabilitiesOn surface
Visual barcodes, object recognitionDocument scanning (helps to have a transparent surface!)G t b d i l ti f bj tGesture‐based manipulation of onscreen objectsIR stylusMultiple hands
Off surfaceFace detection/recognition, gaze, person‐tracking, awarenessStylus + hand combination, hands in space IR remote/pointing device, tracking of devices in 3D
...essentially all vision‐based perceptual user interface techniques...
Volumetric Imaging Touch Screen
TouchLight has no transition cost from on surface to off‐surfaceRemoves the ‘blind spot’ between ‘on the surface’ and 2 feetRemoves the blind spot between on the surface and 2 feet away
?
stereo
ApplicationsPost WIMP, direct manipulation, Minority‐report gesture‐based interfaces
Eye to eye videoconferencingEye to eye videoconferencingClearBoard (Ishii) reduxVisible light surface scanning2.5D interfacesSpatial displaysMagic mirrorMagic mirrorAugmented realityTables and other direct manipulation form factors
video
Really, What’s the Killer App?
What is the whiteboard’s killer app?
PlayAnywhere: A Compact Tabletop Computer Vision System
“Lunchbox” interactive vision system
video
PlayAnywhereShort‐throw projector, very
wide angle lens on the camera
L di t ti & j tiLens distortion & projective transform correction for camera‐projector alignment
Off‐axis IR LED illuminant
Very few assumptions aboutVery few assumptions about the appearance of the surface
All calibration is done at the “factory”
Front Projection Vision SystemsCeiling installation of projector is difficult,
dangerousNot easily movedVibrations in the building are a problemVibrations in the building are a problemUser’s head and hands occlude the system
Digital Desk (Wellner), EnhancedDesk (Koike), Augmented Surfaces (Rekimoto), I/O Bulb (Underkoffler), Visual Touchpad (Malik)
Also see SenseTable (Patten), DiamondTouch
camera
projector
projection surface
( ),(Dietz), SmartSkin (Rekimoto)
Rear Projection Systems
Self‐contained device
Leg room and screen size are difficult to balance
Housing can be large, heavy
MetaDesk (Ullmer), Perceptive Workbench (Leibe), Designer’s Outpost (Klemmer), HoloWall (Matsushita)
PlayAnywhere
Portable
As long as sitting on the same plane, no need to lib t ft icalibrate after moving
Occlusion by hands, not heads
Decently large projection
Allows legs under the table, but
One side of the table is effectively blocked
Detecting touch is tricky
Vision for SensingHigh computational costLow frame rate, high latencyPrecision noisePrecision, noiseCalibration
But…Extreme flexibility
Hands, fingers, visual tags, pages, tangrams, dice, textures, object recognition, OCR ...
Image Processing
Input Lens distortion, projective distortion removed
Shadow‐based touch
Shadow as second projection
Shadow‐based hover
Paper tracking
30Hz, based on finding strong lines
Page tracking
•Sobel edge detection is based on gradients in the horizontal and vertical directions.
22yx GGG +=
vertical directions.
•The strength of an edge at (x, y) is
•Where and are obtained by dot product with masks
xG +1 +2 +1
0 0 0
-1 0 +1
-2 0 +2yG
p
•Orientation of the edge is
-1 -2 -1-1 0 +1
⎟⎟⎠
⎞⎜⎜⎝
⎛=
x
y
GG
arctanθ
•At each pixel location (x,y) we have |G| and theta
•Transform (x,y, theta) to (r, theta), where r is shortest distance from the origin to the line
Page tracking
rdistance from the origin to the line
•Histogram |G| over (r, theta)
r
theta
Page tracking
To find an 8” x 11” rectangle, look for specific pattern of peaks in the histogram
r
theta
8.5”
11”
90 deg
Fast visual codes
Read edge orientation, rotate and read 12 bits blindly, compare against list of known bit patterns
Hough transform for circles, computed from edge image
Rotating, Scaling, Translating ObjectsExisting approaches to freeform manipulation of objects, e.g.
photos
D t th bj t ith id tDecorate the object with widgetsVisually cumbersomeRequires trainingReduces the immediacy of tangible UIs
Track distinct objects (contacts) and compute movementAssumes good tracking/correspondence frame to frameAssumes good tracking/correspondence frame to frameSometimes difficult to define an object in this schemeGross manipulation with the hand is often limited to translation
Tracking is Hard
We fall for it because we have such a strong notion of the cursor
S kid i t ti ith t bl !See kids interacting with tables!
Tracking rarely allows for graceful failure
Tracking reduces the richness of human motion to that of a gnat
Flow Move
Summarize optical flow field as simultaneous translation, rotation, scaling
PlayAnywhere VE
Flow Move
Translation Rotation Scaling
Can solve for simultaneous rotation, translation, scaling via least squares
Surface Computing Challenges
TechnicalProjection: somewhat doable now, affordable tomorrow
Sensing: still researchSensing: still research
Interaction Design is still an open questionHow does it work?
How to break out of how things work today?WIMP isn’t a great fitWIMP isn t a great fit
The key may be diversity of UI
What is it really good for? Intuition only gets you so far.
Bluetooth photo synch
How does it work?
Detect phone‐shaped object?
Connect to each Bluetooth device:
Is it advertising our software service?
Command it to blink the IR port on the phone
Did vision system detect IR port blinking?Did vision system detect IR port blinking?
Start synch over Bluetooth
PlayTogether
video
IR Laser Pointer TrackingTrack shaped laser pointer (hologram)
Theoretically 6 degree of freedom
Today, 4: position, depth, roll
Mini PlayAnywhere
Future Devices
Canesta, VKB, Virtual Devices
Symbol laser projector
A common problem in vision‐based HCI
Tracking the hands, but… how to “drop” it?
How to get to Buxton’s 3‐state model?
null tracking selection
hands off moving click
pen up pen down
Pinching: touching thumb and forefingerUnambiguous to the userDiscrete signal maps to discrete inputStable transition in and outStable transition in and out
Discrete sensing for discrete state
not clicked
hand size
t
closed hand
open hand
not pinched
button state
tclicked
pinch state
t
not pinched
pinched
Pinching: touching thumb and forefinger
Ergonomics
Natural analogues:Tugging on a piece of fabric
Using a stylus
Picking up a small object
Some previous pinching work
Fakespace Pinch GlovesVideoDesk (Myron Krueger)
Visual TouchPad (Malik and Laszlo) Sato et al (demo, this conference)
Above the keyboard vision
Quek & Mysliwiec, FingerMouse Kjeldsen & Kender
Wilson & Cutrell, FlowMouse
Recognition problem
The technique
12
1
One shape Two shapes
Not pinching Pinching
Connected componentsConsider an image as an undirected graph where
each node corresponds to a pixel, and each node has edges to neighboring nodes (pixels) of the same value
A set of pixels is a connected component if for every pair of pixels u and v there is a path from u to v
A connected component can often correspond to a distinct object
Image processing
Background subtraction
Connected components analysisConnected components analysis
Count the number of components, pick the smallest
video
Cursor controlPinching is a natural “clutch”
“tap and a half” for clickOpen, close, open
DraggingDraggingOpen, close quickly
Free transformsTranslation: change in position
Rotation: change in orientation of ellipse
Scale: change size of ellipse
Two hands
Limitations
Only as robust as the segmentation
Dependent on line of sight
Not a full 3‐state interaction model
Tracked position is not the finger tipImplications for direct manipulation framework
Motion is (mostly) relative, like the mouse
The Orb Platform
XWand, CHI 2003
The Orb PlatformNew hardware, designed in coop with
Steve Bathiche (MS Hardware), and Mike Sinclair (MSR)3 magnetometers2 MEMS gyros2 MEMS gyros3 MEMS accelerometersBluetooth support
Vast improvement in orientation sensing over original XWand
Applications in SpotLight, VIBE wall large display, PAN with SmartPhone
More of a platform approachLayout/PCB is easy, software is not, &
people are picky about form factorServe a variety of needs around MSR video
Orientation with Magnetometers & Accelerometers
3 mags or 3 accels alone doesn’t cut it, but combination does
g
N
Take cross product of mags and accels
Caveats
Only correct when still
Magnetic north wanders indoors
This formulation gives priority to one of the sensors
N and g must not be colinear
Coffee Compass
with Raman Sarin
Most interfaces attempt to do t htoo muchAnd become unusable as a result
Coffee compass is familiar, kitschy, easy‐to‐use, delightful, humanizing
“where’s the nearest Starbucks?”
Questions