christian holzpatrick baudisch

high-precision touch inputbased on fingerprint recognition

fachgebiet human-computer interaction

occlusion

fat finger

so touch is inaccurate

or is it?

could it bethat it is not the fingers but our touch devices that are wrong?

Part 1 (science):even though screens are 2D, pointing is not

Part 2 (engineering):sensing fingers in 3D highly accurate touch

no fat fingerwe claim there is

problem

perceivedinstead, almost all observed targeting error comes from

probleminput point

perceived input point problem

target

[Benko, Wilson, & Baudisch 2006]

touch device perceives

offset

why we hope it’s the perceived input point problem?

the fat finger problem, in contrast is always noise = error

we can correct for it

offset

why we hope it’s the perceived input point problem?

the fat finger problem, in contrast is always noise = error

we can correct for it

while there is always an offset, we hypothesize thatthe offset depends on the pointing situation

our main hypothesis

so what does “pointing situation” mean?

!= [iPhone, Wang et al.]

1yaw

!=[Forlines et al., CHI’07]

2pitch

!=

3roll

!=

4 finger shape

!=

4mental model

(… and there might be more e.g., head position/parallax…)

we ran auser studya non 2D-model

xy xy

current model

touch pad

screen

nD xy

proposed model

touch pad

screen

we ran auser studyuser study 1

task

1. target here

2. hit okay

task

1pad rotation (yaw)

90° 45° 15° 0° -15°

roll2roll

3pitch

90° 65° 45° 25° 15°

4user12 participants

(all students, so differencesamong them will be lower bound)

footswitch

on-screeninstructions

controlledhead position parallax

capacitivetouch pad

every trial recorded as a dot at the touch location

dependent

we measure targeting accuracy assuming perfect calibration size of ellipse that contains 95% of all samples.

example

7.5

mm

1.5 cm

main effects forroll, pitch, yaw, & participantID

hypotheses

2 pad rotations× 2 sessions (pitch, roll)× 5 angles× 6 repetitions per angle× 5 blocks

= 600 trials / participant

12 participants design

1 2 3 4 5 6

results

if the additional IVs had no impact,we would expect to see something like this

-15°0°

15°45°90°

rotate condition

no-rotate condition

but touch locations do indeed fall into clusters…

know

not

...

know

use

r

know

yaw

know

use

r...

know

3D

OF

know

use

r...0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

error bars are standard deviation

butt

on s

ize in

cm

for 9

5% a

ccur

acy results

requires 5.2mm button

~three times more accurate allow three times smaller device

trad

ition

alca

paci

tive

requires 15mm button

(participant #4, roll varied)

target

1cm

1pad rotation (yaw)

1pad rotation (yaw)

-15°0°

15°45°90°

rotate condition

no-rotate condition

(participant #4, roll only)2roll

1cm

3pitch 10

25

45

65

90

all data by participant #1-6

1 2 3 4 5 6

tilt

roll4users

7 8 9 10 11 12

tilt

roll

all data by participant #7-124users

know

not

...

know

use

r

know

yaw

know

use

r...

know

3D

OF

know

use

r...0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

error bars are standard deviation

spre

ad in

cm

resultsrequires 5.2mm button

trad

ition

alca

paci

tive

requires 15mm button

how (in)accurate current devices are (button must be that big)

if we knew thepad orientation

if we knewfinger angles

also need to know user ID, or we will overcompensate for people like this one

shouldn’t we be able to make such a device?

Part 1 (science):even though screens are 2D, pointing is not

Part 2 (engineering):sensing fingers in 3D highly accurate touch

optical tracker

what do you mean: “not very practical”? retro reflective markers on finger… 6-16 camera setup…

makes a great “gold standard” implementation to test the concept

mobileok, maybe something a bit more

gets everything a traditional touchpad gets+ roll, pitch, yaw, & participantID

devices that sense touch and fingerprint already exist

this is very different from micro rolls [CHI 2009]

calibrationhave user touch a known target repeatedly and with different finger postures create database of (fingerprint, target offset)

useobtain fingerprint as user touches the devicelook up similar fingerprints in the databaseaggregate associated offsets (k nearest neighbor) and apply it

algorithm

user study 2

1 tracking device

fingerprintoptical tracker

“simulated capacitive” (just contact area)

2rotation

3 roll & pitch

roll -15° 0° 15° 45° 90°

pitch

15°

25°

45°

65°

90°

optical beats simulated capacitive by ~3x(based on user study 1)

fingerprint beats simulated capacitive(let’s find out by how much)

hypotheses

2 rotations× 13 angles× 5 repetitions per angle× 5 blocks

= 650 trials / participant

12 participants design

results

5.00

4.00

3.00

2.00

1.00

0.00

Error bars: +/- 1 SE

Mea

n spr

ead i

n mm

rawcapacitive

rotation-aware

capacitive

fingerprint-based

correction

tracker-based

correction

5.00

4.00

3.00

2.00

1.00

0.00

Error bars: +/- 1 SEM

ean s

prea

d in m

m

rawcapacitive

rotation-aware

capacitive

fingerprint-based

correction

tracker-based

correctionerror bars are standard deviation

spre

ad in

cm results

simulatedcapacitive

5.00

4.00

3.00

2.00

1.00

0.00

Error bars: +/- 1 SEM

ean

spre

ad in

mm

rawcapacitive

rotation-aware

capacitive

fingerprint-based

correction

tracker-based

correctionfingerprint optical

as expected a factor of 3x

works!

potential for improvement

conclusions

benefits

1. make more reliable touch input devicesenter text on mobile touch device with high accuracy

2. avoid need for targeting aidssuch as offset cursor, shift, zooming,as they cost time and make touch less “direct”

3. make smaller mobile touch devicesbring touch input to watch-size mobile devices

use roll/pitch/yaw/userID touch device to

2/3 (7/8 of surface) of “fat finger problem”really stem from an oversimplified model of touch

touch is not 2D

model

find a closed representation of user profile speed up learning

combine with in-cell touch screens make small

next steps

fachgebiet human-computer interaction

thanks to my new group athasso plattner institutein berlin/potsdam

Christian HolzPh.D. Student, masters from Hasso Plattner InstituteMasters project with Steve Feiner at Columbia University, New York

.

Gerry Chuintern at Hasso Plattner InstituteMasters from U of Toronto

.

joe konstan: university of minnesotadaniel fisher: microsoft researchgary marsden: south africa, capetownmichael rohs: telekom labsscott klemmer: stanfordmark billinghurst: hitlab new zealandlucia terrenghi: vodaphone

come visit

fachgebiet human-computer interaction

open Ph.D./post doc position

all people

10 22 45 60 90

-10

0

10

45

90

without sense of rotation

10 22 45 60 90

-10

0

10

45

90

all people

with sense of rotation

0.40

0.30

0.20

0.10

0.00

Error bars: +/- 1 SE

raw capacitive rotation-awarecapacitive

per-anglecapacitive

Mea

n sp

read

in c

m

per user spread

Professor in computer science at Hasso Plattner Institute2002- research scientist at Microsoft Research, Redmond, WA2003- affiliate professor at University of Washington Seattle, WA2000-2002 research scientist at Xerox PARC2000 Ph.D. in computer science from TU Darmstadt

patrickbaudisch

Sean GustafsonPh.D. StudentMasters University of Manitoba, Canada on visualization, off-screen pointing

.

spatial cognitionon mobile