Course 11tracker/media/pdf/SIGGRAPH2001_Slides_11.pdfProvides angle measurements Provides angle...

Course 11Course 11

Tracking: Beyond 15 Minutes of ThoughtTracking: Beyond 15 Minutes of Thought

IntroductionsIntroductions

Danette Danette AllenAllenGreg WelchGreg WelchGary BishopGary Bishop

Why thinking about tracking isso funWhy thinking about tracking isso fun

ItIt’’s a simple problem to states a simple problem to stateIt has a little of everythingIt has a little of everything

• A little physics

• A little electronics

• A little math

• A little signal processing

• A little programming

Why don’t you just…Why don’t you just…

•• Mount TV cameras on the walls?Mount TV cameras on the walls?

•• Use GPS?Use GPS?

•• Use MEMS accelerometers?Use MEMS accelerometers?

•• Use carbon Use carbon nanotubesnanotubes??

Con

fide

nce

Time

The 15 minute effect...The 15 minute effect...

GoalsGoals

•• To get you to the 15 minute point andTo get you to the 15 minute point andbeyond...beyond...

•• To equip you to evaluate the variousTo equip you to evaluate the variousofferings and understand the strengths andofferings and understand the strengths andweaknesses of each.weaknesses of each.

Tracking TechnologiesTracking Technologies

(Danette Allen)

Many ways to slice!Many ways to slice!

ConfigurationConfiguration• Outside-in vs. Inside-out

Type of measurementType of measurement• Absolute vs. Relative

• Range vs. angle

Physical mediumPhysical medium• Five categories

Five (Six) TechnologiesFive (Six) Technologies

InertialInertialAcousticAcousticMagneticMagnetic

MechanicalMechanicalOpticalOptical

Radio (GPS) is the sixth...Radio (GPS) is the sixth...• typically used outdoors

• not addressed in this course

Inertial TrackingInertial Tracking

PassivePassiveNewtonNewton’’s 2nd Law of Motions 2nd Law of Motion•F = ma (linear)

• ττττ = Iαααα (rotational)

No physical limits on working volumeNo physical limits on working volumeAccelerometers and gyroscopesAccelerometers and gyroscopes

• Derivative measurements

AccelerometersAccelerometers• Measure force exerted on a mass since we cannot measure

acceleration directly.

• Proof-mass and damped spring

• Displacement proportional to acceleration

• Potentiometric and piezoelectrictransducers

Inertial TrackingInertial Tracking

m

3 2 1 0 1 2 3

Inertial TrackingInertial Tracking

• Gimbal deflection

GyroscopesGyroscopes• Inertia

rigidity in space

• Precessiona comparatively slow gyration of the rotation axis of aspinning body about another line intersecting it so asto describe a cone (Mirriam-Webster)

(Discovery, 2001)

MicrogyroscopeMicrogyroscope

(Systron Donner, 2001)

MEMSMEMSTuning forkTuning fork

Ring-laserRing-laserand Fiberand Fiber

• Doppler effect

• Beat frequency

Axis of rotation

( )

→→→→⊕→→→

↓

xs

xs

xx

sa

ˆ1ˆ1ˆ?

&&&&&

εUser

LaPlace Transforms = σ + jω

Inertial DriftInertial Drift

Error accumulation due to integrationError accumulation due to integration• Poor SNR at low frequencies

• Inverse square weighting of noise

• Gravity vector misalignment• 1º tilt error over 10 seconds ⇒ 9m position error

Periodic recalibrationPeriodic recalibration• hybrid systems typical

Time [s] to 0.1 [m] ErrorTime [s] to 0.1 [m] Error

Effect of Sensor Noise on Inertial-Only Performance (time to 0.1 meters error)

Rate gyro dynamic range (5rad/s divided by noise) bits

Acc

eler

omet

er d

ynam

ic r

ange

(3G

div

ided

by

nois

e) b

its

The GeometryThe Geometry• The intersection of 2 spheres is a circle.

• The intersection of 3 spheres is 2 points.

• One of the two points easily eliminated

Speed of SoundSpeed of Sound• Varies with temperature and pressure

• ~ 331[m/s] in air at 0º C

• 1 ft/ms ⇒ SLOW!!

UltrasonicUltrasonic• 40 [kHz] typical

Acoustic TrackingAcoustic Tracking

Acoustic Tracking MethodsAcoustic Tracking Methods

Time of FlightTime of Flight• Measures the time required for a sonic pulse or pattern to

travel from a transmitter to a receiver.

• d [m] = v [m/s] * t [s], v = speed of sound (c)

• Absolute range measurement

Phase CoherencePhase Coherence• Measures phase difference between transmitted and received

sound waves

• Relative to previous measurement

• still absolute!!

Phase CoherencePhase Coherence

EquationsEquations• A cos(ωt - φ)

• c[m/s] = λ[m] * f[1/s]

• δ[m] = λ[m] * (φ/2π)

““RelativeRelative”” Result Result• Fractional wavelength

• Need previous range estimate

•No integration!!!

Magnetic TrackingMagnetic Tracking

Three mutually-orthogonal coilsThree mutually-orthogonal coils•

• Each transmitter coil activated serially

• Three measurements apiece(three receiver coils)

• Nine-element measurement for6D position

AC AC vsvs. DC. DC• Ferromagnetic interference

θπθθ

πsin

32cos

32 d

MH

d

Mr

H ==

Mechanical TrackingMechanical Tracking

Ground-based or body-basedGround-based or body-basedUsed primarily for motion captureUsed primarily for motion captureProvide angle and range measurementsProvide angle and range measurements

• gears

• bend sensors

Elegant addition of force feedbackElegant addition of force feedback

Optical TrackingOptical Tracking

Provides angle measurementsProvides angle measurements• One 2D point

defines a ray

• Two 2D pointsdefine a pointfor 3D position

• Additional 2Dpoints requiredfor orientation

Speed of lightSpeed of light• 2.998 * 108 m/s (1 ft/ns)

Cθ, φ

C

Active vs. Passive TargetsActive vs. Passive Targets

Typical detectorsTypical detectors• Video and CCD cameras

• Computer vision techniques

Passive targetsPassive targets• Reflective materials, high contrast patterns

CCD cell/pixel

Active vs. Passive TargetsActive vs. Passive Targets

Typical detectorsTypical detectors• LEPDs

•

• Quad Cells

•

Active targetsActive targets• LEDs

4 1

3 2

( ) ( ) ( ) ( )4321

3241

4321

4321

iiii

iiiiy

iiii

iiiix

++++−+=

++++−+=

( )( )

−=

−=

LxL

IIorL

xLII

0)sinh)(sinh

0 αα x

I

L

I ≈

Many ways to slice!Many ways to slice!

ConfigurationConfiguration• Outside-in vs. Inside-out

Type of measurementType of measurement• Absolute vs. Relative

• Range vs. angle

Physical mediumPhysical medium• Five categories

next

Source/Sensor ConfigurationsSource/Sensor Configurations

(Gary Bishop)(Gary Bishop)

Sensor ConfigurationsSensor Configurations

Geometric arrangement ofGeometric arrangement ofsensors and sources impacts:sensors and sources impacts:• accuracy

• usability

• algorithms

for example CODA mpx30for example CODA mpx30

3 1-D 3 1-D CCDs CCDs are stationaryare stationaryLED targets moveLED targets move

Very interesting optics and sensingVery interesting optics and sensing

CODA mpx30CODA mpx30

•• Measures angles in lab coordinate frameMeasures angles in lab coordinate frame

•• Angle determines a planeAngle determines a plane

•• Intersecting 3 planes determines a pointIntersecting 3 planes determines a point

“Flatland”

CODA mpx30CODA mpx30

Such Such ““outside-looking-inoutside-looking-in”” systems systems

•• measure position very wellmeasure position very well

•• allow many small moving targetsallow many small moving targets

•• use multiple targets to get orientationuse multiple targets to get orientation

•• trade off accuracy and working volumetrade off accuracy and working volume

•• provide larger volume / more accuracy withprovide larger volume / more accuracy withmore sensorsmore sensors

•• use really simple mathuse really simple math

HiBallHiBall

6 2-D sensors and 6 lenses in dodecahedron6 2-D sensors and 6 lenses in dodecahedron10001000’’s of s of LEDs LEDs fixed on the ceilingfixed on the ceilingCalibration gives effectively 26 camerasCalibration gives effectively 26 cameras

•• Measures angles in user coordinate frameMeasures angles in user coordinate frame

•• Angles determine a constraint relatingAngles determine a constraint relating• position

• orientation

• view

• led location

HiBallHiBall

“Flatland”

HiBallHiBall

Such Such ““inside-looking outinside-looking out”” systems systems•• directly measure orientationdirectly measure orientation•• allow large working volume with accuracyallow large working volume with accuracy•• are larger than LED targetsare larger than LED targets•• and thus harder to use for hands, feet, etc.and thus harder to use for hands, feet, etc.

Arc Second VulcanArc Second Vulcan

Sources scan Sources scan ““planes ofplanes oflightlight”” through space through space

Sensors on target detectSensors on target detectpassing planepassing plane

Sources

Sensors

Arc Second VulcanArc Second Vulcan

•• Time of plane passing converts to angle atTime of plane passing converts to angle atthe sourcethe source

•• Measures angles in world frameMeasures angles in world frame

•• Thus like CODA and other Thus like CODA and other ““outside-outside-looking-inlooking-in”” systems systems

•• Direction of Direction of ““lookinglooking”” really isn really isn’’t the issuet the issuebut coordinate frame of measurementbut coordinate frame of measurement

User and SensorUncertainty/Information

User and SensorUncertainty/Information

(Greg Welch)(Greg Welch)

Pose UncertaintyPose Uncertainty

•• Measurement uncertaintyMeasurement uncertainty• Pose estimates from noisy sensor measurements

•• User pose uncertaintyUser pose uncertainty• Noisy and temporally-discrete measurements

• Modeling user motion is difficult [Weber]

• Modeling pose uncertainty is less difficult

Noise-Driven ProcessesNoise-Driven Processes

∫Randomnoise ∫

Positionuncertainty

Velocityuncertainty

Randomnoise

Noisymeasurement

“True” sensormeasurement

Σ

User

Tracker

Random Variables and SignalsRandom Variables and Signals

•• Map sample space Map sample space →→→→→→→→ real numbers real numbers• For example, time to voltage

•• Random SignalsRandom Signals• For example, electrical signals

• Continuous random variables

• Probability over a region of sample space

• Spatial (statistical) and temporal (spectral) aspects

Cumulative Distribution FunctionCumulative Distribution Function

F x P xX ( ) = −∞( ],

1. as

2. as

3. is a non - decreasing function of

F x x

F x x

F x x

X

X

X

( ) → → −∞

( ) → →+∞

( )

0

1

Probability Density FunctionProbability Density Function

f xddx

F xx X( ) = ( )

1. is a non - negative function

2.

f x

f x dx

X

X

( )

( ) =−∞

∞

∫ 1

Probability (Continuous)Probability (Continuous)

P a b f x dxx xa

b

,[ ] = ( )∫

Statistical MomentsStatistical Moments

µmmE X= [ ] =

x f x dxmX ( )

−∞

∞

∫ x p xmX

x

( )∑Continuous: Discrete:

1st Moment or Mean1st Moment or Mean

µ = [ ] =E X

xf x dxX ( )−∞

∞

∫ xp xXx

( )∑Continuous: Discrete:

Central MomentsCentral Moments

c E Xmm= −( )[ ] =µ

x f x dxmX−( ) ( )

−∞

∞

∫ µ

Continuous: Discrete:

x p xmX

x

−( ) ( )∑ µ

2nd Central Moment or Variance2nd Central Moment or Variance

V X E X

E X

[ ] = −( )[ ]= [ ] −

µ

µ

2

2 2

“Mean of square minus square of mean”

Standard DeviationStandard Deviation

σ = [ ]V X

Gaussian/Normal DistributionGaussian/Normal Distribution

x →+∞−∞ µ

f xX ( )σ

f x eX

x

( ) =− −( )1

2

1

2 22

πσσ

µX ~ ,N µ σ 2( )

Autocorrelation (Time Domain)Autocorrelation (Time Domain)

RX τ()

0 ττ–

X1

X2

R E X t X tX τ τ( ) = ( ) +( )[ ]

Spectral Density (Frequency Domain)Spectral Density (Frequency Domain)

S j F R

R e d

X X

Xj

ω τ

τ τωτ

( ) = ( )[ ]

= ( ) −

−∞

∞

∫

The Wiener-Khinchine relationship

White Noise ProcessWhite Noise Process

RX τ()

0 ττ– 0 ω

SX jω()

RA

X ττ

( ) ==

if then

else

0

0S j AX ω( ) =

Growth in Pose UncertaintyGrowth in Pose Uncertainty

V X wdt[ ] = ∫0

w q~ ,N 0( )where

and “white.”

Control of Pose UncertaintyControl of Pose Uncertainty

Measurements ⇒ pose information

Sensor MeasurementsSensor Measurements

BreakBreak

(15 Minutes)(15 Minutes)

Traditional ApproachesTraditional Approaches

(Gary Bishop)(Gary Bishop)

Traditional Solution MethodsTraditional Solution Methods

•• Simple problem: Determine pose givenSimple problem: Determine pose givensensor readings.sensor readings.•• Linear algebra taught us aboutLinear algebra taught us about

N equations in N unknownsN equations in N unknowns•• Each equation is a Each equation is a constraintconstraint•• 3 DOF3 DOF��������3 constraints & 6 DOF3 constraints & 6 DOF��������66

constraintsconstraints••Unfortunately often non-linear constraintsUnfortunately often non-linear constraints

often with multiple solutionsoften with multiple solutions

Range TrackerRange Tracker

Intersect 3 spheres

Unfortunately, the solution is ugly...

SimplifySimplify

a. put mike 0 at origin

b. put mike 1 out X axis 1 unit

c. put mike 2 out Y axis 1 unit

d. all 3 mikes in Z=0 plane

Usual coordinate transform to convert to“lab” coordinates

Simpler Range EquationsSimpler Range Equations

Note ambiguity

Optical with fixed 1D sensorsOptical with fixed 1D sensors

For example, 1D CCD with razor blade castingFor example, 1D CCD with razor blade castinga shadowa shadow

••Calibrate to determine 3D plane equationCalibrate to determine 3D plane equationfrom sensor reading (non-trivial)from sensor reading (non-trivial)

•• For each sensor reading, write a linearFor each sensor reading, write a linearequation relating unknown x,y,z to planeequation relating unknown x,y,z to plane

•• Solve the system of equations for x,y,zSolve the system of equations for x,y,z

Solve...Solve...

Optical with fixed 2D sensorsOptical with fixed 2D sensors

For example, two video cameras looking atFor example, two video cameras looking atLEDs LEDs on the user.on the user.

•• Could treat as four 1-D sensorsCould treat as four 1-D sensors

OROR

•• Calibrate to get ray equation from u,vCalibrate to get ray equation from u,v

•• Rays wonRays won’’t intersect!t intersect!

•• Minimize distance between themMinimize distance between them

Set up equationsSet up equations

Ray equations

Distance

Minimum distance line must beperpendicular to both rays, so...

SolveSolve

Distance out eachray to closest point

Halfway between

B is the baseline

Stochastic ApproachesStochastic Approaches

(Greg Welch)(Greg Welch)

MotivationMotivation

•• Take into accountTake into account• Stochastic nature of sensor signals

• Varying amounts of sensor information

• Model of user motion

•• Combine sensor/measurement informationCombine sensor/measurement information• Combat (otherwise growing) pose uncertainty

• Fuse information from heterogeneous sensors

State-Space ModelsState-Space Models

Begin with difference equation for process

y a y a y uk k k n k k n k+ − − += + + +1 0 1 1, ,K

Re-write as

x

y

y

y

y

a a a a y

y

y

y

k

k

k

k

k n

n n k

k

k

k n

+

+

−

− +

− −

−

−

− +

≡

=

1

1

1

2

0 1 2 1

1

2

1

1 0 0 0

0 1 0 0

0 0

0 0 0 1 0

M

L

L

M O M

+

1

0

0

1

M

u

State-Space ModelsState-Space Models

x Ax Gu

y Hxk k k

k k

+ = +=

1

A xi G

x

y

y

y

y

a a a a y

y

y

y

k

k

k

k

k n

n n k

k

k

k n

+

+

−

− +

− −

−

−

− +

≡

=

1

1

1

2

0 1 2 1

1

2

1

1 0 0 0

0 1 0 0

0 0

0 0 0 1 0

M

L

L

M O M

+

1

0

0

1

M

u

Observer Design ProblemObserver Design Problem

x Ax Gu

z Hx vk k k

k k k

= += +

− −1 1

Process noiseMeasurement noise

Optimal EstimationOptimal Estimation

c a t b t t dtT

= ( ) ( )( )∫ cost , ,0

cost = −a b cost = −( )a b 2

Integral of AbsoluteValue of Error (IAE)

Integral ofSquare of Error (ISE)

The Kalman FilterThe Kalman Filter

R.E. Kalman, 1960R.E. Kalman, 1960• Recursive optimal estimator

•Minimum variance of error

• Versatile & robust

•Estimation•Sensor fusion

• Robotics, navigation, computer vision, economics, …

• http://www.cs.unc.edu/~welch/kalman/

• Java-Based Learning Tool, books, papers, etc.

• ACM SIGGRAPH 2001 tutorial (earlier today)

predictpredict correctcorrect

PREDICTPREDICT

x Axk k−

−= 1

P AP A Qk kT−

−= +1

transitionuncertainty

CORRECTCORRECT

x x K z Hxk k k k= + −( )− −

K P H HP H RkT

kT= +( )− − −1

P I KH Pk k= −( ) −

“denominator”(measurement space)

predictedactual

Hybrid Systems andMulti-Sensor FusionHybrid Systems andMulti-Sensor Fusion

magnetic coil

magnetic coil

magnetic coil

accelerometer

accelerometer

accelerometer

gyroscope gyroscope

gyroscope

Incremental EstimationA Single Constraint at a TimeIncremental EstimationA Single Constraint at a Time

C = constraints needed for a unique solutionM = constraints used per estimate updateC = constraints needed for a unique solutionM = constraints used per estimate update

M = CM = C

M > CM > C

1 < M < C1 < M < C

M = 1111M = 1111

Unique SolutionUnique Solution

Under-constrainedUnder-constrained

SCAATSCAAT

Over-constrainedOver-constrained

Inter-Estimate SummaryInter-Estimate Summary

MultipleConstraints

SCAAT

Measurements(constraints)

Time

Benefits of SCAAT ApproachBenefits of SCAAT Approach

Purposefully using minimal constraintsPurposefully using minimal constraints• Avoid simultaneity assumption

• Temporal improvements

• Simplicity and flexibility

• Online source/sensor autocalibration

• Can be applied to virtually any tracking system

•Measurement model for each type of sensor•Dynamic model for user motion (possibly trivial)

Bishop, 1982

Error SourcesError Sources

(Greg Welch)(Greg Welch)

Error in Head PoseError in Head Pose

•• Hard to fool Hard to fool ““mother naturemother nature””• Lifetime of visual experience and expectations

• Visual-proprioceptive conflicts

• Virtual-real misregistration

•• What to do?What to do?• Some amount of error is unavoidable

• Understand sources and seek to minimize

Error ClassificationError Classification

•• Pose estimate life cyclePose estimate life cycle• Noisy sensor measurement → Estimate →

Transport → Transform → Display

•• Two primary classes of errorTwo primary classes of error• Static (spatial)

• Delay-induced (temporal)

Static Measurement ErrorStatic Measurement Error

•• Static field distortionStatic field distortion• Repeatable error in the measurement data

• “Bias” that might be corrected via calibration

•• Random noise or jitterRandom noise or jitter• Non-repeatable error

• Random (electrical) noise such as described earlier

• Often dependent on the current pose

Pose-Dependent Noise (Example)Pose-Dependent Noise (Example)

Baseline noise ξ0and coefficientsa, b, and c weredetermined off line.

Delay-Induced ErrorDelay-Induced Error

•• Measurement validityMeasurement validity• Good at sample time, then old (aging)

• Finite, non-zero sample time

• Old sample ⇒ misregistration

•• Motion Motion predicitionpredicition• Measure where you are, but want where you will be

• Later w/ Bishop

The Simultaneity AssumptionThe Simultaneity Assumption

MeasureCompute

UserMotion

Time

Moderate arm & wrist translation

Moderate head rotation

1/2 [s] • 3 [m/s] • 20-80 [ms] ⇒ 1-10 [cm]

1/2 [s] • 180 [°/s] • 20-80 [ms] ⇒ 6-25 [cm](at arm’s length)

First-Order Dynamic ErrorFirst-Order Dynamic Error

ε θθdyn,˙= ∆t

εdyn, ˙x x t= ∆

Tracker + graphicspipeline latencyInstantaneous

velocities

Synchronization DelaySynchronization Delay

A.k.a. phase delay or rendezvous delay

PoseEstimateTimeline

Total Tracker LatencyTotal Tracker Latency

Sample the sensor(s)Estimate cyclesynchronization

Estimate the poseServer buffersynchronization

Network transportClient buffersynchronization

Total Tracker ErrorTotal Tracker Error

Static errorSimultaneity assumptionTotal tracker latencyRemaining latency

Closing (Error Sources)Closing (Error Sources)

•• Did I mention error magnification?Did I mention error magnification?

•• Consider the technologyConsider the technology• Understand its limitations

• Stay within the envelope

•• Prediction (next)Prediction (next)

Motion PredictionMotion Prediction

(Gary Bishop)(Gary Bishop)

Motion PredictionMotion Prediction

End-to-end delayEnd-to-end delay•• hurts in VR / hurts worse in ARhurts in VR / hurts worse in AR•• sourcessources

• time to measure pose

• delay in communicating pose

• application response to change

• graphics update

• display refresh

What to do about delay?What to do about delay?

1.1. MonitorMonitor

2.2. MinimizeMinimize

3.3. MitigateMitigate

Latency is not only a tracker problem.

But mitigation is best handled at the tracker.

Can prediction help?Can prediction help?

BlueBlue��������no predictionno predictionRedRed��������w/out inertialw/out inertialGreenGreen��������w/ inertialw/ inertial

Can prediction help?Can prediction help?

Limits to predictionLimits to prediction

Prediction error grows Prediction error grows quadraticallyquadratically with withmotion bandwidth and prediction intervalmotion bandwidth and prediction interval

Prediction ideasPrediction ideas

•• Extrapolate past behavior to the futureExtrapolate past behavior to the future

•• The more history the betterThe more history the better

•• Correlations in the Correlations in the usersusers coordinate frame coordinate frame

•• Inertial sensors helpInertial sensors help

•• Monitor |predicted Monitor |predicted –– actual| for tuning actual| for tuning

•• Use image shifting to reduce jitterUse image shifting to reduce jitter

ConclusionsConclusions

(Gary Bishop)(Gary Bishop)

Final ThoughtsFinal Thoughts

•• No silver bulletNo silver bullet• Tracking anywhere for any purpose is a dream

•• No free lunch, only tradeoffsNo free lunch, only tradeoffs• Energy / Accuracy / Bandwidth / Latency / Noise

•• No end in sightNo end in sight• Lots of possibilities for interesting work

• ReActor not based on any of the principles described here

ResourcesResources

•• http://www.http://www.cscs..uncunc..eduedu/~/~welchwelch//kalmankalman•• check out Course 8 notescheck out Course 8 notes•• Dozens of books on KF, here are a fewDozens of books on KF, here are a few

• “Optimal Estimation with an ...” by Lewis

• “Introduction to Random Signals...” by Brown

• “Kalman Filtering Theory and Practice” by Grewal

•• Beginnings of a tracking bibliographyBeginnings of a tracking bibliography

Exhibits we’re going to check outExhibits we’re going to check out

• 3rd Tech (cool demo)

• 5DT

• Ascension (ReActor is a new method)

• InterSense

• Measurand

• MetaMotion / PhoeniX Technologies / Vicon

• Polhemus

EndEnd