Force from Motion: Decoding Physical Sensation from a First Person Video

Hyun Soo Park, Jyh-Jing Hwang, and Jianbo Shi

Biker

Biker

Mountain biking

Can we understand activity?

Mountain biking

Can we understand detailed activity?

Can we understand physics of activity?

1. Gravity Gyroscope

2. Speed

Accelerometer

Can we understand physics of activity?

Pedaling

Braking

Pedaling Braking

3. Control

Can we understand physics of activity?

First person camera

Can we understand physics of activity?

mg

mg

Roll torque

Pedaling

braking

Thrust force

Pedaling

braking

Thrust force

Air drag

3D reconstruction

3D reconstruction

3D reconstruction

3D camera trajectory

3D scene points

3D reconstruction

3D camera trajectory

3D scene points

Is geometry or kinematics enough to understand the biker’s behaviors?

What causes motion?

mg

1. Compute Gravity

Water

Tree

Horizon

Prediction via CNN

Ground truth

Prediction error: 0.5 degree

(g | )p

( | g)p

Visual Gravity Prediction

ˆ ˆ ˆ∝ ∏1(g | , , ) (g) ( | g)

N

N ii

p p p

mg

mg

mg

2vm

r

2. Compute Speed

cCamera trajectory Velocity: c

vd

=dt

Camera trajectory Velocity:

where is arbitrary scale. α

α cv

d=

dtαc

θb

mg

ctrpetalFθ

θα

θ =

=

sin cos

cosb ctrpetal

ctrpetal

b

b

m a

m

g FDynamic balance:

α θ= btanctrpetal

a

gScale factor:

Measured by 3D reconstruction

9.81 m/s2

Banked turn

3. Compute Control

Pedaling

Braking

= = +∑ ∑2

2

dd

FFt

Fc

m pass actve ivei

Linear force: ( )c tPosition: ( )q tAngle:

F F

= = +∑ ∑2

2

dd

FFt

Fc

m pass actve ivei

Linear force: ( )c tPosition: ( )q tAngle:

NFFF

mg

ctrpetalF

mg

Air drag

= = +∑ ∑2

2

dd

FFt

Fc

m pass actve ivei

Linear force: ( )c tPosition: ( )q tAngle:

thrustF

Pedaling

Braking

= = +∑ ∑2

2

dd

FFt

Fc

m pass actve ivei

Linear force: ( )c tPosition: ( )q tAngle:

+ × = +∑ ∑2

2

d d dd d d

q q qJ J T

t tT

t p actassiv ivee

: Moment of inertia J

pitchT

Angular force (torque):

rollT

yawT

( )c tPosition: ( )q tAngle:

( , ) = ODE( , , , )F TTc q Factpassive passiive acv tivee

( )c tPosition: ( )q tAngle:

( , ) = ODE( , , , )F TTc q Factpassive passiive acv tivee

where

≈ (t,R)

[ ]P = K R t

( , ) = ODE( , , , )F TTc q Factpassive passiive acv tivee

where

≈ (t,R)

[ ]P = K R t

( , ) = ODE( , , , )F TTc q Factpassive passiive acv tivee

≈ (t,R)

where [ ]P = K R t

λ+SfM regminimize ( )F,T,X

E E F,T

subject to

Reprojection error

Inverse control:

(t,R) = ODE( , )F T

( , ) = ODE( , , , )F TTc q Factpassive passiive acv tivee

≈ (t,R)

where [ ]P = K R t

λ+SfM regminimize ( )F,T,X

E E F,T

subject to

Temporal regularization

Inverse control:

(t,R) = ODE( , )F T

Head-mounted camera

Body-mounted camera Body-mounted IMU

Head-mounted IMU

Brake monitoring camera Pedal monitoring camera

Evaluation

Mean: 15.63 degree

Mean: 12.42 degree

Mean: 17.09 degree

Mean: 1.76 degree

Camera IMU

Pedaling activity (acceleration) Braking activity (deceleration)

Pedaling

Braking

Pedaling

Braking

= = +∑ ∑2

2

dd

FFt

Fc

m pass actve ivei

Pedaling activity (acceleration) Braking activity (deceleration)

Pedaling activity (acceleration) Braking activity (deceleration)

Pedaling

Braking

Pedaling activity (acceleration) Braking activity (deceleration)

Pedaling

Braking

Pedaling activity (acceleration) Braking activity (deceleration)

Pedaling

Braking

3D reconstruction

3D reconstruction

3D reconstruction

First person vision Third person vision

Force from Motion: Decoding Physical Sensation from a First Person Video http://www.seas.upenn.edu/~hypar/ffm.html