Big worldofsmallmotions iccp13

A Big World of Small Motions

Michael Rubinstein

MIT CSAIL

ICCP 2013

Imperceptible Changes and Motions in the World

Michael Rubinstein, MIT 2013


Blood flow

Building and structuremovements

Camera moves due to motion of shutter and mirror

BreathingEye movements (Microsaccades)Michael Rubinstein, MIT 2013


Low frequency motions Mid-range frequency motionsMichael Rubinstein, MIT 2013

Talk Overview

• Recap: Eulerian Video Magnification (SIGGRAPH’12)– With Hao-Yu Wu, Eugene Shih, John Guttag,

Fredo Durand, William T. Freeman

• EVM in the Wild

• Phase-based Video Motion Processing (SIGGRAPH’13)– With Neal Wadhwa, Fredo Durand, William T. Freeman


Talk Overview



• EVM in the Wild



Eulerian Video Processing

• Each pixel is processed independently

• We treat each pixel as a time series and apply signal processing to it

y

xtime


Method Pipeline

Spat

ial

Dec

om

posi

tion

Spat

ial

Dec

om

posi

tion

Tem

pora

l fi

lter

ing

Spat

ial

Dec

om

posi

tion

Tem

pora

l fi

lter

ing

α1

α2

αn-1

αn

Σ

Σ

Σ

Σ

Spat

ial

Dec

om

posi

tion

Tem

pora

lfi

lter

ing

α1

α2

αn-1

Rec

onst

ruct

ion

αn

Σ

Σ

Σ

Σ


Color Amplification Results

Source Color-amplified (x100)0.83-1 Hz (50-60 bpm)


Bruce Wayne’s Pulse

Batman Begins (2005), courtesy of Warner Bros. Pictures


Related Work: Pulse Detection in Videos

“Cardiocam” [Pho, Picard, McDuff 2010]

“Vital Signs Camera” – Philips(proprietary)


Now it gets interesting: Why It Amplifies Motion


Relating Temporal and Spatial Changes

Motion-magnified

Courtesy of Lili Sun


Relating Temporal and Spatial Changes

Space

Inte

nsi

ty

0

Signal Shifted signal Motion-magnified signal


Motion Magnification Results

Source Motion-magnified (0.4-3 Hz, x10)


Selective Motion Magnification in Natural Videos

Source(600 fps)

72-92 HzAmplified

Low E (82.4 Hz)

A (110 Hz)100-120 HzAmplified


Related Work: Motion Magnification [Liu 2005]

Liu et al. Motion Magnification, SIGGRAPH 2005

Source Motion-magnified


Related Work: Motion Magnification [Liu 2005]

+ +

++ +

Liu et al. Motion Magnification, 2005Michael Rubinstein, MIT 2013

Talk Overview



• EVM in the Wild



EVM in the Wild: Pregnancy

Original Processed

“Tomez85” https://www.youtube.com/watch?v=J1wvFmWv7zYMichael Rubinstein, MIT 2013

https://www.youtube.com/watch?v=J1wvFmWv7zY

EVM in the Wild: Pregnancy

“Tomez85” https://www.youtube.com/watch?v=gDpNN4g1klUMichael Rubinstein, MIT 2013

https://www.youtube.com/watch?v=gDpNN4g1klU

https://www.youtube.com/watch?v=gDpNN4g1klU

EVM in the Wild: Blood flow Visualization

Institute for Biomedical Engineering, Dresden Germanyhttps://www.youtube.com/watch?v=Nb18CRVmXGY

Red = high blood volumeBlue = low blood volume


https://www.youtube.com/watch?v=Nb18CRVmXGY

EVM in the Wild: Guinea Pig!

“SuperCreaturefan”: “Guinea pig Tiffany is the first rodent on Earth to undergo Eulerian Video Magnification.”http://www.youtube.com/watch?v=uXOSJvNwtIk

Source Motion-magnified


http://www.youtube.com/watch?v=uXOSJvNwtIk

VideoScope by Quanta Research Cambridge


Independent (Real-time) Ports

“webcam-pulse-detector”(Python + openCV)

+tracking

“VAmp - Video Amplifier”(Java)


Talk Overview



• EVM in the Wild



Phase-based Motion Magnification

Source LinearSIGGRAPH’12

Phase-basedSIGGRAPH’13


Phase-based Motion Editing

Exact for sinusoids!

True shifted sinusoid


Linear Pipeline (SIGGRAPH’12)

Laplacian pyramid[Burt and Adelson 1983]

Temporal filtering on intensities


Phase-based Pipeline (SIGGRAPH’13)

Complex steerable pyramid[Portilla and Simoncelli 2000]

Temporal filtering on phases

PhaseAmplitude


Improvement #1: More Amplification

Improves the bound by a factor of 4!

(derivation in the paper)

Amplification factor Motion in the sequence

Range of linear method:

Range of Phase-based method:


Improvement #2: Better Noise Performance

Noise amplified Noise translated


Results: Phase-based vs. Linear

Linear (SIGGRAPH’12) Phase-based (SIGGRAPH’13)

Clipping artifacts nearSharp edges and larger motions


Results: Phase-based vs. Linear

Linear (SIGGRAPH’12) Phase-based (SIGGRAPH’13)Michael Rubinstein, MIT 2013

Phase-based Motion Attenuation

Source Linear Motion attenuation +Color amplification

Amplifies colorAnd motion jointly

Amplifies colorWithout amplifyingmotionMichael Rubinstein, MIT 2013

Phase-based Motion Attenuation

Source Phase-based motion attenuation

Courtesy of YouTube user ComputerPhysicsLabSimilar to Motion Denoising

[Rubinstein et al. 2010][Bai et al. 2012]


Details Skipped


Revealing Invisible Changes in the World

• NSF International Science and Engineering Visualization Challenge (SciVis), 2012

• Science Vol. 339 No. 6119 Feb 1 2013


Conclusions

• The world is full of small motions and changes we cannot normally see

• We develop algorithms to analyze and visualize them through videos

– Many potential uses in medical applications and scientific analysis

• Phase-based Video Motion Processing (to be presented at SIGGRAPH’13)

– More magnification, less noise

– Phase-based motion analysis is around for a while (Fleet and Jepson 1990) but not commonly used for editing

• New paper and code available soon!


Thank you!

Michael RubinsteinMIT CSAIL

(Graduating ~Dec 2013…)

Documents

Big worldofsmallmotions iccp13