Compressive Sensing of High-Dimensional Visual Signals

Aswin C Sankaranarayanan Rice University

Interaction of light with objects

Human skin Sub-surface scattering

Electron microscopy Tomography

Reflection

Refraction

Fog Volumetric scattering

The Plenoptic function Collection of all variations of light in a scene

Adelson and Bergen (91)

space (3D)

angle (2D)

spectrum (1D) time

(1D)

Different slices reveal different scene properties

space (3D)

angle (2D)

spectrum (1D)

time (1D)

Lytro light-field camera

High-speed cameras

Plenoptic function

Hyper-spectral imaging

Sensing the plenoptic function

•  Extremely high-dimensional –  1000 samples/dim == 10^21 dimensional signal –  Greater than all the storage in the world

•  Material costs

–  Sensing beyond the visible spectrum

•  Sensing time can be costly –  Medical imaging

•  Need low-dimensional methods for sensing!

Signal models

Many signals exhibit concise geometric structures

Signal models

Many signals exhibit concise geometric structures

Subspaces Union of subspaces

Sparse models

Low-dim. non-linear models

GOAL: Tractable low-dimensional models for parsimonious sensing and efficient processing of

high-dimensional visual data

Roadmap

space (3D)

spectrum (1D)

time (1D)

A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging

Roadmap

space (3D)

spectrum (1D)

time (1D)

A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging

•  Arbitrary signal

•  Measurement matrix of rank N

•  Number of measurements

•  Least squares recovery

Linear sensing

M measurements

N-dim signal

y = �x+ e

M � N

y � x

•  Is it possible to recover when ? •  No … for arbitrary

Linear sensing

M measurements

N-dim signal

y = �x+ e

x

M < N

y �

x

x

Simple example: 1-sparse signal

•  Signal prior: 1-sparse signal

•  Possible with M = 2 measurements!

compressive measurements

signal

1 nonzero entry

y = �x+ e

y1 =

X

k

xk = signal value

y2 =

X

k

kxk = signal value ⇤ location

K-sparse signals

•  Signal recovery –  K-sparse signal

–  Measurement matrix is i.i.d. sub-Gaussian

–  Number of measurements

–  Recovery via a convex program

compressive measurements

signal

nonzero entries

M ⇠ K log(N/K)

Candes,(Romberg,(Tao((06),(Donoho((06)(

y = �x+ e

•  Signal recovery –  K-sparse signal

–  Measurement matrix is i.i.d. sub-Gaussian

–  Number of measurements

–  Recovery via a convex program

Compressive sensing

compressive measurements

sparse signal

nonzero entries

M ⇠ K log(N/K)

y = �x+ e

Signal prior Measurement matrix design

Sampling rate

Recovery algorithm

Compressive sensing

•  System is under-determined •  Under-sampling factor: M/N

–  Smaller M/N implies gains when sensing is costly

compressive measurements

sparse signal

nonzero entries

y = �x+ e

�Single-Pixel� CS Camera

random pattern on DMD array

DMD DMD

single photon detector image

reconstruction or processing

Kelly lab

scene

Single pixel camera

•  Each configuration of micro-mirrors yield ONE compressive measurement

•  A single photo-

detector tuned to the wavelength of interest

•  Resolution scalable

Kelly lab, Rice University

Digital micro-mirror

device

Photo-detector

First Image Acquisition

target 65536 pixels

1300 measurements (2%)

11000 measurements (16%)

Roadmap

space (3D)

spectrum (1D)

time (1D)

A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging

SPC on a time-varying scene

•  Simple model for recovering video frames from a single-pixel camera (SPC): Group W measurements together to estimate each video frame (image)

•  Value of W ~ “aperture time” of the video camera

t=1 t=W measurements

initial estimate

!me$varying$

scene$

SPC Video Uncertainty Principle

Small%W%Less$mo!on$blur$

More$spa!al$blur$

$

Large%W%More$mo!on$blur$

Less$spa!al$blur$

$

t=1 t=W (small) t=W (large)

[Wakin, 2010]

W"

total%M

SE%

SPC Video Uncertainty Principle t=1 t=W (small) t=W (large)

Find$and$exploit$$

op/mal%balance%%between$spa!al$$

and$mo!on$blurs$

CS-MUVI (CS MUltiscale VIdeo recovery)

•  Step 1: Given knowledge of the speed of the scene’s motion, choose W to optimize the spatial/temporal resolution tradeoff

measurements

W"total$MSE$

[AS,$Studer,$Baraniuk$2012]$

t=T t=1 t=t0 t=t0+W t=W

CS-MUVI (CS MUltiscale VIdeo recovery)

•  Step 1: Choose W to optimize the spatial/temporal resolution tradeoff

•  Step 2: Acquire full-rate measurements using “low-pass” codes matched to blur induced by W W"

total$MSE$

measurements t=T t=1 t=t0 t=t0+W t=W

CS-MUVI (CS MUltiscale VIdeo recovery)

•  Step 2: Acquire full-rate measurements using low-pass codes matched to blur induced by W

•  Step 3: Recover low-resolution video frames from low-pass measurements (can use simple least squares)

b1 bj bF

measurements

low-resolution estimates

t=T t=1 t=t0 t=t0+W t=W

CS-MUVI (CS MUltiscale VIdeo recovery)

b1 bj bF

measurements

low-resolution estimates

t=T t=1 t=t0 t=t0+W t=W

•  Step 3: Recover low-resolution video frames from low-pass measurements (can use simple least squares)

•  Step 4: Estimate the optical flow between low-resolution frames

CS-MUVI (CS MUltiscale VIdeo recovery)

b1 bj bF

measurements

low-resolution estimates

t=T t=1 t=t0 t=t0+W t=W

•  Step 5: Acquire 2nd set of interleaved measurements using “full-resolution” codes

•  Step 6: Recover high-resolution video frames from 2nd set of measurements + low-res video frames, low-res optical flow, sparsity

Multiscale Sensing Codes

•  Problem: 2x measurement rate (low-res and high-res measurement codes)

•  Solution: Design full-res codes that become optimally matched low-res codes when downsampled in space (automatically interleaved)

•  Low-res: Hadamard code

•  High-res: Upsample Hadamard code using random innovations

Multiscale Sensing Codes

AS, Studer, Baraniuk (12)

1. Start with a row of the Hadamard

matrix

2. Upsample$3. Add

high-freq component

Key Idea: Constructing measurement matrices that have good properties when downsampled

Result

CS-MUVI on SPC

Single pixel camera setup

CS-MUVI: IR spectrum

Joint work with Xu, Studer, Kelly, Baraniuk

InGaAs Photo-detector (Short-wave IR) SPC sampling rate: 10,000 sample/s Number of compressive measurements: M = 16,384 Recovered video: N = 128 x 128 x 61 = 61*M

Recovered Video

Preview (initial estimate) Upsampled 4x

CS-MUVI on SPC

Conventional CS-based recovery

CS-MUVI Joint work with Xu, Studer, Kelly, Baraniuk

More results •  Number of compressive measurements: 65536 •  Total duration of data acquisition: 6 seconds

•  Reconstructed video resolution: 128x128x256

Preview (6 different videos) *animated gifs*

CS-MUVI (6 different videos) *animated gifs*

More results •  Number of compressive measurements: 65536 •  Total duration of data acquisition: 6 seconds

•  Reconstructed video resolution: 128x128x256

Preview (6 different videos) *animated gifs*

CS-MUVI (6 different videos) *animated gifs*

Static Image Full res.

Static Image 2x down.

Static Image 4x down.

CS-MUVI Preview

CS-MUVI 64x Comp.

Achievable Spatial resolution •  Comparing a recovered frame to a static image of the

scene

CS-MUVI summary

•  Key points –  Signal prior: Motion model –  Measurement matrix: Multi-

scale design

•  A practical video recovery

algorithm for the SPC –  Scales across wavelengths –  Extensions to hyper-spectral

video camera

•  Coming soon … to a camera near you

Roadmap

space (3D)

spectrum (1D)

time (1D)

A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging

Hyper-spectral data

•  Hyper-spectral data provides fingerprints for materials

•  Wide range of applications –  Agriculture, mineralogy –  Microscopy –  Surveillance, chemical

imaging

Hyper-spectral image of the Deepwater Horizon oil spill

(Image courtesy SpecTIR.com)

Hyper-spectral data

Signal models for HS data

•  Low rank

–  Rank depends on number of materials in the scene

–  Spectral unmixing problem

•  Sparsity –  be the image at wavelength i –  Sparse in a wavelet basis xi

[x1, x2, . . . , xL]

Low rank (subspace models)

•  Individual spectral images lie close to a subspace

•  Concise geometric structure: d-dim. space

Subspace

spanned by C

xi = C↵i

C 2 RN⇥d ↵i 2 Rd

d ⌧ N

C

Recovery problem

Given Recover Basis subspace coeff.

Challenges: Bilinearity of unknowns Sparsity of

C↵1,↵2, . . . ,↵T

yi = �ixi + ei

= �iC↵i + ei

C

Given

Structured measurement matrix

Recovery problem

yi = �ixi + ei

= �iC↵i + ei

[y1, y2, . . . yL] = �C [↵1, ↵2, . . . ↵L] + E

Singular value decomposition of [y1, y2, . . . yL] = U⌃V T

yi =

yiyi

�=

��i

�C↵i + ei

yi =

yiyi

�=

��i

�C↵i + ei

Given

Structured measurement matrix 1.  SVD to recover subspace coefficients 2.  Solve convex program to recover C

Recovery problem

yi = �ixi + ei

= �iC↵i + ei

min kCk1 s.t. 8i, kyi � �iC↵ik2 ✏

CS - Linear dynamical systems

•  Measurement matrix is structured

–  Alleviates the bilinearity of unknowns –  Solving a sequence of linear and convex programs –  A sparse prior to estimate subspace basis C

Common measurements

Innovations measurements

Estimate coefficients

Estimate matrix C

Scene

T:x1 Φ

tΦ~

T:y1

T:y~1

T:ˆ 1α

C

tt ˆCx α=T:x1

AS, Turaga, Chellappa and Baraniuk (10)(12)

Hyper-spectral imaging

•  Weather data –  2300 Spectral bands in

mid-wave to long-wave IR (3.74 – 15.4 microns)

–  Spatial resolution 128 x 64 –  Rank 5 Ground

Truth

2% 200x %1%%

M/N = 10% M/N = 2% M/N = 1%

(rank = 20)

Broader applications •  Low-rank models are

widely applicable

•  Video MRI

Ground truth M/N = 2%

Broader applications •  Low-rank models are

widely applicable

•  Video MRI

•  Classification on compressive data –  Purposive sensing

Traffic video

Ground truth Recovered video M/N = 4%

Classification performance in [%] (CS-LDS at 4% under-sampling)

AS(et(al.((10)(

Expt%1% Expt%2% Expt%3% Expt%4%

Oracle$LDS$ 85.71$ 85.93$ 87.5$ 92.06$

CSOLDS$ 84.12$ 87.5$ 89.06$ 85.91$

Summary

•  Visual signals are very high-dimensional –  Interplay between concise signal models and novel

imaging architectures

•  Signal models beyond sparsity –  Rich models from computer vision and image processing –  Not always easy to bridge the gap –  Measurement matrices beyond random iid

Collaborators$

Richard$Baraniuk$

Kevin$Kelly$ Christoph$Studer$ Lina$Xu$ Yun$Li$Pavan$Turaga$

Rama$Chellappa$

References$

Park and Wakin, “A multi-scale framework for compressive sensing of videos,” PCS 2009 Reddy, Veeraraghavan, Chellappa, “P2C2: Programmable pixel compressive cameras for high speed imaging,” CVPR 2011 Sankaranarayanan, Studer, Baraniuk, “CS-MUVI: Video compressive sensing for spatial multiplexing cameras,” ICCP 2012 Sankaranarayanan, Turaga, Chellappa, Baraniuk, “Compressive sensing of dynamic scenes,” SIIMS (under review)

Low resolution estimate t=T t=1 t=t0 t=t0+W t=W measurements

initial estimate

•  Initial estimate is called the preview •  Fast to compute!!!

•  Linear estimation •  Fast transform

•  Initial estimate is low-resolution

CS-MUVI (CS MUltiscale VIdeo recovery)

•  Estimate the optical flow between

low-resolution frames

•  Recovery of video at HIGH resolution with sparse models and optical-flow constraints

motion compensation

t=T t=1 t=t0 t=t0+W t=W measurements

initial estimate

CS-MUVI (CS MUltiscale VIdeo recovery)

motion compensation

t=T t=1 t=t0 t=t0+W t=W measurements

initial estimate

minPF

i=1 k xik1s.t

kyt � �txtk ✏

OF constraints between xt and xt�1