Introduction to Compressed Sensing and its applications [email protected]

Introduction to Compressed Sensing and its applications

[email protected]

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

2

Age of Digital World

• Our life revolve around the digital world– Entertainment, communication, business, life !!

• Digital bits streams running at the background is expected to deliver “natural” performance.– Surround sound, 3D TV, sixth sense !!


3

Human centric conversion process


4

Nyquist Theorem

• Band limited signal with highest frequency of B Hz can be reconstructed perfectly from its samples with rate > 2B. (Nyquist Rate).

• Relation in X(t) and X(nT).– Digital operation replacing analog counter parts.– Relationship in power spectral densities of analog

and discrete random process. • Estimation and detection by DSP is possible.

21-May-14


5

Spectrums of time domain signal and its samples


6

Wide band signal acquisition


7

ADC

• Analog to digital converters forms heart.• Physical (analog) information streams of

numbers digital processing by software.• Intriguing task Snap shot of fast varying

signal + acquiring measurements.• Unprecedented strain on ADC’s and DSP.

– Demand is ever increasing.

21-May-14


8

Ever increasing demand

• After sampling, we retain large number of bits.

• Conventional solution to storage space – Sampling Compression (exploiting redundancy)


9

Ultimate question[1]

• Why so much effort is spent on acquiring (sampling) the data(redundancy) when most of it will be thrown away (compression)?

• Can’t we directly measure the part (information) which will not be thrown away?

• Why can’nt we ask such a question??[1] D.L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol. 52, no.4, pp.

1289-1306, Sep. 2006.


10

Idea of simultaneous compression and sampling


11


12

Common link

• Basis Function• Coefficients

• Signal

• k - sparse signal 𝑥 ≅ Φ 𝜃• Support of non zero indices for denoted as 𝜃𝑆( ) 𝜃 .


13

Transform domain image representation


14

Basics of Compressed Sensing

• Demo


15

Whittaker–Shannon–Kotelnikov (WSK) v/s

Compressive sensing

WSK• Greater than Nyquist

rate.• Uniform / Non uniform

sampling• Non uniform sampling is

based on Lagrange interpolation.

• Theory developed for continuous time signals

CS• Sub Nyquist sampling.• Randomized

measurement matrix.• Samples are inner

product.

• Initial focus on finite dimensional signal.


16

Whittaker–Shannon–Kotelnikov (WSK) v/s

Compressive sensingWSK

• No underlying signal structure.

• Transform coding based compression.

• Does not consider Hardware implementation

CS• Initial sparse. Now

structure beyond sparsity is explored.

• Signal structure over and above sparsity gives higher compression.

• Structured non random measurement matrix.


17

Basic CS frame work

• 𝑥 is a N x 1 vector, • 𝑦 is a M x 1 vector with M << N

• is a random measurement matrix.• Sparsifying dictionary of basis

.


18

Recovery

• Recover• Given: and• : Class with all k sparse signal.• CS makes exhaustive search in such that

• Solution using optimization problem


19

Issues with

• • optimization is computationally very

expensive: it is an non-deterministic polynomial-time (NP) hard problem.

• E.g. M = 1000, N = 5000, k = 100• search space.• Non convex


20

Questions to be tackled

1. Can this problem be solved by any other mechanisms?

2. How can we get an estimated solution and what level of estimation accuracy is acceptable? (after all Engineers always looks for the workable approximate solutions !!!);

3. What kind of approximation will yield the solution closer to the desired one?


21

Convex and Non convex set


22

Convex relaxation - norm

• • • Set is convex, so above

problem is also convex.• A strictly posed convex problems leads to a

close form solution and guaranteed to converge at the local minima.

.


23

norm

• Find Xi’s for which norm is minimal.• It is not strictly convex and it may have multiple

solution.• However, these solutions are 1. clustered around in a convex set as all optimal

solutions will have an penalty and their combination would also be convex;

2. the set is bounded; 3. among them at least one has at most k non zero

elements.


24

Best possible approximation

•

• approximation ℓ𝑝 norm with <1 .𝑝


25

Heart: The measurement matrix

• ; : with M << N• Basis Fixed independent of signal.• Most fundamental design questions:1. how much of information about signal x is

retained in its linear measurements y ?; 2. how can the linear measurements y uniquely

represent x?; 3. how can the original signal be recovered from

its measurement?


26

The measurement matrix

• It is rank deficient with non empty Null space

• Consequence: • Unable to recover the signal from

measurements.• Design: For distinct k sparse signalsShould have a unique measurement


27

Characterization of Uniqueness

• Spark = Sparse + Rank• If spark( ) > 2k uniquely

represents a k- sparse signal belonging to class

• The spark of the measurement matrix is used

to ensure stability and consistency.• Computing is search over all sub-matrices.


28

Coherence

• Coherence can be used to identify the sparse signal.

• It is describing the dependency between two columns.

• Supremum on sparsity k is


29

Uniqueness property in presence of noise

• Modified • In CS, it is assumed that is available during

sparse signal recovery.• Modified • Measurement process should be robust to

such noise.• Sparse recovery is possible if


30

Restricted Isometric property (RIP)

• A matrix satisfies (k, ) restricted iso-metry property of order k if for ,

• Isometry is a function between the two spaces which has a property to preserve distance between each pair of points.


31

Building Sensing Matrices

• Vandermonde matrix has spark M + 1 • : Geometric progression.• Poor conditioning. • Gabor Frame = n x n time shift matrix

• Bernoulli, Gaussian, or sub Gaussian

.

.


32

Research avenue

• Structured measurement matrix: Application dependent.

• Subjected to the physical constraint of the application.

M.F. Duarte & Y.C. Eldar, "Structured compressed sensing: from theory to applications," IEEE Trans. Sig. Proc., vol.59, no. 9, pp. 4053-4085, Sept. 2011.


33

CS recovery

• CS recovery = L1 minimization• The choice of algorithm is based on various factors,

namely: 1. signal reconstruction timing from measurement

vector; 2. the number of measurement required for recovery

to determine the storage requirements;3. the simplicity of the implementation; 4. possible portability to the hardware for execution;5. fidelity of the signal recovery.


34

BP: Basis Pursuit, BPIC: Basis Pursuit with Inequality Constraint, BPDN: Basis Pursuit De-noising, MP: Matching Pursuit, OMP: Orthogonal Matching Pursuit, IHT: Iterative Hard Thresholding

21-May-14


35

Basic

•

• Its power stems from the fact that it converts the search into convex problem and provides the accurate recovery.

• Basis pursuit (BP): norm has a tendency to locate the sparse solutions if ever they exist.

21-May-14


36

2D reconstruction

(a) (b)• a) original Image N = 6400 (80 x 80); b) reconstructed

Image with M = 2400, MSE: 0.0 reconstruction time: 18.64 sec.


37

BPIC

Bound based on noise. Could be user defined


38

N = 1024; k = 50; M = 220; MSE=2.02 x 10-4


39

(a) (b)• (a) reconstructed Image M = 1600, MSE: 0.25,

reconstruction time: 16.27 sec; • (b) reconstructed Image M = 800, MSE: 0.31,

reconstruction time: 31.19 sec


40

Watermarking application


41

Detector


42

Over determined system

• with M > N• Construct matrix s.t = 0.

• Estimate using CS formulation.• s.t


43

Fidelity Non malicious

Malicious manipulations


44

Single Pixel Camera

• webee.technion.ac.il


45

Summary

• One of the most exciting domain.• Interdisciplinary: signal processing, statistics,

probability theory, computer science, optimization, linear programming.

• Look beyond the random measurement matrix.

• Developing a better signal models: finite rate innovation (FRI), Xampling framework


46

References• D.L. Donoho, "Compressed sensing", IEEE Trans. Inf. Theory, vol. 52, no.4, pp. 1289 - 1306, Sep. 2006. • E.J. Candès & T.Tao, "Near optimal signal recovery from random projections: Universal encoding

strategies," IEEE Trans. Inf. Theory, vol.52, no. 12, pp.5406 - 5425, Dec. 2006.• E.J. Candès, J. Romberg, & T. Tao, "Robust uncertainty principles: exact signal reconstruction from

highly incomplete frequency information," IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489 - 509, Dec. 2006.

• Richard Baraniuk, "Compressive sensing," IEEE Sig. Proc. Mag., vol. 24, no. 4, pp. 118 -124, 2007.• M.F. Duarte & Y.C. Eldar, "Structured compressed sensing: from theory to applications," IEEE Trans. Sig.

Proc., vol.59, no. 9, pp. 4053-4085, Sept. 2011.• Compressed sensing, Theory and applications, (eds. Y.C. Eldar & Gitta Kutyniok), Cambridge university

press, Cambridge, UK, 2012.• E. J. Candès, J. Romberg and T. Tao, “Stable signal recovery from incomplete and inaccurate

measurements”, Comm. Pure Appl. Math., vol.59,pp. 1207–1223, 2006.• B. K. Natarajan, “Sparse approximate solutions to linear systems”, SIAM Journal on computing, vol. 24,

pp.227–234, 1995.• Nonlinear Optimization: Complexity Issues, S. A. Vavasis, Oxford University Press, New York, 1991.• E. J. Candès and T. Tao, "The power of convex relaxation: Near-optimal matrix completion," IEEE Trans.

Inform. Theory, vol. 56, no. 5, pp. 2053-2080, 2009.• Convex Optimization, Stephen Boyd & Lieven Vandenberghe, Cambridge University Press, 2004.• S. S. Chen, D. L. Donoho, & M. A. Saunders, " Atomic decomposition by basis pursuit," SIAM J. Scientific

Computing, vol. 20, no. 1, pp.33–61, 1998.


47

Compressive sensing solvers

• http://users.ece.gatech.edu/~justin/l1magic/ • http://sparselab.stanford.edu/• http://www.lx.it.pt/~mtf/GPSR/• http://www.stanford.edu/~boyd/l1_ls/• http://www.personal.soton.ac.uk/tb1m08/sparsify/

sparsify.html• http://www.lx.it.pt/~mtf/SpaRSA/• https://sites.google.com/site/igorcarron2/

cs#reconstruction (Comprehensive Listing of solvers)• http://videolectures.net/bmvc09_chellappa_cscv/

http://videolectures.net/bmvc09_chellappa_cscv/


48

[email protected]

Thank you

Documents

Introduction to Compressed Sensing and its applications [email protected]