Décompositions parcimonieuses: l’approche gloutonne

Cédric Herzet Inria Rennes - Télécom Bretagne

1

Sparse data abound in Nature

Image wavelet decomposition Voice spectrogram

Exploiting sparse prior may allow to reconstruct unobserved dataCourtesy of G. Peyré

data + sparse prior

The sparse inversion problem

y = x

measurements unknown sparse vector

Can we recover x from y by exploiting the sparsity of x?

A 2 Rm⇥n

The target problem: find the best approximation in a union of subspaces

5

yV1

V2

V3

minV 2⌃

minz2V

ky � zk22

Solve:

Let ⌃ = [iVi ⇢ Rm

The inner minimization has a tractable solution

6

Solve:

yV1

minz2V1

ky � zk22

⌘ least square problem

Complexity: O(m dim(V1)3)

The overall complexity scales linearly with the number of subspaces

7

minV 2⌃

minz2V

ky � zk22

Solve:

Let ⌃ = [iVi ⇢ Rm

yV1

V2

V3

Complexity:

card(⌃)⇥O(m dim(V1)3)

The «sparse» paradigm involves a large number of subspaces

8

Solve: minV 2⌃k

minz2V

ky � zk22

where ⌃k = [S:card(S)kspan (AS), A 2 Rm⇥n

card(⌃k) =

✓nk

◆!!!

The sparse approximation problem is NP-Hard... see e.g, [Natarajan 95], [Foucart 10]

Different approaches to deal with the NP-hardness of the sparse problem

9

• Integer programming: [Bourguignon 16], [Miller 02, Chapter 3]

• Greedy approaches (today’s talk)

• (Convex) relaxation: [Gorodnitsky 97], [Chen 99], [Wipf 04], [Cemgil 07]

Greedy procedures construct sequentially the approximation subspace

10

V (1) ! V (2) ! . . . V (l)

z(l) 2 V (l).

Greedy procedure generates:

In the sparse context: S(1) ! S(2) ! . . . S(l)

z(l) 2 span�AS(l)

�

MP, OMP and OLS

11

MP, OMP, OLS are forward greedy algorithms

12

The algorithms generate

S(1) ! S(2) ! . . . S(l)

with

S(l+1) = S(l) [ {j}

1 atom of the dictionary is added at each iteration

The approximation problem is easy to solve for unions of 1D subspaces

13

Consider: minV 2⌃1

minz2V

ky � zk22 where ⌃1 = [1inspan (ai)

We have: minV 2⌃1

minz2V

ky � zk22 ⌘ mini2[1,n]

minz2span(ai)

ky � zk22| {z }

achieved for z = hai,yiai

= mini2[1,n]

n

kyk22 � hai,yi2o

The best approximation subspace maximizes |hai,yi|

MP strategy: first iteration

14

Evaluate: j = argmax

i2[1,n]|hai,yi|

Set: S(1) = {j}z(1) = haj ,yiaj

MP strategy for the subsequent iterations: one-dimensional update of the current estimate

15

Let

We compute a 1D correction to as

r(l�1) , y � z(l�1) (current approximation error)

z(l�1) minV 2⌃1

minz2V

���r(l�1) � z���2

2

minV 2⌃1

minz2V

���r(l�1) � z���2

2= min

i2[1,n]

⇢���r(l�1)���2

2�

Dai, r

(l�1)E2

�As previously, we find:

MP strategy: iterations > 1

16

Evaluate:

Set:

j = argmax

i2[1,n]

���Dai, r

(l�1)E���

z(l) ! y in both finite and infinite dimensional spaces[Jones 87], [Mallat 93]

Convergence can be slow: MP may select the same atom many times

S(l) = S(l�1) [ {j}

z(l) = z(l�1) +Daj , r

(l�1)Eaj

Suboptimality of MP’s approximation update

17

Evaluate:

Set:

j = argmax

i2[1,n]

���Dai, r

(l�1)E���

2 span(AS(l))

... but we usually have

z(l) 6= argminz2span(AS(l))

ky � zk22

S(l) = S(l�1) [ {j}

z(l) = z(l�1) +Daj , r

(l�1)Eaj

OMP strategy

18

Evaluate:

Set:

j = argmax

i2[1,n]

���Dai, r

(l�1)E���

z(l) ! y in both finite and infinite dimensional spacesOMP never selects the same atom twice!

S(l) = S(l�1) [ {j}z(l) = argmin

z2span(AS(l))ky � zk22

OLS strategy

19

Evaluate:

Set:

z(l) ! y in both finite and infinite dimensional spacesOLS never selects the same atom twice!

S(l) = S(l�1) [ {j}z(l) = argmin

z2span(AS(l))ky � zk22

j = argmini2[1,n]

minz2span(AS(l�1)[i)

ky � zk22

Complexity

20

Let

Complexity MP :

Complexity OMP :

Complexity OLS :

O(mn)

O�mn+ k3 + km

�

O�(k3 + km)n

�

A 2 Rm⇥n

Complexity MP Complexity OMP Complexity OLS Thus:

A first insight into the performance

21

Let be the residual at iteration l.r(l�1)

Then,

���r(l�1)���2�

���r(l)MP

���2�

���r(l)OMP

���2�

���r(l)OLS

���2

MP, OMP and OLS in the literature

22

MP :

OMP : [Pati 93], [Zhang 93], [Davis 94],

[Mallat 93], «projection pursuit» [Friedman 81],[Huber 85],

OLS : [Chen 89],

«forward selection» [Miller 02],

«greedy algorithm» [Natarajan 95],

«order recursive matching pursuit» [Cotter 99],

«optimized orthogonal matching pursuit» [Reibollo-Neira 02],

«pure orthogonal matching pursuit» [Foucart 11]

«pure greedy» [Temlyakov 08]

«orthogonal greedy algorithm» [Temlyakov 08],

Other types of greedy algorithms

23

Multiple selections :

Approximate update :

Backward recursion :

«gradient pursuit» [Blumensath 08]

«stagewise OMP» [Donoho 12]

«backward greedy» [Couvreur 00]

Forward-backward recursion : [Efroymson 60], [Broersen 86],

«SMLR» [Kormylo 99], [Haugland 07]

«Bayesian Pursuit» [Herzet 10, 12],

«FoBa» [Zhang 11], SBR [Soussen 11], ...

Randomized selection : [Elad 09], [Divekar 10]

«Weak» greedy: [Temlyakov 00], [Gribonval 01]

Under which conditions can OMP solve the target approximation problem?

24

Is the solution of the greedy algorithm close to the optimal one?

25

9?C such that���y � z(k)

��� C minz2⌃k

ky � zk

y V1

V2

V3

Cdist(y,⌃)

z(k)

see e.g., [Temlyakov 08]

A more simple question is: can one identify the support of y?

26

y

V1

V2

V3

?= z(k)

if y 2 span (AS),

z(k) = y

S(k) = S ?

A tight condition of success in k steps: the «exact recovery condition»

27

S(k) = S for any y 2 span (AS), AS full rank, iff

ERC(S) < 1,

where

ERC(S) , max

i/2S

��A+S ai

��1.

Valid for OMP [Tropp 05], MP [Gribonval 06], OLS [Soussen 13]

Let card(S) = k.

Checking the ERC for all supports of size k is a combinatorial task...

28

iff max

S:card(S)=kERC(S) < 1,

S(k) = S for

⇢any y 2 span (AS), AS full rank,

any S, card(S) = k,

Conditions based on the mutual coherence are simple to evaluate

29

OMP succeeds in k steps for all S, card(S) = k if

µ <1

2k � 1

where µ = max

i 6=j|hai,aji|.

• Valid for MP, OLS [Tropp 05], l1 and l0 minimizations [Gribonval 03]

• The condition is sharp [Cai 11]

Definition: the restricted isometry constant

The restricted isometry constant of order s is the smallest

constant �s such that

(1� �s)kxk2 kAxk2 (1 + �s)kxk2

holds for all s-sparse vectors x.

For Gaussian random dictionaries A 2 Rm⇥n,

�s < ↵ with high probability if

see e.g., [Foucart 10, Theorem 9.2]

m � C↵�2s log⇣ns

⌘,

Conditions based on RICs for the success of OMP in k steps

OMP succeeds in k steps for all S, card(S) = k if

The condition �k+1 < 1pk+1

is sharp [Wen 13].

�k+1 < 13pk

[Davenport 10]�k+1 < 1

2pk+1

[Huang 11]�k+1 < 1

2pk

[Liu 12]�k+1 < 1p

k+1[Maleh 11], [Mo 12], [Wang 12]

�k+1 <p4k+1�12k [Chang 14]

�k+1 < 1pk+1

[Mo 15]

Goal of compressive sensing: recover x with as few measurements as possible

y = xAmeasurements unknown sparse vector

How many (random) measurements do we need to recover k-sparse vector x from y?

The coherence and RIC conditions require m~k2 to be satisfied

) m > C 0k2

(µ < 1

2k�1

µ �q

n�mm(n�1) [Foucart 13, Th 5.7]

m⌧n) m > (2k � 1)2

⇢�k+1 < 1p

k+1

�s < ↵ if m � C↵�2s log�ns

�

The recovery performance seems to evolve linearly with sparsity

m=(2k-1)2

simulated pointsnum

ber

of m

easu

rem

ents

number of nonzero elements

n=512 A ~ random algo = OMP

linear behavior!

Open question

[Rauhut 08]:

[Donoho 06]:

• Uniform recovery can not be achieved with m = O(k)

Is m > Ck2 a necessary condition for the recovery of

any support S, card(S) = k, in k steps?

• Uniform recovery in k steps is not possible with m = O(k3/2)

• Conjecture: Uniform recovery in k steps requires m = O(k2)

Towards the average case analysis

• What is proportion of supports S for which ERC(S) � 1?• If ERC(S) � 1, what are the y for which OMP fails?

2 main questions:

Recovery guarantees for decaying signals

Hypothesis: |x1| � |x2| � . . . � |xk|

[Davenport 10]: If �k+1 < 1

3 and

|xi||xi+1| � f(�

k+1),

then OMP recovers any support S in k steps.

See also [Soussen 13], [Ehler 14], [Herzet 16]

Success of OMP in more than k steps

38

OMP in more than k steps

Strategy:

Success: If y = ASxS and S ✓ S, we have xS = xS

xS\S = 0.

i) Run OMP until r(l) = 0

ii) Solve xS = argminxS

��y �ASxS

��iii) Set Sfinal = supp

�xS

�

Open question

Is there a polynomial-time tight condition ensuring the success of OMP in more than k iterations?

Conditions based on RIC

See also [Livshitz 14], [Wang 16] for more refined analyses

[Zhang 11] : S ⇢ S(30k)if �31k < 1/3

[Foucart 11] : S ⇢ S(12k)if �20k 1/6

[Foucart 13] : S ⇢ S(24k)if �26k < 1/6

Thank you for attention!

42

Recovery of a given support from random measurements

S(k) = S for any y 2 span (AS) and AS full rank,

with probability exceeding

1� 2 exp

n

� m

2Ck

o

, if

m � 2Ck log n

[Tropp 07], [Fletcher 09], [Lin 13]

Let A ⇠ N (0, 1pm) and card(S) = k.

Bibliography

44

Bibliography

45

Bibliography

46

Bibliography

47