Two Applications of the Gaussian Poincaré Inequality in ... · Two Applications of the Gaussian...

Two Applications of the Gaussian PoincaréInequality in the Shannon Theory

Vincent Y. F. Tan (Joint work with Silas L. Fong)

National University of Singapore (NUS)

2016 International Zurich Seminar on Communications

Vincent Tan (NUS) Applications of the Poincaré Inequality IZS 2016 1 / 17

Gaussian Poincaré Inequality

Theorem

For Zn iid∼ N (0, 1) and any differentiable mapping f such that

E[(f (Zn))2] <∞, and E[‖∇f (Zn)‖2] <∞

we havevar[f (Zn)] ≤ E[‖∇f (Zn)‖2].

Controlling the variance of f (Zn) that is a function of i.i.d. randomvariables in terms of the gradient of f (Zn)

Using the Gaussian Poincaré inequality for appropriate f ,

var[f (Zn)] = O(n).

Theorem

E[(f (Zn))2] <∞, and E[‖∇f (Zn)‖2] <∞

var[f (Zn)] = O(n).

Theorem

E[(f (Zn))2] <∞, and E[‖∇f (Zn)‖2] <∞

var[f (Zn)] = O(n).

Gaussian Poincaré Inequality in Shannon Theory

Polyanskiy and Verdú (2014) bounded the KL divergence betweenthe empirical output distribution of AWGN channel codes PYn andthe n-fold product of the CAOD P∗Y , i.e.,

D(PYn‖(P∗Y)n)

Often we need to bound the variance of certain log-likelihoodratios (dispersion)

Demonstrate its utility by establishing

Strong converse for the Gaussian broadcast channels

Properties of the empirical output distribution of delay-limited codesfor quasi-static fading channels

D(PYn‖(P∗Y)n)

Gaussian Broadcast Channel

Assume g1 = g2 = 1 and σ22 > σ2

Input Xn must satisfy

‖Xn‖22 =

n∑i=1

X2i ≤ nP

‖Xn‖22 =

n∑i=1

X2i ≤ nP

‖Xn‖22 =

n∑i=1

X2i ≤ nP

Capacity Region

An (n,M1n,M2n, εn)-code consists ofan encoder f : {1, . . . ,M1n} × {1, . . . ,M2n} → Rn such that thepower constraint is satisfied;

two decoders ϕj : Rn → {1, . . . ,Mjn} for j = 1, 2;

such that the average error probability

P(n)e := Pr(W1 6= W1 or W2 6= W2) ≤ εn.

(R1,R2) is achievable⇔ ∃ a sequence of (n,M1n,M2n, εn)-codess.t.

lim infn→∞

log Mjn ≥ Rj, j = 1, 2, and

limn→∞

εn = 0.

Capacity region C is the set of all achievable rate pairs

Capacity Region

P(n)e := Pr(W1 6= W1 or W2 6= W2) ≤ εn.

lim infn→∞

limn→∞

εn = 0.

Capacity Region

P(n)e := Pr(W1 6= W1 or W2 6= W2) ≤ εn.

lim infn→∞

limn→∞

εn = 0.

Capacity Region

Cover (1972) and Bergmans (1974) showed that

C = RBC =⋃

α∈[0,1]

R(α) ={(R1,R2) : R1 ≤ C

(αPσ2

),R2 ≤ C

((1− α)PαP + σ2

C(x) =12

log(1 + x).

Direct part: Random coding + Superposition coding

Converse part: Fano’s inequality + Entropy power inequality

Capacity Region

C = RBC =⋃

α∈[0,1]

R(α) ={(R1,R2) : R1 ≤ C

(αPσ2

),R2 ≤ C

((1− α)PαP + σ2

C(x) =12

log(1 + x).

Capacity Region

C = RBC =⋃

α∈[0,1]

R(α) ={(R1,R2) : R1 ≤ C

(αPσ2

),R2 ≤ C

((1− α)PαP + σ2

C(x) =12

log(1 + x).

Capacity Region

P(n)e → 0

P(n)e 6→ 0 P(n)

e → 1?

Capacity Region

P(n)e → 0

P(n)e 6→ 0 P(n)

e → 1?

Capacity Region

P(n)e → 0

P(n)e 6→ 0

P(n)e → 1?

Capacity Region

P(n)e → 0

P(n)e 6→ 0 P(n)

e → 1?

Strong converse vs weak converse

Can we claim that if (R1,R2) /∈ C, then

P(n)e → 1?

Indeed!

Sharp phase transition between what’s possible and what’s not

The strong converse has been established for only degradeddiscrete memoryless BC

Ahlswede, Gács and Körner (1976) used the blowing-up lemma

BUL doesn’t work for continuous alphabets [but see Wu and Özgür(2015)]

Oohama (2015) uses properties of the Rényi divergence

Good bounds between the Rényi divergence Dα(P‖Q) and therelative entropy D(P‖Q) exist for finite alphabets

P(n)e → 1? Indeed!

ε-Capacity Region

(R1,R2) is ε-achievable⇔ ∃ a sequence of (n,M1n,M2n, εn)-codess.t.

lim infn→∞

lim supn→∞

εn ≤ ε.

Capacity region Cε is the set of all achievable rate pairs

Strong converse holds iff Cε does not depend on ε.

We already know that

RBC = C0 ⊂ Cε

Strong converse

Theorem

The Gaussian BC satisfies the strong converse property:

Cε = RBC, ∀ ε ∈ [0, 1)

Key ideas in proof:

Derive an appropriate information spectrum converse bound

Use the Gaussian Poincaré inequality to bound relevant variances

Weak Converse for GBC [Bergmans (1974)]

Step 1: Invoke Fano’s inequality to assert that for any sequence ofcodes with vanishing error probability εn → 0,

Rj ≤1n

I(Wj;Ynj ) + o(1), ∀ j ∈ {1, 2}.

Step 2: Single-letterize and entropy power inequality

I(W1;Yn1 ) ≤ nI(X;Y1|U)

EPI≤ nC

(αPσ2

)I(W1;Yn

1 ) + I(W2;Yn2 ) ≤ nI(U;Y2)

EPI≤ nC

((1− α)PαP + σ2

Weak Converse for GBC [Bergmans (1974)]

Step 1: Invoke Fano’s inequality to assert that for any sequence ofcodes with vanishing error probability εn → 0,

Rj ≤1n

I(Wj;Ynj ) + o(1), ∀ j ∈ {1, 2}.

Step 2: Single-letterize and entropy power inequality

I(W1;Yn1 ) ≤ nI(X;Y1|U)

EPI≤ nC

(αPσ2

)I(W1;Yn

1 ) + I(W2;Yn2 ) ≤ nI(U;Y2)

EPI≤ nC

((1− α)PαP + σ2

Strong Converse for DM-BC [Ahlswede et al. (1976)]

Step 1: Invoke the blowing-up lemma to assert that for anysequence of codes with non-vanishing error probability ε ∈ [0, 1),

Rj ≤1n

I(Wj;Ynj ) + o(1), ∀ j ∈ {1, 2}.

Step 2: Single-letterize

I(W1;Yn1 ) ≤ nI(X;Y1|U),

I(W1;Yn1 ) + I(W2;Yn

2 ) ≤ nI(U;Y2)

where Ui := (W2,Y i−11 ). One also uses the degradedness

condition here:

I(W2,Y i−12 ,Y i−1

1 ;Y2i) = I(Ui;Y2i).

Strong Converse for DM-BC [Ahlswede et al. (1976)]

Step 1: Invoke the blowing-up lemma to assert that for anysequence of codes with non-vanishing error probability ε ∈ [0, 1),

Rj ≤1n

I(Wj;Ynj ) + o(1), ∀ j ∈ {1, 2}.

Step 2: Single-letterize

I(W1;Yn1 ) ≤ nI(X;Y1|U),

I(W1;Yn1 ) + I(W2;Yn

2 ) ≤ nI(U;Y2)

where Ui := (W2,Y i−11 ). One also uses the degradedness

condition here:

I(W2,Y i−12 ,Y i−1

1 ;Y2i) = I(Ui;Y2i).

Our Strong Converse Proof for Gaussian BC

Convert code defined based on avg error prob ≤ ε to one basedon max error prob ≤

√ε =: ε′ w/o loss in rate [Telatar]

Establish information spectrum bound. For every (w1,w2), everycode with max error prob ≤ ε′ satisfies

ε′ ≥ Pr(

logP(Yn

1 |w1)

P(Yn1 )≤ nR1 − γ1(w1,w2)

)− n2e−γ1(w1,w2)

{2n(R1+R2)

∫D1(w1)

P(yn1)P(w2|w1, yn

1) dyn1 > n2

}Indicator term is often negligible (by Markov’s inequality)

Establish a bound on the coding rate

nR1 ≤ E[

logP(Yn

1 |w1)

P(Yn1 )

1− ε′var[

logP(Yn

1 |w1)

P(Yn1 )

]+ 3 log n

ε′ ≥ Pr(

logP(Yn

1 |w1)

P(Yn1 )≤ nR1 − γ1(w1,w2)

)− n2e−γ1(w1,w2)

{2n(R1+R2)

∫D1(w1)

P(yn1)P(w2|w1, yn

1) dyn1 > n2

Indicator term is often negligible (by Markov’s inequality)

nR1 ≤ E[

logP(Yn

1 |w1)

P(Yn1 )

1− ε′var[

logP(Yn

1 |w1)

P(Yn1 )

]+ 3 log n

ε′ ≥ Pr(

logP(Yn

1 |w1)

P(Yn1 )≤ nR1 − γ1(w1,w2)

)− n2e−γ1(w1,w2)

{2n(R1+R2)

∫D1(w1)

P(yn1)P(w2|w1, yn

1) dyn1 > n2

nR1 ≤ E[

logP(Yn

1 |w1)

P(Yn1 )

1− ε′var[

logP(Yn

1 |w1)

P(Yn1 )

]+ 3 log n

ε′ ≥ Pr(

logP(Yn

1 |w1)

P(Yn1 )≤ nR1 − γ1(w1,w2)

)− n2e−γ1(w1,w2)

{2n(R1+R2)

∫D1(w1)

P(yn1)P(w2|w1, yn

1) dyn1 > n2

nR1 ≤ E[

logP(Yn

1 |w1)

P(Yn1 )

1− ε′var[

logP(Yn

1 |w1)

P(Yn1 )

]+ 3 log n

Use the Gaussian Poincaré inequality with careful identification off and peak power constraint ‖Xn‖2 ≤ nP to assert that

logP(Yn

1 |w1)

P(Yn1 )

]= O(n).

Thus we conclude that

nR1 ≤ I(W1;Yn1 ) + O(

√n), ∀ ε ∈ [0, 1).

Backoff term is of the order O(1/√

n), i.e.,

λ log M∗1n + (1− λ) log M∗2n = nCλ + O(√

Cλ := maxα∈[0,1]

{λC(αPσ2

)+ (1− λ)C

((1− α)PαP + σ2

)}but nailing down the constant (dispersion) seems challenging.

logP(Yn

1 |w1)

P(Yn1 )

]= O(n).

nR1 ≤ I(W1;Yn1 ) + O(

√n), ∀ ε ∈ [0, 1).

n), i.e.,

Cλ := maxα∈[0,1]

{λC(αPσ2

)+ (1− λ)C

((1− α)PαP + σ2

)}but nailing down the constant (dispersion) seems challenging.

logP(Yn

1 |w1)

P(Yn1 )

]= O(n).

nR1 ≤ I(W1;Yn1 ) + O(

√n), ∀ ε ∈ [0, 1).

n), i.e.,

Cλ := maxα∈[0,1]

{λC(αPσ2

)+ (1− λ)C

((1− α)PαP + σ2

)}but nailing down the constant (dispersion) seems challenging.Vincent Tan (NUS) Applications of the Poincaré Inequality IZS 2016 14 / 17

Another Application: Quasi-Static Fading Channels

Consider the channel model

Yi =√

HXi + Zi, i = 1, . . . , n

where Zi are independent standard normal random variables and

E[1/H] ∈ (0,∞)

If fading state is h and message is w, codeword is fh(w) ∈ Rn.

Long-term power constraint

Mn∑w=1

PH(h)‖fh(w)‖2 dh ≤ nP

Delay-limited capacity [Hanly and Tse (1998)], i.e., the maximumtransmission rate under the constraint that the maximal errorprobability over all H > 0 vanishes

Yi =√

HXi + Zi, i = 1, . . . , n

E[1/H] ∈ (0,∞)

Mn∑w=1

Yi =√

HXi + Zi, i = 1, . . . , n

E[1/H] ∈ (0,∞)

Mn∑w=1

Yi =√

HXi + Zi, i = 1, . . . , n

E[1/H] ∈ (0,∞)

Mn∑w=1

Vanishing Normalized Relative Entropy

The delay-limited capacity [Hanly and Tse (1998)] is

C(PDL), where PDL :=P

E[1/H]

For any sequence of capacity-achieving codes with vanishingmaximum error probability

limn→∞

D(PYn‖(P∗Y)n)→ 0 where P∗Y = N (0,PDL).

Every good code is s.t. the induced output distribution “looks like”the n-fold CAOD.

Extend to the case where the error probability is non-vanishing

Control a variance term

logPYn|Xn,H(Yn|Xn, h)

PYn|H(Yn|h)

]= O(n).

E[1/H]

limn→∞

PYn|H(Yn|h)

]= O(n).

E[1/H]

limn→∞

PYn|H(Yn|h)

]= O(n).

E[1/H]

limn→∞

PYn|H(Yn|h)

]= O(n).

Concluding Remarks

Gaussian Poincaré inequality is useful for Shannon-theoreticproblems with uncountable alphabets

var[f (Zn)] ≤ E[‖∇f (Zn)‖2].

Allows us to establish strong converses and properties of goodcodes by controlling variance of log-likelihood ratios

For more details, please see

Arxiv: 1509.01380 (Strong converse for Gaussian broadcast)

Arxiv: 1510.08544 (Empirical output distribution of good codes forfading channels)

Concluding Remarks

var[f (Zn)] ≤ E[‖∇f (Zn)‖2].

Concluding Remarks

var[f (Zn)] ≤ E[‖∇f (Zn)‖2].

Two Applications of the Gaussian Poincaré Inequality in ... · Two Applications of the Gaussian...

Documents

From Newton to Poincaré - ELTE

Poincaré Intuition Mathématiques

CERCLE Poincaré - École Polytechnique

Poincaré, el último matemático universalista

Poincaré - O Valor Da Ciência

COMPARISON AND ANTI-CONCENTRATION BOUNDS FOR …vchern/papers/AntiConcentrationVectors-v10.pdfconcentration inequality for the maximum of a Gaussian random vector, which derives a

Dynamics of a planar Coulomb gas - icerm.brown.edu...Dynamics of a planar Coulomb gas Poincaré inequality Fokker–Planck evolution equation and generator Let pt: Rd ÑR be the density

Termodinamica di Henri Poincaré

· 4 In memory of A. Ehrhard The Gaussian isoperimetric inequality, and its related concentration phenomenon, is one of the most important properties of Gaussian measures. These

TopWis Poincaré : de e-klas

Sobolevräume und Poincaré-Ungleichungdiening/ws14/seminar/haslauer.pdf · LMUMünchen,Germany EliasHaslauer Sobolevräume und Poincaré-Ungleichung SeminarNumerischeAnalysisbei

Poincaré, Filósofo das Ciências: Problemas e Perspectivascfcul.fc.ul.pt/coloquios/coloquio_poincare/ABSTRACTS.pdf · Poincaré, Filósofo das Ciências: Problemas e Perspectivas

Henri Poincaré

UNIVERSITÉ HENRI POINCARÉ, NANCY 1

HENRI POINCARÉ - FlacsoAndes

A Poincaré inequality on loop spaces - Xue-Mei · A Poincaré inequality on loop spaces ... relates to the geometry and the topology of the path space and its subspaces. For the

Poincaré Plots: A MiniReview - PhysioNet · Poincaré plot Poincaré HRV plot is a graph in which each RR interval is plotted against next RR interval (a type of delay map)

Jules henri poincaré

Henri poincaré metodologia exponer

Streamline Poincaré-Friedrichs inequality and ... · Streamline Poincar e-Friedrichs inequality and preconditioning estimates J anos Kar atson karatson@cs.elte.hu Department of Applied