New Results in Coding and Communication Theory

1

Communication Systems Laboratory

New Results in Coding and Communication Theory

Communication Systems Laboratoryhttp://www.ee.ucla.edu/~csl/

Rick Wesel

At the Aerospace CorporationAugust 11, 2008

BikeXie

Yuan-Mao Chang

Tom Courtade

Jiadong Wang

MiguelGriot


Recent Results

• A more efficient LDPC decoding algorithm

• Dynamically scheduled LDPC decoder that converge to a better frame error rate than previous state-of-the-art.

• A powerful family of low-rate turbo codes for “rate-less”applications at the physical layer.

• Turbo Code with closest-to-Shannon performance through novel nonlinear labeling.

• New information theoretic results and nonlinear turbo codes for multi-rate broadcast.

2


Recent Results







Simultaneous (Flooding) Schedule

• On every iteration– All variable nodes are

simultaneously updated– All check nodes are

simultaneously updated

3


Standard Sequential Schedule (SSS)a.k.a Layered Belief Propagation (LBP)

• Update messages sequentially:– [Mansour 03] (CN sequence)– [Kfir 03] (VN sequence)– [Hocevar 04] (LBP)– [Sharon 04] (Serial Schedule)– [Zhang 05] (Shuffled BP)– [Radosavljevic 05]


Results N=1944 Rate=1/2

0 5 10 15 20 25 30 35 40 45 5010

-4

10-3

10-2

10-1

100

Iterations

FE

R

Eb/No = 1.75 dB

Simultaneous (Flooding)LBP

Layered belief propagation isabout twice as fast a flooding.

4


Informed Dynamic Scheduling• Sequential scheduling is a good idea• What is the best sequence of updates?• Use the current information in the graph to

choose the next message to be sent• This is called Informed Dynamic Scheduling

(IDS)


Residual Belief Propagation (RBP) [Elidan 06]

• Define residual as:

• As BP converges, the residuals go to 0• RBP is a greedy algorithm that propagates

the message with the largest residual

new oldr m m= −

5


• Messages are Log-Likelihood Ratios (LLRs)

• Message-generation equations:

RBP for LDPC decoding

( )\j i a j j

j i

v c c v va N v c

m m C→ →∈

= +∑

( )\2 atanh tanh

2b i

i j

b i j

v cc v

v N c v

mm →

→∈

⎛ ⎞⎛ ⎞= × ⎜ ⎟⎜ ⎟⎜ ⎟⎝ ⎠⎝ ⎠

∏

( )( )

0log

1j

j

p v

p v

⎛ ⎞=⎜ ⎟⎜ ⎟=⎝ ⎠


• Initially propagate the channel information

• Propagate message with biggest residual

• The variable-to-check messages that change will have the same residual so they are the biggest

• Therefore, RBP can be simplified– Propagate check-to-variable

message with biggest residual– Propagate variable-to-check

messages that change

RBP for LDPC decoding

6



0 5 10 15 20 25 30 35 40 45 5010

-4

10-3

10-2

10-1

100

Iterations

FE

R

Eb/No = 1.75 dB

Simultaneous (Flooding)LBPRBP

Residual Belief Propagationstarts out fast but hits a nastyerror floor.


Dynamic Node-wise Scheduling (NS)

• Find the check-to-variable message with the biggest residual.

• Update the check-node.• Update the variable-to-

check messages that change.

7


A better answer in fewer iterations:Results N=1944 Rate=1/2

0 5 10 15 20 25 30 35 40 45 5010

-5

10-4

10-3

10-2

10-1

100

Iterations

FE

R

Eb/No = 1.75 dB

Simultaneous (Flooding)

LBPRBP

NS


We understand faster, but why better?

• Performance plots show that informed dynamic scheduling strategies perform not only faster but better than LBP.

• There are several noise realizations that aren’t corrected after 200 LBP iterations but are corrected after few ANS iterations.

• This difference can’t be explained with the argument of the “wasted” updates.

• Informed Dynamic Scheduling solves “trapping-set” errors that neither LBP nor flooding scheduling can solve.

8


• RBP greediness makes it converge faster but less often

• Node-wise Scheduling (NS) simultaneously propagates all the outgoing messages from a check node

• The check-node updated is the one that has the message with the biggest residual

• This means that correcting messages are simultaneously sent to all variable nodes that could be in error

• NS converges slower than RBP but more often

Node-wise Scheduling (NS)


IDS “Iterations”• There aren’t iterations in IDS strategies

• In order to compare the scheduling strategies we count an iteration after the number of check-to-variable messages propagated is the same as in a flooding or LBP iteration

• The message generation complexity of all the strategies is the same

• Residual computation makes the IDS strategies more complex than flooding and LBP

9


• Residual computation requires the values of the messages to be propagated.

• Many of those computations are then “wasted” since many of those messages aren’t propagated.

• Using the min-sum check-node update to compute residuals significantly reduces the complexity.

• Approximate RBP (ARBP) and Approximate NS (ANS) are the min-sum versions of RBP and NS respectively.

Complexity


– Check-to-variable message equation:

–

– Thus a check-node update involves:• Find the two variable-to-check messages with

the smallest reliabilities• Compute the signs of the check-to-variable

messages• Assign the correct reliability

Min-sum

( )( ) ( ) ( )

\\

sgn mini j b i b i

b i jb i j

c v v c v cv N c vv N c v

m m m→ → →∈∈

= ⋅∏

10



0 5 10 15 20 25 30 35 40 45 5010

-5

10-4

10-3

10-2

10-1

100

Iterations

FE

R

Eb/No = 1.75 dB

RBPARBPNSANS



1 1.5 2 2.510

-5

10-4

10-3

10-2

10-1

100

Eb/No

FE

R

FloodingLBPANS

11


Results 802.11n code rate 1/2

0 20 40 60 80 100 120 140 160 180 20010

-5

10-4

10-3

10-2

10-1

100

Iterations

FE

R

Eb/No = 1.75 dB

Simultaneous (Flooding)LBPARBPANS


Results 802.11n code rate 5/6

0 20 40 60 80 100 120 140 160 180 20010

-5

10-4

10-3

10-2

10-1

100

Iterations

FE

R

SNR = 6 dB

Simultaneous (Flooding)LBPARBPANS

12


Recent Results







Recent Results






13


Nonlinear Turbo Codes

TCM

Interleaver

TCM

0k

0k

k

• All existing turbo codes use linear convolutional component codes.

• Miguel Griot developed new tools that allow us to design and analyze nonlinear component trellis codes.


Traditional Linear Turbo Codes

NLTC

S

LUTbk

cnνΠ

1

2

State Machine

Linear Function

14


Nonlinear Turbo Codes

NLTC

S

LUTbk

cnνΠ

1

2

State Machine

ANY Function


Motivations• Applications where a non-uniform distribution

of ones and zeros are required for maximum transmission rate. (Linear codes provide equally likely ones and zeros).

• Higher-order modulations:

CC

Interleaver

CC

0k

0k

kMapper

Mapper

Trellis codedmodulation

15


Higher-order modulations• The serial concatenation of a convolutional

code with a mapping could be too restrictive.• We can use any function that assigns

constellation points to each of the branches of the trellis.

NLTC

S

LUTbk

cnνΠ

1

2

State Machine

f(u,v)u

A constellation

point.


Analytical Bit Error Rate bound• We provide a method to predict the BER of parallel

concatenated non-linear codes over asymmetric channels, in particular the Z-Channel, under Maximum Likelihood decoding.

• We extend the uniform interleaver analysis proposed in [Benedetto ‘96].

• Uniform interleaver: given the two constituent codes, average over all possible interleavers of a certain length K.

• Key difference: nonlinearity of the constituent codes. We cannot assume that the all-zero codeword is transmitted. We need to average over all possible codewords.

16


A new uniform interleaver• A new probabilistic device:

• Uniform interleaver for linear codes (Benedetto): “A uniform interleaver of length k is a probabilistic device that maps a given input word of Hamming weight w into all distinct permutations of it with equal probability”.

– This definition is useful only when assuming the all-zero codeword is transmitted.

• New definition for ANY code: “A uniform interleaver of length k is a probabilistic device the chooses between all possible positionpermutations with equal probability. Then, for each position, the value of the symbol can be changed to any other value with equal probability”.

– This new definition includes the linear case and leads to similar conclusions for both linear and nonlinear constituent codes.


Trellis structure and effective free distance

• The new uniform interleaver analysis allows to show that a recursive encoder is required for nonlinear codes as well.

• Benedetto showed that an important metric to maximize in the constituent linear codes is the effective free distance. We extend this notion for nonlinear codes:

• Effective free distance: minimum distance in the output produced by any two possible input words with Hamming distance 2.

• This definition includes the linear case.

17


Example: 2 bits/s/Hz 8PSK

• Constrained capacity: 2.8 dB.• We compare against the best previously published

16-state turbo code [Fragouli’01].

• Goal: maximize the effective free distance, where the distance in this case is the squared Euclidean distance.

416-St CC Mapper

3

416-St CC Mapper

3

Interleaver Nonlinear trellis code



• Achieving greater effective free distance than linear codes:– This is a fully connected trellis: given any two states S1 and

S2, there are 16 trellis paths from S1 to S2 after two trellis sections.

– Using a linear convolutional code and a mapper, all the searches we and previous works have done, have at least on pair of those paths that have a distance of

0

1

15

S1

S2

21 0.5858d =

22 2d =

212 1.171573d =

18



• Achieving greater effective free distance than linear codes:

– We found a nonlinear labeling that guarantees that for each pair of those paths, there is at least one trellis sections where the constellation points have squared Euclidean distance 2.

– There could still be a path in more than 2 trellis sections withdistance less than 2. By a search in the trellis structure we avoid that.

– The resulting nonlinear constituent code has an effective free distance equal to 2.

0

1

15

S1

S2

21 0.5858d =

22 2d =


0.2 dB gain for rate-2/3, 16-statesover the best published 10,000 bit code.

2.8 2.9 3 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.810

-7

10-6

10-5

10-4

10-3

10-2

10-1

100

BE

R

Eb/N0 [dB]

[Fragoul'01]

Bound [Fragouli'01]

PC-NLTCM

Bound PC-NLTCM

19


Observations• Interleaver design plays an important role in

the code’s performance as shown in [Fragouli’01]. The uniform interleaver analysis averages over all possible interleaver, which explains the difference in the error flour between simulations and the bound.

• However, the BER bounding analysis helps in the design process.


Recent Results






20


Broadcast Channels• One transmitter broadcasts information

messages to several receivers.

• Receivers decode messages without collaboration.

• Still an open problem. (outer bounds [Sato78], [Marton79], inner bounds [Van75], [Cover75], [Hajek79], [Marton79])

39

XY1

Y2

( )1 |p y x

( )2 |p y x


Degraded Broadcast Channel• Physically Degraded Broadcast Channels

– The worse channel is a further distortion of the better channel.

• Stochastically Degraded Broadcast Channels– The marginal distribution of the worse

channel could have been produced by a further distortion of the better channel.

40

( )1 1

2 1 2 1| ( | ) ( | )y

p y x p y x q y yy∈

= ∑

( )1 |p y x ( )2 1|q y yX Y1 Y2

21


• The capacity region is the convex hull of the closure of all rate pairs (R1, R2) satisfying

• for some joint distribution

41

1 1 2( ; | ),R I X Y X≤

2 2 2( ; ),R I X Y≤

2 1 2 2 2 1 2( , , , ) ( ) ( | ) ( , | ).p x x y y p x p x x p y y x=

( )1 |p y x ( )2 1|q y yX Y1 Y2( )2|p x xX2

Degraded Broadcast Capacity Region[Cover72][Bergmans73][Gallager74]


Successive Encoding and Decoding

• Successive, but joint, encoding

• User 2 is decoded treating user 1 as noise.• Successive decoding for user 1

42

1W

2W Joint Encoder

Encoder 2nX2

nX

1nY

Decoder 2

Decoder 1“Subtract”

2ˆ nX

1nY

2( )p x 2( | )p x x

( )1 |p y x ( )2 1|q y yX Y1 Y2( )2|p x xX2

22


Degraded Gaussian Broadcast Channel– Channel Model

– General Transmission Strategy

43

+

+ +

Successive Decoder

Decoder 2

1W

2W

nX

2nY

1nY

nNΔ1nN

1nN

1W

2W

Joint Encoder

X2Y

1Y

NΔ1N

1N

+

+ +

( )210,σ∼ N

( )20,σΔ∼ N( )210,σ∼ N


Degraded Gaussian Broadcast Channel– Channel Model

– Optimal Transmission Strategy [Bergmans74]

44

+

+ +

Successive Decoder

Decoder 2

1W

2W

nX

2nY

1nY

nNΔ1nN

1nN

X2Y

1Y

NΔ1N

1N

+

+ +

( )210,σ∼ N

( )20,σΔ∼ N( )210,σ∼ N

+1W

2W 2nX

1nX

23


Degraded Binary Symmetric Broadcast Channel

– Channel Model

– Optimal Transmission Strategy Optimal Transmission Strategy [Cover72][Bergmans73]

45

X2Y

1Y

NΔ1N

1N

XOR

XOR XOR

XOR

XOR XOR

Successive Decoder

Decoder 2

1W

2W

nX

2nY

1nY

nNΔ1nN

1nN

XOR

1W

2W 2nX

1nX


Broadcast Z Channels• Broadcast Z Channels

• Broadcast Z channels are stochastically degraded broadcast channels.

46

XY1

Y2

1

0

1

0

1

0

1α

2α 1 20 1α α< < <

X Y1

1α

Y2

αΔ2 1

11α αα

αΔ

−=

−

24


Degraded Z Broadcast Channel

– Channel Model

– Optimal Transmission Strategy

47

1 1Pr( 1)N α= =

Pr( 1)N αΔ Δ= =X2Y

1Y

NΔ1N

1N

OR

OR OR

1 1Pr( 1)N α= =

OR

OR OR

Successive Decoder

Decoder 2

1W

2W

nX

2nY

1nY

nNΔ1nN

1nN

OR

1W

2W 2nX

1nX

– Channel Model

– Optimal Transmission Strategy[Xie07]


Proving the Optimal Transmission Strategy

All rate pairs on the boundary of the capacity region can be achieved with the following strategy:

48

X Y1

1α

Y2

αΔ

X2

1p1q2q

2pγ

0,γ =

1 1 1(1 ) (1 )1

1 1,(1 )( 1)H q

e α αα − − ≤ ≤− +

2 1 2 1 21 2 1 2 1 1 1 1

2 1 2 1 1

1 (1 ) log(1 (1 ))( (1 )) (1 )log ( ( (1 )) (1 )).(1 ) log(1 (1 ))

q q qH q q H q q Hq q q

α αα α α αα α

− − − −− − − = − − −

− − −

25


A class of degraded broadcast channels

• function-like component channels

• have the same alphabet. • The channel function is commutative.• The channel function is associative.

The Department of Electrical Engineering UCLA 49

X2Y

1Y

NΔ1N

1N

f

f f

1 1 2, , , ,X N N Y YΔ


A Conjecture• We believe that the optimal transmission

strategies for this class of degraded broadcast channels are independent encoding schemes.


f

f f

Successive Decoder

Decoder 2

1W

2W

nX

2nY

1nY

nNΔ1nN

1nN

f1W

2W 2nX

1nX

26


Explicit Broadcast Z Channel Capacity Region

– The boundary of the capacity region is

where parameters satisfy

51

1 2 1 1 2 1 1

2 2 1 2 2 1 2

( (1 )) ((1 ))( (1 )) ( (1 ))

R q H q q q HR H q q q H q

α αα α

= − − −⎧⎨ = − − −⎩

1 1 1(1 ) /(1 )1

1 1,(1 )( 1)H q

e α αα − − ≤ ≤− +

1 2,q q

2 1 2 1 21 2 1 2 1 1 1 1

2 1 2 1 1

1 (1 ) log(1 (1 ))( (1 )) (1 )log ( ( (1 )) (1 )).(1 ) log(1 (1 ))

q q qH q q H q q Hq q q

α αα α α αα α

− − − −− − − = − − −

− − −


Z-channel Successive Decoder• Decoder structure of the successive decoder

for user 1

52

1 21 1 2

2

ˆif 0ˆ( , )

ˆif 1y x

y e y xerasure x

=⎧= = ⎨ =⎩

27


Nonlinear Turbo Codes• Nonlinear turbo codes can provide a

controlled distribution of ones and zeros.• Nonlinear turbo codes designed for Z

channels are used. [Griot06]

• Encoding structure of nonlinear turbo codes:

53


Simulation Results

54

The cross probabilities of the broadcast Z channel are

The simulated rates are very close to the capacity region.

Only 0.04 bits or less away from optimal rates in R1.

Only 0.02 bits or less away from optimal rates in R2.

1 20.15, 0.6.α α= =

28


References• [Cover98] T. M. Cover, “Comments on broadcast channels,”

IEEE Trans. Inform. Theory, vol. IT-44, pp. 2524-2530, 1998• [Cover75] T. M. Cover, “An achievable rate region for the

broadcast channel,” IEEE Trans. Inform. Theory, vol. IT-21 ,pp. 399-404, 1975

• [Hajek79] B. E. Hajek & M. B. Pursley, “Evaluation of an achievable rate region for the broadcast channel,” IEEE Trans. Inform. Theory, vol. IT-25 ,pp. 36-46, 1979

• [Gallager74] R. G. Gallager, “Capacity and coding for degraded broadcast channels,” Probl. Pered. Inform., vol. 10, no. 3, pp. 3–14, July–Sept. 1974; translated in Probl. Inform. Transm., pp. 185–193, July–Sept. 1974.

• [Benzel79] R. Benzel, “The capacity region of a class of discrete additive degraded interference channels,” IEEE Trans. Inform. Theory, vol. IT-25, pp. 228–231, Mar. 1979.



References• [Sato78] H. Sato, “An outer bound to the capacity region of

broadcast channels,” IEEE Trans. Inform. Theory, vol. IT-24 ,pp. 374-377, 1978.

• [VAN75] E. van der Meulen, “Random coding theorems for the general discrete memoryless broadcast channel,” IEEE Trans. Inform. Theory, vol. IT-21, pp. 180-190, 1975.

• [Marton79] K. Marton, “A coding theorem for the discrete memoryless broadcast channel,” IEEE Trans. Inform. Theory, vol. IT-25, pp. 306–311,1979.

• [Cover72] T. M. Cover, “Broadcast channels,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 2–14, Jan. 1972.

• [Bergmans73] P. P. Bergmans, “Random coding theorem for broadcast channels with degraded components,” IEEE Trans. Inform. Theory, vol. IT-19, pp. 197–207, Mar. 1973.


29


References• [Bergmans74] P. P. Bergmans, “A simple converse for

broadcast channels with additive white Gaussian noise,” IEEE Trans. Inform. Theory, vol. IT-20, pp. 279–280, Mar. 1974.

• [Griot06] M.Griot, A. I. Vila Casado and R. D. Wesel, "Non-linear Turbo Codes for Interleaver-Division Multiple Access on the OR Channel". Globecom 2006, 27 Nov. - 1 Dec., San Francisco, USA.

• [Xie07] B. Xie, M. Griot, A. I. Vila Casado and R. D. Wesel, "Optimal Transmission Strategy and Capacity Region for Broadcast Z Channels", IEEE Information Theory Workshop 2007.


Documents

New Results in Coding and Communication Theory