Transmitters Compression

8/8/2019 Transmitters Compression

1/18

Increasing Information per Bit Information in a source

Mathematical Models of Sources

Information Measures

Compressing information Huffman encoding

Optimal Compression for DMS?

Lempel-Ziv-Welch Algorithm For Stationary Sources?

Practical Compression

Quantization of analog data Scalar Quantization

Vector Quantization

Model Based Coding

Practical Quantization Q-law encoding Delta Modulation

Linear Predictor Coding (LPC)


2/18

Huffman encoding Variable length binary code for DMS

finite alphabet, fixed probabilities

Code satisfies thePrefix Condition

Codes are instantaneously and unambiguously

decodable as they arrive

e.g, 0,10,110,111 is OK0,01,011,111 is not OK 01 = 0,111, or 01


3/18

Huffman encoding

Use Probabilities to order coding priorities of letters Low probability get codes first (more bits) This smoothes out the information per bit


4/18

Huffman encoding Use a code tree to make the code

Combine the symbols with lowest probability to make a new block symbol Assign a 1 to one of the old symbols code word and 0 to the other symbol

Now reorder and combine the two lowest probability symbols of the new set

Each time the synthesized block symbol has lowest probability the code wordsget shorter

x1 00x2 01

x3 10

x4 110

x5 1110

x6 11110

x7 11111

D0 D1 D2 D3 D4


5/18

Huffman encoding

Result: Self Information or Entropy

H(X) = 2.11 (The best possible average number of bits)

Average number of bits per letter

21.2

)(7

1

_

!

! !k

kkxPnR

nk= number of bits per symbol

So the efficiency = %5.95)(

_!

R

XH


6/18

Huffman encoding

Lets compare to simple 3 bit code

x1 00 2 0.35 000 3 0.35

x2 01 2 0.3 001 3 0.3x3 10 2 0.2 010 3 0.2

x4 110 3 0.1 011 3 0.1

x5 1110 4 0.04 100 3 0.04

x6 11110 5 0.005 101 3 0.005x7 11111 5 0.005 110 3 0.005

R= 2.21 R= 3

Efficiency 95% 70%


7/18

Huffman encoding

Another example

x1 00x2 010

x3 011

x4 100

x5 101

x6 110

x7 1110

X8 1111

D0 D1 D2 D3


8/18

Huffman encoding

Multi-symbol Block codes Use symbols made of original symbols

)(lim

1)(

,

_

_

2

772111

XHJ

R

JXH

J

R

LJxxxxxx

J

J

j

!

!!

gp

- Symbols

Can show the new codes information per bit satisfies:)(_

JR

So large enough block code gets you as close to H(X) as you want


9/18

Huffman encoding

Lets consider

a J=2 block

code example


10/18

Encoding Stationary Sources

Now there are joint probabilities of blocks

of symbols that depend on previous bits

)()()()(

)|()|()|()()(

321

1121312121

k

kkk

xPxPxPxP

xxxPxxxPxxPxPxxxP

-

---

{

!

Unless DMS

Can show the joint entropy is:

!

e

!k

m

m

kkk

XH

XXXXHXXHXHXXXH

1

12112121

)(

)|()|()()( ...

Which means less bits than a symbol by symbol code can be used


11/18

Encoding Stationary Sources H(X| Y) is the conditional entropy

! !

!n

i

m

j

jijji yxyyxYXH1 1

2 )(log)()()(

Joint (total) probability ofxi

Information inxi givenyj

))()(

)((log)()()(

1 1

2 i

n

i

m

j j

ij

iij xPyP

xyPxPxyPYXH

! !

!

Can show:


12/18

Conditional Entropy Plotting this forn = m = 2 we see that when

Ydepends strongly on Xthen H(X|Y) is low

Conditional Entropy H(X|Y) vs P(X=0) for various P(Y|X)

0.00E+00

2.00E-01

4.00E-01

6.00E-01

8.00E-01

1.00E+00

1.20E+00

0.00E+00 2.00E-01 4.00E-01 6.00E-01 8.00E-01 1.00E+00 1.20E+00

P(X=0)

H(X|

Y)

0.05

0.15

0.35

0.5

0.75

0.9999

P(Y=0|X=0)

P(Y=1|X=1)


13/18

Conditional Entropy To see how P(X|Y) and P(Y|X) relate consider:

They are very simlar when P(X=0) ~ 0.5

0.00E+00

2.00E-01

4.00E-01

6.00E-01

8.00E-01

1.00E+00

1.20E+00

0.00E+00 2.00E-01 4.00E-01 6.00E-01 8.00E-01 1.00E+00 1.20E+00

P(X=0)

0.05

0.15

0.35

0.5

0.75

0.95

P(X=0| =0)

P(Y=0|X=0)


14/18

Optimal Codes for Stationary Sources

Can show that for large blocks of symbols

Huffman encoding is efficient

gp

p

e

gp

e

!

gg

J

XHRXH

J

JXHRXH

XXXHJXH

JJ

JJ

0

)()(

1)()(

)(

1

)(

_

_

21

I

I

.Define

Then Huffman code gives:

Now if

Get:

i.e., Huffman is optimal


15/18

Lempel-Ziv-Welch Code

Huffman encoding is efficient but need to

know joint probabilities of large blocks of

symbols

Finding joint probabilities is hard

LZW is independent of source statistics

Is a universal source code algorithm

Is not optimal


16/18

Lempel-Ziv-Welch

Build a table from strings not already in the table Output table location for strings in the table

Build the table again to decode

Input String = /WED/WE/WEE/WEB/WET

Characters Input Code Output New code value New String

/ / 256 /W

W W 257 WE

E E 258 ED

D D 259 D/

/W 256 260 /WE

E E 261 E/

/WE 260 262 /WEE

E/ 261 263 E/W

WE 257 264 WEB

B B 265 B/

/WE 260 266 /WET

T T

Source: http://dogma.net/markn/articles/lzw/lzw.htm


17/18

Lempel-Ziv-Welch

DecodeInput Codes: / W E D 256 E 260 261 257 B 260 T

Input/

NEW_CODEOLD_CODE

STRING/

OutputCHARACTER New table entry

/ / /

W / W W 256 = /W

E W E E 257 = WE

D E D D 258 = ED

256 D /W / 259 = D/

E 256 E E 260 = /WE

260 E /WE / 261 = E/

261 260 E/ E 262 = /WEE

257 261 WE W 263 = E/W

B 257 B B 264 = WEB

260 B /WE / 265 = B/

T 260 T T 266 = /WET


18/18

Lempel-Ziv-Welch Typically takes hundreds of entries to table

before compression occurs

Some nasty patents make licensing an issue

Documents

Transmitters Compression