Lecture 4:Lecture 4:Data Compression TechniquesData Compression Techniques

TSBK01 Image Coding and Data CompressionTSBK01 Image Coding and Data Compression

Jörgen AhlbergDiv. of Sensor Technology

Swedish Defence Research Agency (FOI)

OutlineOutline

Huffman codingHuffman coding Arithmetic codingArithmetic coding

Application: JBIGApplication: JBIG

Universal codingUniversal coding LZ-codingLZ-coding

LZ77, LZ78, LZWLZ77, LZ78, LZW

Applications: GIF and PNGApplications: GIF and PNG

RepetitionRepetition

Coding:Coding: Assigning binary codewords to Assigning binary codewords to (blocks of) source symbols.(blocks of) source symbols.

Variable-lengthVariable-length codes (VLC) and codes (VLC) and fixed-fixed-lengthlength codes. codes.

Instantaneous codes Instantaneous codes ½½ Uniqely decodable Uniqely decodable codes codes ½½ Non-singular codes Non-singular codes ½½ All codes All codes

Tree codesTree codes are instantaneous. are instantaneous. Tree code Tree code ,, Kraft’s Inequality. Kraft’s Inequality.

Creating a Code: The Data Creating a Code: The Data Compression ProblemCompression Problem

Assume a source with an alphabet Assume a source with an alphabet AA and and known symbol probabilities known symbol probabilities {{ppii}}..

GoalGoal: Chose the codeword lengths as to : Chose the codeword lengths as to minimize the bitrate, i.e., the average number minimize the bitrate, i.e., the average number of bits per symbol of bits per symbol llii ¢¢ ppii..

Trivial solution:Trivial solution: llii = 0 = 0 88 ii.. Restriction:Restriction: We want an instantaneous code, We want an instantaneous code,

so so 22-l-l

ii ·· 1 1 (KI) must be valid. (KI) must be valid. Solution Solution (at least in theory)(at least in theory):: llii = – = – loglog p pii

In practice…In practice…

Use some nice algorithm to find the code Use some nice algorithm to find the code treetree– Huffman codingHuffman coding– Tunnstall codingTunnstall coding

Huffman CodingHuffman Coding

Two-step algorithm:Two-step algorithm:1.1. Iterate:Iterate:

– Merge the least probable symbols.Merge the least probable symbols.– Sort.Sort.

2.2. Assign bits.Assign bits.

a

d

b

c

0.5

0.25

0.125

0.125

0.5

0.25

0.5

0.25

0.5

Merge

Sort

Assign

0

1

0

10

11

0

10

110

111Get code

Coding of the BMSCoding of the BMS

Trick:Trick: Code blocks of symbols (extended source). Code blocks of symbols (extended source). Example:Example: pp11 = ¼ , = ¼ , pp22 = ¾. = ¾.

Applying the Huffman algorithm directly:Applying the Huffman algorithm directly:1 bit/symbol1 bit/symbol..

BlockBlock PP(block)(block) CodeCode

0000 9/169/16 00 ))

0101 3/163/16 1010 approx 0.85 approx 0.85

1010 3/163/16 110110 bits/symbolbits/symbol

1111 1/161/16 111111

Huffman Coding: Pros and ConsHuffman Coding: Pros and Cons

++ Fast implementations.Fast implementations.++ Error resilient: resynchronizes in Error resilient: resynchronizes in ~ ~ ll2 2 steps.steps.- The code tree grows exponentially when the The code tree grows exponentially when the

source is extended.source is extended.- The symbol probabilities are built-in in the The symbol probabilities are built-in in the

code.code.Hard to use Huffman coding for extended Hard to use Huffman coding for extended sources / large alphabets or when the sources / large alphabets or when the symbol probabilities are varying by time.symbol probabilities are varying by time.

Arithmetic CodingArithmetic Coding

Shannon-Fano-EliasShannon-Fano-Elias Basic idea:Basic idea: Split the interval [0,1] according Split the interval [0,1] according

to the symbol probabilities.to the symbol probabilities. Example:Example: AA = {a,b,c,d}, = {a,b,c,d}, PP = {½, ¼, 1/8, 1/8}. = {½, ¼, 1/8, 1/8}.

b

c

a

0.6

0.5

0.5

0.20.8

0.2

0.2 Start in b.Code the sequence c c a.

cba 0.9

Code the sequence c c a.) Code the interval [0.9, 0.96]

Bit Interval Decoder

1 c0.5 - 1

1 0.75 - 1

1 0.875 - 1

0 0.875 - 0.9375

1 c a0.90624 - 0.9375

b c0.5 10

cba

0.9 10.96 0.98

An Image Coding ApplicationAn Image Coding Application

Consider the image content in a local environment Consider the image content in a local environment of a pixel as of a pixel as a state in a Markov modela state in a Markov model..

Example (binary image):Example (binary image):

Such an environment is called a Such an environment is called a contextcontext.. A probability distribution for A probability distribution for XX can be estimated for can be estimated for

each state. Then arithmetic coding is used.each state. Then arithmetic coding is used. This is the basic idea behind the JBIG algorithm This is the basic idea behind the JBIG algorithm

for binary images and data.for binary images and data.

X

0 0 11 0

Flushing the CoderFlushing the Coder

The coding process is ended (restarted) and The coding process is ended (restarted) and the coder flushedthe coder flushed– after a given number of symbols (FIVO)after a given number of symbols (FIVO)

oror– When the interval is too small for a fixed When the interval is too small for a fixed

number of output bits (VIFO).number of output bits (VIFO).

Universal CodingUniversal Coding

A universal coder doesn’t need to know the A universal coder doesn’t need to know the statistics in advance. Instead, estimate from statistics in advance. Instead, estimate from data.data.

Forward estimation:Forward estimation: Estimate statistics in a Estimate statistics in a first pass and transmit to the decoder.first pass and transmit to the decoder.

Backward estimation:Backward estimation: Estimate from already Estimate from already transmitted (received) symbols.transmitted (received) symbols.

Universal Coding: ExamplesUniversal Coding: Examples

1.1. An adaptive arithmetic coderAn adaptive arithmetic coder

2.2. An adaptive dictionary techniqueAn adaptive dictionary technique– The LZ coders The LZ coders [Sayood 5][Sayood 5]

3.3. An adaptive Huffman coder An adaptive Huffman coder [Sayood 3.4][Sayood 3.4]

Arithmeticcoder

Statisticsestimation

Ziv-Lempel Coding (ZL or LZ)Ziv-Lempel Coding (ZL or LZ)

Named after J. Ziv and A. Lempel (1977).Named after J. Ziv and A. Lempel (1977). Adaptive dictionary technique.Adaptive dictionary technique.

– Store previously coded symbols in a buffer.Store previously coded symbols in a buffer.– Search for the current sequence of symbols to Search for the current sequence of symbols to

code.code.– If found, transmit buffer offset and length.If found, transmit buffer offset and length.

LZ77LZ77

a b c a b d a c a b d e e e fc

Search buffer Look-ahead buffer

Output triplet <offset, length, next>

123456783

8 0 13 d 0 e 2 f

2

Transmitted to decoder:

If the size of the search buffer is N and the size of the alphabet is Mwe need

bits to code a triplet.

Variation:Variation: Use a VLC to code the triplets!PKZip, Zip, Lharc,PNG, gzip, ARJ

Drawback with LZ77Drawback with LZ77

Repetetive patterns with a period longer Repetetive patterns with a period longer than the search buffer size are not found.than the search buffer size are not found.

If the search buffer size is 4, the sequenceIf the search buffer size is 4, the sequence a b c d e a b c d e a b c d e a b c d e …a b c d e a b c d e a b c d e a b c d e …will be expanded, not compressed.will be expanded, not compressed.

LZ78LZ78

Store patterns in a dictionaryStore patterns in a dictionary Transmit a tuple <Transmit a tuple <dictionary indexdictionary index, , nextnext>>

LZ78LZ78

Output tuple <dictionary index, next>

Dictionary:

1 a

2 b

3 c

4 a b

5 a b c

a b c a b a b c

0 aTransmitted to decoder: 0 b 0 c 1 b 4 c

Decoded: a b c a a bb c

Strategy needed for limiting dictionary size!Strategy needed for limiting dictionary size!

LZWLZW

Modification to LZ78 by Terry Welch, 1984.Modification to LZ78 by Terry Welch, 1984. Applications: GIF, v42bisApplications: GIF, v42bis Patented by UniSys Corp.Patented by UniSys Corp. Transmit only the dictionary index.Transmit only the dictionary index. The alphabet is stored in the dictionary in The alphabet is stored in the dictionary in

advance.advance.

LZWLZW

Output: dictionary index

Encoder dictionary:

1 a

2 b

3 c

4 d

5 a b

a b c a b a b c

1

Transmitted:

2 3 5 5

Decoded:

a b c a b a b

6 bc

7 ca

8 aba

9 abc

Decoder dictionary:

1 a

2 b

3 c

4 d

5 a b

6 bc

7 ca

8 aba

Input sequence:

And now for some applications:GIF & PNG

GIFGIF

CompuServe Graphics Interchange Format (1987, CompuServe Graphics Interchange Format (1987, 89).89).

Features:Features:– Designed for up/downloading images to/from BBSes via Designed for up/downloading images to/from BBSes via

PSTN.PSTN.– 1-, 4-, or 8-bit 1-, 4-, or 8-bit colour palettescolour palettes..– Interlace for Interlace for progressive decodingprogressive decoding (four passes, starts (four passes, starts

with every 8th row).with every 8th row).– Transparent colourTransparent colour for non-rectangular images. for non-rectangular images.– Supports multiple images in one file (”animated GIFs”).Supports multiple images in one file (”animated GIFs”).

GIF: MethodGIF: Method

Compression by LZW.Compression by LZW. Dictionary size 2Dictionary size 2bb+1+1 8-bit symbols 8-bit symbols

– bb is the number of bits in the palette. is the number of bits in the palette.

Dictionary size doubled if filled (max 4096).Dictionary size doubled if filled (max 4096). Works well on computer generated images.Works well on computer generated images.

GIF: ProblemsGIF: Problems

Unsuitable for natural images (photos):Unsuitable for natural images (photos):– Maximum 256 colors (Maximum 256 colors ()) bad quality). bad quality).– Repetetive patterns uncommon (Repetetive patterns uncommon ()) bad bad

compression).compression).

LZW patented by UniSys Corp.LZW patented by UniSys Corp. Alternative: PNGAlternative: PNG

PNG: Portable Network GraphicsPNG: Portable Network Graphics

Designed to replace GIF.Designed to replace GIF. Some features:Some features:

– Indexed Indexed oror true-colour images true-colour images ((·· 16 bits per plane) 16 bits per plane)..– Alpha channel.Alpha channel.– Gamma information.Gamma information.– Error detection.Error detection.

No support for multiple images in one file.No support for multiple images in one file.– Use MNG for that.Use MNG for that.

Method:Method:– Compression by LZ77 using a 32KB search buffer.Compression by LZ77 using a 32KB search buffer.– The LZ77 triplets are Huffman coded.The LZ77 triplets are Huffman coded.

More information:More information: www.w3.org/TR/REC-png.html www.w3.org/TR/REC-png.html

SummarySummary

Huffman codingHuffman coding– Simple, easy, fastSimple, easy, fast– Complexity grows exponentially with the block lengthComplexity grows exponentially with the block length– Statistics built-in in the codeStatistics built-in in the code

Arithmetic codingArithmetic coding– Complexity grows linearly with the block sizeComplexity grows linearly with the block size– Easily adapted to variable statistics Easily adapted to variable statistics )) used for coding of Markov used for coding of Markov

sourcessources Universal codingUniversal coding

– Adaptive Huffman or arithmetic coderAdaptive Huffman or arithmetic coder– LZ77: Buffer with previously sent sequences <offset,length,next>LZ77: Buffer with previously sent sequences <offset,length,next>– LZ78: Dictionary instead of buffer <index,next>LZ78: Dictionary instead of buffer <index,next>– LZW: Modification to LZ78 <index>LZW: Modification to LZ78 <index>

Summary, contSummary, cont

Where are the algorithms used?Where are the algorithms used?– Huffman coding:Huffman coding: JPEG, MPEG, PNG, … JPEG, MPEG, PNG, …– Arithmetic coding: Arithmetic coding: JPEG, JBIG, MPEG-4, …JPEG, JBIG, MPEG-4, …– LZ77:LZ77: PNG, PKZip, Zip, gzip, … PNG, PKZip, Zip, gzip, …– LZW:LZW: compress, GIF, v42bis, … compress, GIF, v42bis, …

FinallyFinally

These methods work best if the source These methods work best if the source alphabet is small and the distribution alphabet is small and the distribution skewed.skewed.– TextText– GraphicsGraphics

Analog sources (images, sound) require Analog sources (images, sound) require other methodsother methods– complex dependenciescomplex dependencies– accepted distortionaccepted distortion