Hamming CodesHamming Codes11/17/0411/17/04

HistoryHistory In the late 1940’s Richard Hamming In the late 1940’s Richard Hamming

recognized that the further evolution of recognized that the further evolution of computers required greater reliability, in computers required greater reliability, in particular the ability to not only detect particular the ability to not only detect errors, but correct them. His search for errors, but correct them. His search for error-correcting codes led to the error-correcting codes led to the Hamming Codes, perfect 1-error Hamming Codes, perfect 1-error correcting codes, and the extended correcting codes, and the extended Hamming Codes, 1-error correcting and Hamming Codes, 1-error correcting and 2-error detecting codes.2-error detecting codes.

UsesUses Hamming Codes are still widely used Hamming Codes are still widely used

in computing, telecommunication, in computing, telecommunication, and other applications.and other applications.

Hamming Codes also applied inHamming Codes also applied in Data compressionData compression Some solutions to the popular puzzle Some solutions to the popular puzzle

The Hat GameThe Hat Game Block Turbo CodesBlock Turbo Codes

A [7,4] binary Hamming A [7,4] binary Hamming CodeCode

Let our codeword be (xLet our codeword be (x11 x x22 … x … x77) ) εε F F2277

xx33, x, x55, x, x66, x, x77 are chosen according to are chosen according to the message (perhaps the message the message (perhaps the message itself is (xitself is (x33 x x55 x x66 x x7 7 )).)).

xx44 := x := x55 + x + x66 + x + x77 (mod 2) (mod 2) xx22 := x := x33 + x + x66 + x + x77

xx11 := x := x33 + x + x55 + x + x77

[7,4] binary Hamming [7,4] binary Hamming codewordscodewords

A [7,4] binary Hamming A [7,4] binary Hamming CodeCode

Let a = xLet a = x44 + x + x55 + x + x66 + x + x77 (=1 iff one (=1 iff one of these bits is in error)of these bits is in error)

Let b = xLet b = x22 + x + x33 + x + x66 + x + x77 Let c = xLet c = x11 + x + x33 + x + x55 + x + x7 7

If there is an error (assuming at If there is an error (assuming at most one) then abc will be binary most one) then abc will be binary representation of the subscript of representation of the subscript of the offending bit. the offending bit.

A [7,4] binary Hamming A [7,4] binary Hamming CodeCode

If (yIf (y11 y y22 … y … y77) is received and abc ≠ ) is received and abc ≠ 000, then we assume the bit abc is in 000, then we assume the bit abc is in error and switch it. If abc=000, we error and switch it. If abc=000, we assume there were no errors (so if assume there were no errors (so if there are three or more errors we there are three or more errors we may recover the wrong codeword).may recover the wrong codeword).

Definition: Generator Definition: Generator and Check Matricesand Check Matrices

For an [n, k] linear code, the For an [n, k] linear code, the generator matrix is a k×n matrix for generator matrix is a k×n matrix for which the row space is the given which the row space is the given code.code.

A check matrix for an [n, k] is a A check matrix for an [n, k] is a generator matrix for the dual code. generator matrix for the dual code. In other words, an (n-k)×k matrix In other words, an (n-k)×k matrix MM for which for which MxMx = 0 for all x in the code. = 0 for all x in the code.

A Construction for binary A Construction for binary Hamming CodesHamming Codes

For a given r, form an r × 2For a given r, form an r × 2rr-1 matrix -1 matrix M, the columns of which are the binary M, the columns of which are the binary representations (r bits long) of 1, …, representations (r bits long) of 1, …, 22rr-1. -1.

The linear code for which this is the The linear code for which this is the check matrix is a [2check matrix is a [2rr-1, 2-1, 2rr-1 – r] binary -1 – r] binary Hamming Code = {Hamming Code = {xx=(x=(x11 x x22 … x … x nn) : ) : MxMxTT = 0}. = 0}.

Example Check MatrixExample Check Matrix A check matrix for a [7,4] binary A check matrix for a [7,4] binary

Hamming Code:Hamming Code:

Syndrome DecodingSyndrome Decoding Let Let yy = (y = (y11 y y22 … y … ynn) be a received ) be a received

codeword.codeword. The syndrome of y is The syndrome of y is SS:=:=LLrryyTT. If . If

SS=0 then there was no error. If =0 then there was no error. If SS ≠ ≠ 0 then S is the binary representation 0 then S is the binary representation of some integer 1 ≤ t ≤ n=2of some integer 1 ≤ t ≤ n=2rr-1 and -1 and the intended codeword is the intended codeword is xx = (y = (y11 … y … yrr+1 … y+1 … ynn).).

Example Using LExample Using L33

Suppose (1 0 1 0 0 1 0) is received. Suppose (1 0 1 0 0 1 0) is received.

100 is 4 in binary, so the intended 100 is 4 in binary, so the intended codeword was (1 0 1 1 0 1 0).codeword was (1 0 1 1 0 1 0).

Extended [8,4] binary Extended [8,4] binary Hamm. CodeHamm. Code

As with the [7,4] binary Hamming Code:As with the [7,4] binary Hamming Code: xx33, x, x55, x, x66, x, x77 are chosen according to the are chosen according to the

message. message. xx44 := x := x55 + x + x66 + x + x77 xx22 := x := x33 + x + x66 + x + x7 7 xx11 := x := x33 + x + x55 + x + x7 7

Add a new bit xAdd a new bit x00 such that such that xx00 = x = x1 1 ++ xx2 2 + x+ x33 + x + x44 + x + x55 + x + x66 + x + x7 7 . i.e., the . i.e., the

new bit makes the sum of all the bits zero. new bit makes the sum of all the bits zero. xx00 is called a parity check. is called a parity check.

Extended binary Hamming Extended binary Hamming CodeCode

The minimum distance between any The minimum distance between any two codewords is now 4, so an two codewords is now 4, so an extended Hamming Code is a 1-error extended Hamming Code is a 1-error correcting and 2-error detecting correcting and 2-error detecting code.code.

The general construction of a [2r, 2r-The general construction of a [2r, 2r-1 - r] extended code from a [2r –1, 2r 1 - r] extended code from a [2r –1, 2r –1 – r] binary Hamming Code is the –1 – r] binary Hamming Code is the same: add a parity check bit.same: add a parity check bit.

Check Matrix Check Matrix Construction of Extended Construction of Extended

Hamming CodeHamming Code The check matrix of an extended The check matrix of an extended

Hamming Code can be constructed Hamming Code can be constructed from the check matrix of a Hamming from the check matrix of a Hamming code by adding a zero column on the code by adding a zero column on the left and a row of 1’s to the bottom. left and a row of 1’s to the bottom.

Decoding Extended Decoding Extended Hamming CodeHamming Code

q-ary Hamming Codesq-ary Hamming Codes The binary construction generalizes The binary construction generalizes

to Hamming Codes over an alphabet to Hamming Codes over an alphabet A={0, …, q}, q ≥ 2.A={0, …, q}, q ≥ 2.

For a given r, form an r × (qFor a given r, form an r × (qrr-1)/(q-1) -1)/(q-1) matrix matrix MM over A, any two columns of over A, any two columns of which are linearly independent.which are linearly independent.

MM determines a [(q determines a [(qrr-1)/(q-1), (q-1)/(q-1), (qrr-1)/(q--1)/(q-1) – r] (= [n,k]) q-ary Hamming Code 1) – r] (= [n,k]) q-ary Hamming Code for which M is the check matrix. for which M is the check matrix.

Example: ternary [4, 2] Example: ternary [4, 2] HammingHamming

Two check matrices for the some Two check matrices for the some [4, 2] ternary Hamming Codes:[4, 2] ternary Hamming Codes:

Syndrome decoding: the q-Syndrome decoding: the q-ary caseary case

The syndrome of received word The syndrome of received word yy, , SS:=:=MyMyTT, will be a multiple of one of , will be a multiple of one of the columns of the columns of MM, say , say SS==ααmmii,, αα

scalar, scalar, mmii the i the ithth column of column of MM. . Assume an error vector of weight 1 Assume an error vector of weight 1 was introduced was introduced yy = = xx + (0 … + (0 … αα … 0… 0), ), αα in the iin the ithth spot spot..

Example: q-ary SyndromeExample: q-ary Syndrome [4,2] ternary with check matrix[4,2] ternary with check matrix

, word (0 1 1 1) received., word (0 1 1 1) received.

So decode (0 1 1 1) as So decode (0 1 1 1) as (0 1 1 1) – (0 0 2 0) = (0 1 2 1).(0 1 1 1) – (0 0 2 0) = (0 1 2 1).

Perfect 1-error Perfect 1-error correctingcorrecting

Hamming Codes are perfect 1-error Hamming Codes are perfect 1-error correcting codes. That is, any received correcting codes. That is, any received word with at most one error will be word with at most one error will be decoded correctly and the code has decoded correctly and the code has the smallest possible size of any code the smallest possible size of any code that does this.that does this.

For a given r, any perfect 1-error For a given r, any perfect 1-error correcting linear code of length n=2correcting linear code of length n=2rr-1 -1 and dimension n-r is a Hamming Code.and dimension n-r is a Hamming Code.

Proof: 1-error correctingProof: 1-error correcting A code will be 1-error correcting if A code will be 1-error correcting if

spheres of radius 1 centered at spheres of radius 1 centered at codewords cover the codespace, andcodewords cover the codespace, and

if the minimum distance between any if the minimum distance between any two codewords ≥ 3, since then spheres two codewords ≥ 3, since then spheres of radius 1 centered at codewords will of radius 1 centered at codewords will be disjoint.be disjoint.

Proof: 1-error correctingProof: 1-error correcting Suppose codewords Suppose codewords xx, , yy differ by 1 differ by 1

bit. Then bit. Then xx--yy is a codeword of weight is a codeword of weight 1, and 1, and MM((xx--yy) ≠ 0. Contradiction. If ) ≠ 0. Contradiction. If xx, , yy differ by 2 bits, then differ by 2 bits, then MM((xx--yy) is ) is the difference of two multiples of the difference of two multiples of columns of columns of MM. No two columns of . No two columns of MM are linearly dependent, so are linearly dependent, so MM((xx--yy) ≠ 0, ) ≠ 0, another contradiction. Thus the another contradiction. Thus the minimum distance is at least 3.minimum distance is at least 3.

PerfectPerfect A sphere of radius A sphere of radius δδ centered at x is centered at x is

S Sδδ(x)={y in A(x)={y in Ann : d : dHH(x,y) ≤ (x,y) ≤ δδ}. }. Where A is the alphabet, FWhere A is the alphabet, Fqq, and d, and dHH is is the Hamming distance.the Hamming distance.

A sphere of radius e contains A sphere of radius e contains words.words. If C is an e-error correcting code then If C is an e-error correcting code then

, so , so . .

PerfectPerfect This last inequality is called the This last inequality is called the

sphere packing bound for an e-error sphere packing bound for an e-error correcting code C of length n over Fcorrecting code C of length n over Fmm: :

where n is the length of the code where n is the length of the code and in this case e=1.and in this case e=1.

A code for which equality holds is A code for which equality holds is called perfect.called perfect.

Proof: PerfectProof: Perfect

The right side of this, for e=1 is The right side of this, for e=1 is qqnn/(1+n(q-1))./(1+n(q-1)).

The left side is qThe left side is qn-rn-r where n= where n= (q(qrr-1)/(q-1). -1)/(q-1). qqn-rn-r(1+n(q-1)) = q(1+n(q-1)) = qn-rn-r(1+(q(1+(qrr-1)) = q-1)) = qnn..

ApplicationsApplications Data compression.Data compression. Turbo CodesTurbo Codes The Hat GameThe Hat Game

Data CompressionData Compression Hamming Codes can be used for a form of Hamming Codes can be used for a form of

lossy compression.lossy compression. If n=2If n=2rr-1 for some r, then any n-tuple of -1 for some r, then any n-tuple of

bits bits xx is within distance at most 1 from a is within distance at most 1 from a Hamming codeword Hamming codeword cc. Let G be a . Let G be a generator matrix for the Hamming Code, generator matrix for the Hamming Code, and and mGmG==cc. .

For compression, store For compression, store xx as as mm. For . For decompression, decode decompression, decode mm as as cc. This . This saves r bits of space but corrupts (at saves r bits of space but corrupts (at most) 1 bit.most) 1 bit.

The Hat GameThe Hat Game A group of n players enter a room A group of n players enter a room

whereupon they each receive a hat. Each whereupon they each receive a hat. Each player can see everyone else’s hat but not player can see everyone else’s hat but not his own. his own.

The players must each simultaneously The players must each simultaneously guess a hat color, or pass.guess a hat color, or pass.

The group loses if any player guesses the The group loses if any player guesses the wrong hat color or if every player passes.wrong hat color or if every player passes.

Players are not necessarily anonymous, Players are not necessarily anonymous, they can be numbered.they can be numbered.

The Hat GameThe Hat Game Assignment of hats is assumed to be Assignment of hats is assumed to be

random.random. The players can meet beforehand to The players can meet beforehand to

devise a strategy.devise a strategy. The goal is to devise the strategy The goal is to devise the strategy

that gives the highest probability of that gives the highest probability of winning.winning.

SourceSource ““Notes on Coding Theory” by J.I. Notes on Coding Theory” by J.I.

HallHallhttp://www.mth.msu.edu/~jhall/http://www.mth.msu.edu/~jhall/classes/codenotes/coding-notes.htmlclasses/codenotes/coding-notes.html