Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
1
Digital Communication SystemsECS 452
Asst. Prof. Dr. Prapun Suksompong(ผศ.ดร.ประพันธ ์สขุสมปอง)
1. Intro to Digital Communication Systems
Office Hours: BKD, 6th floor of Sirindhralai building
Tuesday 14:20-15:20Wednesday 14:20-15:20Friday 9:15-10:15
2
“The fundamental problem of
communication is that of
reproducing at one point either exactly or approximately a message selected at another point.”
Shannon, Claude. A Mathematical Theory Of
Communication. (1948)
Shannon: Father of the Info. Age
3 [http://www.uctv.tv/shows/Claude-Shannon-Father-of-the-Information-Age-6090][http://www.youtube.com/watch?v=z2Whj_nL-x8]
Documentary Co-produced by the
Jacobs School, UCSD-TV, and the California Institute for Telecommunications and Information Technology
Won a Gold award in the Biography category in the 2002 Aurora Awards.
C. E. Shannon (1916-2001)
4
1938 MIT master's thesis: A Symbolic Analysis of Relay and Switching Circuits
Insight: The binary nature of Booleanlogic was analogous to the ones and zeros used by digital circuits.
The thesis became the foundation of practical digital circuit design.
The first known use of the term bit to refer to a “binary digit.”
Possibly the most important, and also the most famous, master’s thesis of the century.
It was simple, elegant, and important.
C. E. Shannon: Master Thesis
5
Boole/Shannon Celebration
6
Events in 2015 and 2016 centered around the work of George Boole, who was born 200 years ago, and Claude E. Shannon, born 100 years ago.
Events were scheduled both at the University College
Cork (UCC), Ireland and the Massachusetts
Institute of Technology (MIT)
http://www.rle.mit.edu/booleshannon/
An Interesting Book
7
The Logician and the Engineer: How George Boole and Claude Shannon Created the Information Age
by Paul J. Nahin
ISBN: 9780691151007
http://press.princeton.edu/titles/9819.html
C. E. Shannon (Con’t)
8
1948: A Mathematical Theory of Communication Bell System Technical Journal,
vol. 27, pp. 379-423, July-October, 1948.
September 1949: Book published. Include a new section by Warren Weaver that applied Shannon's theory to human communication.
Create the architecture and concepts governing digital communication.
Invent Information Theory: Simultaneously founded the subject, introduced all of the major concepts, and stated and proved all the fundamental theorems.
A Mathematical Theory of Communication
9
Link posted in the “references” section of the website.
[An offprint from the Bell System Technical Journal]
C. E. Shannon
10 …with some remarks by Toby Berger.
Claude E. Shannon Award
11
Claude E. Shannon (1972)
David S. Slepian (1974)
Robert M. Fano (1976)
Peter Elias (1977)
Mark S. Pinsker (1978)
Jacob Wolfowitz (1979)
W. Wesley Peterson (1981)
Irving S. Reed (1982)
Robert G. Gallager (1983)
Solomon W. Golomb (1985)
William L. Root (1986)
James L. Massey (1988)
Thomas M. Cover (1990)
Andrew J. Viterbi (1991)
Elwyn R. Berlekamp (1993)
Aaron D. Wyner (1994)
G. David Forney, Jr. (1995)
Imre Csiszár (1996)
Jacob Ziv (1997)
Neil J. A. Sloane (1998)
Tadao Kasami (1999)
Thomas Kailath (2000)
Jack KeilWolf (2001)
Toby Berger (2002)
Lloyd R. Welch (2003)
Robert J. McEliece (2004)
Richard Blahut (2005)
Rudolf Ahlswede (2006)
Sergio Verdu (2007)
Robert M. Gray (2008)
Jorma Rissanen (2009)
Te Sun Han (2010)
Shlomo Shamai (Shitz) (2011)
Abbas El Gamal (2012)
Katalin Marton (2013)
János Körner (2014)
Arthur Robert Calderbank (2015)
Alexander S. Holevo (2016)
[ http://www.itsoc.org/honors/claude-e-shannon-award ]
IEEE Richard W. Hamming Medal
12
1988 - Richard W. Hamming1989 - Irving S. Reed1990 - Dennis M. Ritchie and Kenneth L. Thompson1991 - Elwyn R. Berlekamp1992 - Lotfi A. Zadeh1993 - Jorma J. Rissanen1994 - Gottfried Ungerboeck1995 - Jacob Ziv1996 - Mark S. Pinsker1997 -Thomas M. Cover1998 - David D. Clark1999 - David A. Huffman2000 - Solomon W. Golomb2001 - A. G. Fraser2002 - Peter Elias
2003 - Claude Berrou and Alain Glavieux2004 - Jack K. Wolf2005 - Neil J.A. Sloane2006 -Vladimir I. Levenshtein2007 - Abraham Lempel2008 - Sergio Verdú2009 - Peter Franaszek2010 -Whitfield Diffie, Martin Hellman and Ralph Merkle2011 -Toby Berger2012 - Michael Luby, Amin Shokrollahi2013 - Arthur Robert Calderbank2014 -Thomas Richardson and Rüdiger L. Urbanke2015 - Imre Csiszar2016 - Abbas El Gamal
“For contributions to Information Theory, including source coding and its applications.”
[http://www.ieee.org/documents/hamming_rl.pdf]
[http://www.cvaieee.org/html/toby_berger.html]
Information Theory
13
The science of information theory tackles the following questions [Berger]
1. What is information, i.e., how do we measure it quantitatively?
2. What factors limit the reliability with which information generated at one point can be reproduced at another, and what are the resulting limits?
3. How should communication systems be designed in order to achieve or at least to approach these limits?
Elements of communication sys.
14
Noise, Interference,Distortion
ReceiverTransmitterInformation Source DestinationChannel
ReceivedSignal
TransmittedSignalMessage Message
ModulationCoding
Analog (continuous)Digital (discrete)
+ Transmission loss (attenuation)
AmplificationDemodulationDecodingFiltering
(ECS 332)
The Switch to Digital TV
15
Japan: Starting July 24, 2011, the analog broadcast has ceased and only digital broadcast is available.US: Since June 12, 2009, full-power television stations nationwide have been broadcasting exclusively in a digital format.Thailand: Use DVB-T2. Launched in 2014.
[https://upload.wikimedia.org/wikipedia/commons/thumb/b/bd/Digital_broadcast_standards.svg/800px-Digital_broadcast_standards.svg.png]
News: The Switch to Digital Radio
16
Norway (the mountainous nation of 5 million) is the first country to start shutting down its national FM radio network in favor of digital radio.
Start on January 11, 2017 At which point, 99.5% of the population has access to DAB reception
with almost three million receivers sold. 70% of Norwegian households regularly tune in digitally
Take place over a 12-month period, conducting changes region by region. By the end of the year all national networks will be DAB-only, while
local broadcasters have five years to phase out their FM stations.
New format: Digital Audio Broadcasting (DAB)
http://gizmodo.com/norway-is-killing-fm-radio-tomorrow-1791019824http://www.worlddab.org/country-information/norwayhttp://www.smithsonianmag.com/smart-news/norway-killed-radio-star-180961761/http://www.latimes.com/world/la-fg-norway-radio-20170114-story.htmlhttps://www.newscientist.com/article/2117569-norway-is-first-country-to-turn-off-fm-radio-and-go-digital-only/
Digital Audio Broadcasting
17
Initiated as a European research project in the 1980s.
The Norwegian Broadcasting Corporation (NRK) launched the first DAB channel in the world on 1 June 1995 (NRK Klassisk)
The BBC and Swedish Radio (SR) launched their first DAB digital radio broadcasts in September 1995.
Audio quality varies depending on the bitrate used.
The Switch to DAB in Norway
18
Co-exist with FM since 1995. Provide a clearer and more reliable network that can better
cut through the country's sparsely populated rocky terrain. FM has always been problematic in Norway since the nation’s
mountains and fjords makes getting clear FM signals difficult.
Offer more channels at a fraction of the cost. Allow 8 times as many radio stations Norway currently has five national FM radio stations. With DAB, it will be able to have around 40.
The FM radio infrastructure was coming to the end of its life, Need to either replace it or fully commit to DAB anyway
Can run at lower power levels the infrastructure electricity bills are lower
The Switch to Digital Radio
19
Switzerland and Denmark are also interested in phasing out FM Great Britain says it will look at making the switch
once 50 percent of listeners use digital formats currently at 35 percent Unlikely to happen before 2020.
and when the DAB signal reaches 90 percent of the population. Germany had set a 2015 date for dumping FM many years ago, but
lawmakers reversed that decision in 2011. In North America,
FM radio, which has been active since the 1940s, shows no sign of being replaced any time soon, either in the United States or Canada.
There are around 4,000 stations using HD radio technology in the United States, and HD radio receivers are now common fixtures in new cars.
In Thailand, NBTC planed to start digital radio trial within 2018.
20
Selected by the U.S. FCC in 2002 as a digital audio broadcasting method for the United States.
Embed digital signal “on-frequency” immediately above and below a station’s standard analog signal
Provide the means to listen to the same program in either HD (digital radio with less noise) or as a standard broadcast (analog radio with standard sound quality).
Spectrum of FM broadcast station
without HD Radio with HD Radio
[ https://en.wikipedia.org/w
iki/HD
_Radio
]
Countries using DAB/DMB
21 https://en.wikipedia.org/wiki/Digital_audio_broadcasting
Pokémon Communications
22
Pikachu's language
23
Some of Pikachu's speech is consistent enough that it seems that some phrases actually mean something.
Pikachu always uses "Pikapi" when referring to Ash (notice that it sounds somewhat similar to "Satoshi").
Pi-Kachu: He says this during the sponsor spots in the original Japanese, Pochama (Piplup)
Pikachu-Pi: Kasumi (Misty)
Pika-Chu: Takeshi (Brock), Kibago (Axew)
Pikaka: Hikari (Dawn)
PiPiPi: Togepy (Togepi)
PikakaPika: Fushigidane (Bulbasaur)
PikaPika: Zenigame (Squirtle), Mukuhawk (Staraptor), Goukazaru (Infernape) or Gamagaru (Palpitoad)
PiPi-kachu: Rocket-dan (Team Rocket)
Pi-Pikachu: Get da ze! (He says this after Ash wins a Badge, catches a new Pokémon or anything similar.)
Pikachu may not be the only one to use this phrase, as other Pokémon do this as well. For example, when Iris caught Emolga, Axew said Ax-Axew (Ki-Kibago in the Japanese).
Pika-Pikachu: He says this when referring to himself.
Four-symbol variable-length code?
[https://www.youtube.com/watch?v=XumQrRkGXck]
Rate-Distortion Theory
24
The theory of lossy source coding
1
Digital Communication SystemsECS 452
Asst. Prof. Dr. Prapun [email protected]
2. Source Coding
Office Hours: BKD, 6th floor of Sirindhralai building
Tuesday 14:20-15:20Wednesday 14:20-15:20Friday 9:15-10:15
Elements of digital commu. sys.
2
Noise & In
terferen
ce
Information Source
Destination
Channel
ReceivedSignal
TransmittedSignal
Message
Recovered Message
Source Encoder
Channel Encoder
DigitalModulator
Source Decoder
Channel Decoder
DigitalDemodulator
Transmitter
Receiver
Remove redundancy
Add systematic redundancy
Main Reference
3
Elements of Information Theory
2006, 2nd Edition
Chapters 2, 4 and 5
‘the jewel in Stanford's crown’
One of the greatest information theorists since Claude Shannon (and the one most like Shannon in approach, clarity, and taste).
The ASCII Coded Character Set
4[The ARRL Handbook for Radio Communications 2013]
0 16 32 48 64 80 96 112
US UK
(American Standard Code for Information Interchange)
Introduction to Data Compression
5 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/compressioncodes ]
English Redundancy: Ex. 1
6
J-st tr- t- r--d th-s s-nt-nc-.
English Redundancy: Ex. 2
7
yxx cxn xndxrstxndwhxt x xm wrxtxngxvxn xf x rxplxcx xllthx vxwxls wxth xn 'x' (t gts lttl hrdr f y dn'tvn kn whr th vwls r).
English Redundancy: Ex. 3
8
To be, or xxx xx xx, xxxx xx xxx xxxxxxxx
Entropy Rate of Thai Text
9
Ex. Source alphabet of size = 4
10
Ex. DMS (1)
11
, , , ,X a b c d e
a c a c e c d b c ed a e e d a b b b db b a a b e b e d cc e d b c e c a a ca a e a c c a a d cd e e a a c a a a bb c a e b b e d b cd e b c a e e d d cd a b c a b c d d ed c e a b a a c a d
Information Source
1 , , , , ,50, otherwise
X
x a b c d ep x
Approximately 20% are letter ‘a’s[GenRV_Discrete_datasample_Ex1.m]
Ex. DMS (1)
12 [GenRV_Discrete_datasample_Ex1.m]
clear all; close all;
S_X = 'abcde'; p_X = [1/5 1/5 1/5 1/5 1/5];
n = 100;MessageSequence = datasample(S_X,n,'Weights',p_X)MessageSequence = reshape(MessageSequence,10,10)
>> GenRV_Discrete_datasample_Ex1
MessageSequence =
eebbedddeceacdbcbedeecacaecedcaedabecccabbcccebdbbbeccbadeaaaecceccdaccedadabceddaceadacdaededcdcade
MessageSequence =
eeeabbacdeeacebeeeadbcadcccdcebdcacccaedebabcbedacdceeeacadddbccbdcbacdeecdedccaeddcbaaeddcecabacdae
Ex. DMS (2)
13
1,2,3,4X
1 , 1,21 , 2,41 , 3,480, otherwise
X
x
xp x
x
Information Source
Approximately 50% are number ‘1’s
2 1 1 2 1 4 1 1 1 11 1 4 1 1 2 4 2 2 13 1 1 2 3 2 4 1 2 42 1 1 2 1 1 3 3 1 11 3 4 1 4 1 1 2 4 14 1 4 1 2 2 1 4 2 14 1 1 1 1 2 1 4 2 42 1 1 1 2 1 2 1 3 22 1 1 1 1 1 1 2 3 22 1 1 2 1 4 2 1 2 1
[GenRV_Discrete_datasample_Ex2.m]
Ex. DMS (2)
14 [GenRV_Discrete_datasample_Ex2.m]
clear all; close all;
S_X = [1 2 3 4]; p_X = [1/2 1/4 1/8 1/8];
n = 20;
MessageSequence = randsrc(1,n,[S_X;p_X]);%MessageSequence = datasample(S_X,n,'Weights',p_X);
rf = hist(MessageSequence,S_X)/n; % Ref. Freq. calc.stem(S_X,rf,'rx','LineWidth',2) % Plot Rel. Freq.hold onstem(S_X,p_X,'bo','LineWidth',2) % Plot pmfxlim([min(S_X)-1,max(S_X)+1])legend('Rel. freq. from sim.','pmf p_X(x)')xlabel('x')grid on
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
x
Rel. freq. from sim.pmf pX(x)
15
DMS in MATLABclear all; close all;
S_X = [1 2 3 4]; p_X = [1/2 1/4 1/8 1/8]; n = 1e6;
SourceString = randsrc(1,n,[S_X;p_X]);
rf = hist(SourceString,S_X)/n; % Ref. Freq. calc.stem(S_X,rf,'rx','LineWidth',2) % Plot Rel. Freq.hold onstem(S_X,p_X,'bo','LineWidth',2) % Plot pmfxlim([min(S_X)-1,max(S_X)+1])legend('Rel. freq. from sim.','pmf p_X(x)')xlabel('x')grid on
SourceString = datasample(S_X,n,'Weights',p_X);
Alternatively, we can also use
[GenRV_Discrete_datasample_Ex.m]
A more realistic example of pmf:
16 [http://en.wikipedia.org/wiki/Letter_frequency]
Relative freq. of letters in the English language
A more realistic example of pmf:
17
Relative freq. of letters in the English languageordered by frequency
[http://en.wikipedia.org/wiki/Letter_frequency]
Example: ASCII Encoder
18
Characterx
Codewordc(x)
⋮E 1000101
⋮L 1001100
⋮O 1001111
⋮V 1010110
⋮
SourceEncoder
Information Source
“LOVE”“1001100100111110101101000101”
>> M = 'LOVE';>> X = dec2bin(M,7);>> X = reshape(X',1,numel(X))X =1001100100111110101101000101
MATLAB:
c(“L”) c(“O”) c(“V”) c(“E”)
Cod
eboo
k
Remark:numel(A) = prod(size(A))(the number of elements in matrix A)
The ASCII Coded Character Set
19[The ARRL Handbook for Radio Communications 2013]
0 16 32 48 64 80 96 112
A Byte (8 bits) vs. 7 bits
20
>> dec2bin('I Love ECS452',7)ans =1001001010000010011001101111111011011001010100000100010110000111010011011010001101010110010
>> dec2bin('I Love ECS452',8)ans =01001001001000000100110001101111011101100110010100100000010001010100001101010011001101000011010100110010
>> dec2bin('I Love You',8)ans =01001001001000000100110001101111011101100110010100100000010110010110111101110101
Geeky ways to express your love
21
>> dec2bin('i love you',8)ans =01101001001000000110110001101111011101100110010100100000011110010110111101110101
https://www.etsy.com/listing/91473057/binary-i-love-you-printable-for-your?ref=sr_gallery_9&ga_search_query=binary&ga_filters=holidays+-supplies+valentine&ga_search_type=all&ga_view_type=galleryhttp://mentalfloss.com/article/29979/14-geeky-valentines-day-cardshttps://www.etsy.com/listing/174002615/binary-love-geeky-romantic-pdf-cross?ref=sr_gallery_26&ga_search_query=binary&ga_filters=holidays+-supplies+valentine&ga_search_type=all&ga_view_type=galleryhttps://www.etsy.com/listing/185919057/i-love-you-binary-925-silver-dog-tag-can?ref=sc_3&plkey=cdf3741cf5c63291bbc127f1fa7fb03e641daafd%3A185919057&ga_search_query=binary&ga_filters=holidays+-supplies+valentine&ga_search_type=all&ga_view_type=gallery http://www.cafepress.com/+binary-code+long_sleeve_tees
Morse code
22
Telegraph network
Samuel Morse, 1838
A sequence of on-off tones (or , lights, or clicks)
(wired and wireless)
Example
23 [http://www.wolframalpha.com/input/?i=%22I+love+you.%22+in+Morse+code]
Example
24
Morse code: Key Idea
25
Frequently-used characters are mapped to short codewords.
Relative frequencies of letters in the English language
Morse code: Key Idea
26
Frequently-used characters (e,t) are mapped to short codewords.
Relative frequencies of letters in the English language
Morse code: Key Idea
27
Frequently-used characters (e,t) are mapped to short codewords.
Basic form of compression.
รหสัมอร์สภาษาไทย
28
Example: ASCII Encoder
29
Character Codeword
⋮E 1000101
⋮L 1001100
⋮O 1001111
⋮V 1010110
⋮
SourceEncoder
Information Source
“LOVE”“1001100100111110101101000101”
>> M = 'LOVE';>> X = dec2bin(M,7);>> X = reshape(X',1,numel(X))X =1001100100111110101101000101
MATLAB:
Another Example of non-UD code
30
x c(x)
A 1
B 011
C 01110
D 1110
E 10011
Consider the string 011101110011.
It can be interpreted as CDB: 01110 1110 011 BABE: 011 1 011 10011
Game: 20 Questions
31
20 Questions is a classic game that has been played since the 19th century.
One person thinks of something (an object, a person, an animal, etc.)
The others playing can ask 20 questions in an effort to guess what it is.
20 Questions: Example
32
Shannon–Fano coding
33
Proposed in Shannon’s “A Mathematical Theory of Communication” in 1948
The method was attributed to Fano, who later published it as a technical report. Fano, R.M. (1949). “The transmission of information”.
Technical Report No. 65. Cambridge (Mass.), USA: Research Laboratory of Electronics at MIT.
Should not be confused with Shannon coding, the coding method used to prove Shannon's
noiseless coding theorem, or with Shannon–Fano–Elias coding (also known as Elias coding), the
precursor to arithmetic coding.
Prof. Robert Fano (1917-2016)Shannon Award (1976 )
Huffman Code
34
MIT, 1951 Information theory class taught by Professor Fano. Huffman and his classmates were given the choice of
a term paper on the problem of finding the most efficient binary code.
or a final exam.
Huffman, unable to prove any codes were the most efficient, was about to give up and start studying for the final when he hit upon the idea of using a frequency-sorted binary tree and quickly proved this method the most efficient.
Huffman avoided the major flaw of the suboptimal Shannon-Fanocoding by building the tree from the bottom up instead of from the top down.
David Huffman (1925–1999)Hamming Medal (1999)
Huffman’s paper (1952)
35[D. A. Huffman, "A Method for the Construction of Minimum-Redundancy Codes," in Proceedings of the IRE, vol. 40, no. 9, pp. 1098-1101, Sept. 1952.][ http://ieeexplore.ieee.org/document/4051119/ ]
Huffman coding
36 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/compressioncodes ]
Ex. Huffman Coding in MATLAB
37 [Huffman_Demo_Ex1]
Observe that MATLAB automatically give the expected length of the codewords
pX = [0.5 0.25 0.125 0.125]; % pmf of XSX = [1:length(pX)]; % Source Alphabet[dict,EL] = huffmandict(SX,pX); % Create codebook
%% Pretty print the codebook.codebook = dict;for i = 1:length(codebook)
codebook{i,2} = num2str(codebook{i,2});endcodebook
%% Try to encode some random source stringn = 5; % Number of source symbols to be generatedsourceString = randsrc(1,10,[SX; pX]) % Create data using pXencodedString = huffmanenco(sourceString,dict) % Encode the data
[Ex. 2.31]
Ex. Huffman Coding in MATLAB
38
codebook =
[1] '0' [2] '1 0' [3] '1 1 1'[4] '1 1 0'
sourceString =
1 4 4 1 3 1 1 4 3 4
encodedString =
0 1 1 0 1 1 0 0 1 1 1 0 0 1 1 0 1 1 1 1 1 0
[Huffman_Demo_Ex1]
Ex. Huffman Coding in MATLAB
39 [Huffman_Demo_Ex2]
pX = [0.4 0.3 0.1 0.1 0.06 0.04]; % pmf of XSX = [1:length(pX)]; % Source Alphabet[dict,EL] = huffmandict(SX,pX); % Create codebook
%% Pretty print the codebook.codebook = dict;for i = 1:length(codebook)
codebook{i,2} = num2str(codebook{i,2});endcodebook
EL
[Ex. 2.32]
The codewords can be different from our answers found earlier.
The expected length is the same.
>> Huffman_Demo_Ex2
codebook =
[1] '1' [2] '0 1' [3] '0 0 0 0' [4] '0 0 1' [5] '0 0 0 1 0'[6] '0 0 0 1 1'
EL =
2.2000
Ex. Huffman Coding in MATLAB
40
pX = [1/8, 5/24, 7/24, 3/8]; % pmf of XSX = [1:length(pX)]; % Source Alphabet[dict,EL] = huffmandict(SX,pX); % Create codebook
%% Pretty print the codebook.codebook = dict;for i = 1:length(codebook)
codebook{i,2} = num2str(codebook{i,2});endcodebook
EL
[Exercise]
codebook = [1] '0 0 1'[2] '0 0 0'[3] '0 1' [4] '1'
EL =1.9583
>> -pX*(log2(pX)).'ans =
1.8956
Let’s talk about TV series on HBO
41
Silicon Valley (TV series)
42
Focus around six young men who found a startup company in Silicon Valley.
In the first season, the company develop a “revolutionary” data compression algorithm: The “middle-out” algorithm
Behind the Scene
43
When Mike Judge set out to write Silicon Valley, he wanted to conceive a simple, believable widget for his characters to invent.
He teamed with Stanford electrical engineering professorTsachyWeissman and a PhD student Vinith Misra
They came up with a fictional breakthrough compression algorithm. “We had to come up with an approach that isn’t possible today,
but it isn’t immediately obvious that it isn’t possible,” says Misra.
The writers also coined a metric, the “Weissman Score,” for characters to use when comparing compression codes.
[May 2014 issue of Popular Science]
Middle-Out Algorithm: Into the Real World
44
Something like it can be found in Lepton A new lossless image compressor created by Dropbox.
Lepton reduces the file size of JPEG-encoded images by as much as 22 percent, yet without losing a single bit of the original.
How is this possible? Middle-out. Well, it’s much more complicated than that, actually. The middle-out bit comes toward the end of the decompression
bit.
Lepton is open source, and Dropbox has put the code for it on GitHub.
[ https://techcrunch.com/2016/07/14/dropboxs-lepton-lossless-image-compression-really-uses-a-middle-out-algorithm/ ]
Weissman Score: Into the Real World
45
It’s hard to convey to a lay audience that one compression algorithm is better than another.
Created by Misra (Prof. Weissman’s PhD student) for the show.
Metrics for compression algorithms that rate not only the amount of compression but the processing speed, are hard to find.
http://spectrum.ieee.org/view-from-the-valley/computing/software/a-madefortv-compression-metric-moves-to-the-real-worldhttp://spectrum.ieee.org/view-from-the-valley/computing/software/a-madefortv-compression-algorithm
Summary
46
A good code must be uniquely decodable (UD). Difficult to check.
A special family of codes called prefix(-free) code is always UD. They are also instantaneous.
Huffman’s recipe Repeatedly combine the two least-likely (combined) symbols Automatically give prefix code
For a given source’s pmf, Huffman codes are optimal among all UD codes for that source.
47
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-0.4
-0.35
-0.3
-0.25
-0.2
-0.15
-0.1
-0.05
0
x
Entropy and Description of RV
48 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/information-entropy ]
Entropy and Description of RV
49 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/information-entropy ]
Kronecker Product
50
An operation on two matrices of arbitrary size
Named after German mathematician Leopold Kronecker.
If A is an m-by-n matrix and B is a p-by-q matrix, then the Kronecker product AB is the mp-by-nq matrix
Example
11 1
1
.n
m mn
a B a B
a B a B
A B
· · 2·0 2·5 0 5 0 102 · · 2·6 2·7 6 7 12 14
3 4 3·0 3·5 4·0 4·5 0 15 0 203·6 3·7 4·6 4·7 18
1 11
21 24
0 50 5 6
2
1
8
7.
16 7
Use kron(A,B)in MATLAB.
Kronecker Product
51
>> p = [0.9 0.1]p =
0.9000 0.1000>> p2 = kron(p,p)p2 =
0.8100 0.0900 0.0900 0.0100>> p3 = kron(p2,p)p3 =
Columns 1 through 70.7290 0.0810 0.0810 0.0090 0.0810 0.0090 0.0090
Column 80.0010
Huffman Coding: Source Extension
52
1 2 3 4 5 6 7 80.4
0.5
0.6
0.7
0.8
0.9
1
n: order of extension
i.i.d.
BernoullikX p0.1p
nL
1
0.533
0.645
[Ex.2.40]
Huffman Coding: Source Extension
53
1 2 3 4 5 6 7 80.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Order of source extension
H X
1H Xn
n
i.i.d.
BernoullikX p0.1p
nL
[Ex.2.40]
Final Remarks about Huffman Code
54
Huffman coding compresses an i.i.d. source with a known pmf pX(x) to its entropy limit H(X).
Sensitive to the assumed distribution. If the code is designed for some incorrect distribution, a penalty
is incurred.
What compression can be achieved if the true pmf pX(x) is unknown? One may assume uniform pmf Inefficient if the actual pmf is not uniform.
Lempel-Ziv algorithm
55
Often used in practice to compress data that cannot be modeled simply
Could also be described as adaptive dictionary compression algorithms.
Ziv and Lempel wrote their papers in 1977 and 1978.
The two papers describe two distinct versions of the algorithm. LZ77: sliding window Lempel–Ziv LZ78: tree-structured Lempel–Ziv
Lempel-Ziv algorithm
Arithmetic Coding
56
The Huffman coding procedure is optimal for encoding a random variable with a known pmf that has to be encoded symbol by symbol.
Coding Inefficiency of Huffman Code: The codeword lengths for a Huffman code were restricted to be
integer-valued There could be a loss of up to 1 bit per symbol in coding efficiency.
We could alleviate this loss by encoding blocks of source symbols The complexity of this approach increases exponentially with block
length n. In arithmetic coding, instead of using a sequence of bits to
represent a symbol, we represent it by a subinterval of the unit interval.
1
Digital Communication SystemsECS 452
Asst. Prof. Dr. Prapun [email protected]
3 Discrete Memoryless Channel (DMC)
Office Hours: BKD, 6th floor of Sirindhralai building
Tuesday 14:20-15:20Wednesday 14:20-15:20Friday 9:15-10:15
Elements of digital commu. sys.
2
Noise & In
terferen
ce
Information Source
Destination
Channel
ReceivedSignal
TransmittedSignal
Message
Recovered Message
Source Encoder
Channel Encoder
DigitalModulator
Source Decoder
Channel Decoder
DigitalDemodulator
Transmitter
Receiver
Remove redundancy(compression)
Add systematic redundancy to combat errors introduced by the channel
System considered previously
3
Noise & In
terferen
ce
Information Source
Destination
Channel
ReceivedSignal
TransmittedSignal
Message
Recovered Message
Source Encoder
Channel Encoder
DigitalModulator
Source Decoder
Channel Decoder
DigitalDemodulator
Transmitter
Receiver
Remove redundancy
Add systematic redundancy
System considered in this section
4
Noise & In
terferen
ce
Information Source
Destination
Channel
ReceivedSignal
TransmittedSignal
Message
Recovered Message
Source Encoder
Channel Encoder
DigitalModulator
Source Decoder
Channel Decoder
DigitalDemodulator
Transmitter
Receiver
Remove redundancy
Add systematic redundancy
EquivalentChannel
X: channel input
Y: channel output
MATLAB
6
%% Generating the channel input xx = randsrc(1,n,[S_X;p_X]); % channel input
%% Applying the effect of the channel to create the channel output yy = DMC_Channel_sim(x,S_X,S_Y,Q); % channel output
function y = DMC_Channel_sim(x,S_X,S_Y,Q)%% Applying the effect of the channel to create the channel output yy = zeros(size(x)); % preallocationfor k = 1:length(x)
% Look at the channel input one by one. Choose the corresponding row% from the Q matrix to generate the channel output.y(k) = randsrc(1,1,[S_Y;Q(find(S_X == x(k)),:)]);
end
[DMC_Analysis_demo.m]
[DMC_Channel_sim.m]
Ex: BSC
7
>> BSC_demoans =1 0 1 1 1 1 1 1 1 1 1 1 0 1 0 1 1 1 1 1ans =1 1 1 1 1 1 1 1 1 0 1 1 0 1 0 1 1 1 1 1
Elapsed time is 0.134992 seconds.
%% Simulation parameters% The number of symbols to be transmittedn = 20; % Channel Input S_X = [0 1]; S_Y = [0 1];p_X = [0.3 0.7];% Channel Characteristicsp = 0.1; Q = [1-p p; p 1-p];
[BSC_demo.m]
[Example 3.2]
p_X =0.3000 0.7000
p_X_sim =0.1500 0.8500
q =0.3400 0.6600
q_sim =0.1500 0.8500
Q =0.9000 0.10000.1000 0.9000
Q_sim =0.6667 0.33330.0588 0.9412
PE_sim =0.1000
PE_theretical =0.1000
Rel. freq. from the simulation
8
%% Statistical Analysis% The probability values for the channel inputsp_X % Theoretical probabilityp_X_sim = hist(x,S_X)/n % Relative frequencies from the simulation% The probability values for the channel outputsq = p_X*Q % Theoretical probabilityq_sim = hist(y,S_Y)/n % Relative frequencies from the simulation% The channel transition probabilities from the simulationQ_sim = [];for k = 1:length(S_X)
I = find(x==S_X(k)); LI = length(I);rel_freq_Xk = LI/n; yc = y(I);cond_rel_freq = hist(yc,S_Y)/LI; Q_sim = [Q_sim; cond_rel_freq];
endQ % Theoretical probabilityQ_sim % Relative frequencies from the simulation
[DMC_Analysis_demo.m]
Ex: BSC
9
>> BSC_demoans =1 0 1 1 1 1 1 1 1 1 1 1 0 1 0 1 1 1 1 1ans =1 1 1 1 1 1 1 1 1 0 1 1 0 1 0 1 1 1 1 1
Elapsed time is 0.134992 seconds.
%% Simulation parameters% The number of symbols to be transmittedn = 20; % Channel Input S_X = [0 1]; S_Y = [0 1];p_X = [0.3 0.7];% Channel Characteristicsp = 0.1; Q = [1-p p; p 1-p];
[BSC_demo.m]
p_X =0.3000 0.7000
p_X_sim =0.1500 0.8500
q =0.3400 0.6600
q_sim =0.1500 0.8500
Q =0.9000 0.10000.1000 0.9000
Q_sim =0.6667 0.33330.0588 0.9412
PE_sim =0.1000
PE_theretical =0.1000
[Example 3.2]
Because there are only 20 samples, we can’t expect the relative freq. from the simulation to match the specified or calculated probabilities.
Ex: BSC
10
%% Simulation parameters% The number of symbols to be transmittedn = 1e4; % Channel Input S_X = [0 1]; S_Y = [0 1];p_X = [0.3 0.7];% Channel Characteristicsp = 0.1; Q = [1-p p; p 1-p];
[BSC_demo.m]
>> BSC_demo
p_X =0.3000 0.7000
p_X_sim =0.3037 0.6963
q =0.3400 0.6600
q_sim =0.3407 0.6593
Elapsed time is 0.922728 seconds.
Q =0.9000 0.10000.1000 0.9000
Q_sim =0.9078 0.09220.0934 0.9066
PE_sim =0.0930
PE_theretical =0.1000
Ex: DMC
11
p_X =0.2000 0.8000
p_X_sim =0.2000 0.8000
q =0.3400 0.3600 0.3000
q_sim =0.4000 0.3500 0.2500
Q =0.5000 0.2000 0.30000.3000 0.4000 0.3000
Q_sim =0.7500 0 0.25000.3125 0.4375 0.2500
>> sym(Q_sim)ans =[ 3/4, 0, 1/4][ 5/16, 7/16, 1/4]PE_sim =
0.7500
>> DMC_demoans =1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 0 1 1 0 1ans =1 3 2 2 1 2 1 2 2 3 1 1 1 3 1 3 2 3 1 2
x:
y:
0
1
1
3
0.5
0.3
X Y
%% Simulation parameters% The number of symbols to be transmittedn = 20; % General DMC% Ex. 3.6 amd 3.12 in lecture note% Channel Input S_X = [0 1]; S_Y = [1 2 3];p_X = [0.2 0.8];% Channel CharacteristicsQ = [0.5 0.2 0.3; 0.3 0.4 0.3];
[DMC_demo.m]
2.
.
Ex: DMC
12
>> p = [0.2 0.8]p =
0.2000 0.8000>> p = [0.2 0.8];>> Q = [0.75 0 0.25; 0.3125 0.4375 0.25];>> p*Qans =
0.4000 0.3500 0.2500
Block Matrix Multiplications
13
10 6 6 4 39 7 3 5 9
2 2 5 10 2 10 2 53 3 4 5 10 5 3 63 3 4 1 1 5 5 67 2 5 3 10 6 10 38 3 6 9 8 3 6 5
108 73 136 175 150 193 126 149155 85 164 224 213 197 158 165
A BC D
E F
AC+BE AD+BF
10 6 6 4 39 7 3 5 9
2 2 5 10 2 10 2 53 3 4 5 10 5 3 63 3 4 1 1 5 5 67 2 5 3 10 6 10 38 3 6 9 8 3 6 5
108 73 136 175 150 193 126 149155 85 164 224 213 197 158 165
X G H
XG XH
Review: Evaluation of Probability from the Joint PMF Matrix
14
Consider two random variables X and Y.
Suppose their joint pmf matrix is
Find
0.1 0.1 0 0 0 0.1 0 0 0.1 00 0.1 0.2 0 00 0 0 0 0.3
2 3 4 5 6xy
1346
3 4 5 6 75 6 7 8 96 7 8 9 108 9 10 11 12
2 3 4 5 6xy
1346
Step 1: Find the pairs (x,y) that satisfy the condition“x+y < 7”
One way to do this is to first construct the matrix of x+y.
,X YP
x y
Review: Evaluation of Probability from the Joint PMF Matrix
15
Consider two random variables X and Y.
Suppose their joint pmf matrix is
Find
0.1 0.1 0 0 0 0.1 0 0 0.1 00 0.1 0.2 0 00 0 0 0 0.3
2 3 4 5 6xy
1346
3 4 5 6 75 6 7 8 96 7 8 9 108 9 10 11 12
2 3 4 5 6xy
1346
Step 2: Add the corresponding probabilities from the joint pmf (matrix)
,X YP
x y7 0.1 0.1 0.1
0.3
Review: Evaluation of Probability from the Joint PMF Matrix
16
Consider two random variables X and Y.
Suppose their joint pmf matrix is
Find
0.1 0.1 0 0 0 0.1 0 0 0.1 00 0.1 0.2 0 00 0 0 0 0.3
2 3 4 5 6xy
1346
,X YP
Review: Sum of two discrete RVs
17
Consider two random variables X and Y.
Suppose their joint pmf matrix is
Find
0.1 0.1 0 0 00.1 0 0 0.1 00 0.1 0.2 0 00 0 0 0 0.3
2 3 4 5 6xy
1346
3 4 5 6 75 6 7 8 96 7 8 9 108 9 10 11 12
2 3 4 5 6xy
1346
,X YP
x y7 0.1
Ex: DMC
18
>> p = [0.2 0.8];>> Q = [0.5 0.2 0.3; 0.3 0.4 0.3];>> p*Qans =
0.3400 0.3600 0.3000>> P = (diag(p))*QP =
0.1000 0.0400 0.06000.2400 0.3200 0.2400
>> sum(P)ans =
0.3400 0.3600 0.3000
for Naïve Decoder
20 [DMC_Analysis_demo.m]
%% Naive Decoderx_hat = y;
%% Error ProbabilityPE_sim = 1-sum(x==x_hat)/n % Error probability from the simulation
% Calculation of the theoretical error probabilityPC = 0;for k = 1:length(S_X)
t = S_X(k);i = find(S_Y == t);if length(i) == 1
PC = PC+ p_X(k)*Q(k,i);end
endPE_theretical = 1-PC
Formula derived in 3.19 of lecture notes
Ex: Naïve Decoder and BAC
21
p_X =0.5000 0.5000
p_X_sim =0.7000 0.3000
q =0.5500 0.4500
q_sim =0.6500 0.3500
Q =0.7000 0.30000.4000 0.6000
Q_sim =0.7143 0.28570.5000 0.5000
PE_sim =0.3500
PE_theretical =0.3500
>> BAC_demoans =0 0 0 1 1 0 0 1 0 0 0 0 1 0 0 1 0 1 0 0ans =0 0 1 1 0 0 0 1 1 1 0 0 1 0 0 0 0 0 1 0
x:
y:
0
1
0
1
0.3
0.7
0.4
0.6
X Y
%% Simulation parameters% The number of symbols to be transmittedn = 20; % Binary Assymmetric Channel (BAC)% Ex 3.8 in lecture note (11.3 in [Z&T, 2010])% Channel Input S_X = [0 1]; S_Y = [0 1];p_X = [0.5 0.5];% Channel CharacteristicsQ = [0.7 0.3; 0.4 0.6];
[BAC_demo.m]
.
.
720
[Ex. 3.18]
22
p_X =0.5000 0.5000
p_X_sim =0.5043 0.4957
q =0.5500 0.4500
q_sim =0.5532 0.4468
Q =0.7000 0.30000.4000 0.6000
Q_sim =0.7109 0.28910.3928 0.6072
PE_sim =0.3405
PE_theretical =0.3500
0
1
0
1
0.3
0.7
0.4
0.6
X Y
%% Simulation parameters% The number of symbols to be transmittedn = 1e4; % Binary Assymmetric Channel (BAC)% Ex 3.8 in lecture note (11.3 in [Z&T, 2010])% Channel Input S_X = [0 1]; S_Y = [0 1];p_X = [0.5 0.5];% Channel CharacteristicsQ = [0.7 0.3; 0.4 0.6];
.
.
10 20 30 40 50 60 70 80 90 100
10
20
30
40
50
60
70
80
90
10010 20 30 40 50 60 70 80 90 100
10
20
30
40
50
60
70
80
90
100
[BAC_demo.m]
Ex: Naïve Decoder and BAC [Ex. 3.18]
Ex: Naïve Decoder and DMC
23
p_X =0.2000 0.8000
p_X_sim =0.2000 0.8000
q =0.3400 0.3600 0.3000
q_sim =0.4000 0.3500 0.2500
Q =0.5000 0.2000 0.30000.3000 0.4000 0.3000
Q_sim =0.7500 0 0.25000.3125 0.4375 0.2500
PE_sim =0.7500
PE_theretical =0.7600
>> DMC_demoans =1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 0 1 1 0 1ans =1 3 2 2 1 2 1 2 2 3 1 1 1 3 1 3 2 3 1 2
x:
y:
0
1
1
3
0.5
0.3
X Y
%% Simulation parameters% The number of symbols to be transmittedn = 20; % General DMC% Ex. 3.16 in lecture note% Channel Input S_X = [0 1]; S_Y = [1 2 3];p_X = [0.2 0.8];% Channel CharacteristicsQ = [0.5 0.2 0.3; 0.3 0.4 0.3];
2.
.
20 420
[Same samples as in Ex. 3.6]
[Ex. 3.21]
[DMC_demo.m]
24
p_X =0.2000 0.8000
p_X_sim =0.2011 0.7989
q =0.3400 0.3600 0.3000
q_sim =0.3387 0.3607 0.3006
Q =0.5000 0.2000 0.30000.3000 0.4000 0.3000
Q_sim =0.4943 0.1914 0.31430.2995 0.4033 0.2972
PE_sim =0.7607
PE_theretical =0.7600
0
1
1
3
0.5
0.3
X Y
%% Simulation parameters% The number of symbols to be transmittedn = 1e4; % General DMC% Ex. 3.16 in lecture note% Channel Input S_X = [0 1]; S_Y = [1 2 3];p_X = [0.2 0.8];% Channel CharacteristicsQ = [0.5 0.2 0.3; 0.3 0.4 0.3];
2.
.
10 20 30 40 50 60 70 80 90 100
10
20
30
40
50
60
70
80
90
10010 20 30 40 50 60 70 80 90 100
10
20
30
40
50
60
70
80
90
100
[DMC_demo.m]
Ex: Naïve Decoder and DMC [Ex. 3.21]
DIY Decoder
25
>> DMC_decoder_DIY_demoans =1 0 1 1 1 1 1 0 1 1 0 1 1 1 1 0 0 1 0 1ans =2 1 1 3 3 1 2 2 1 2 1 2 3 1 1 3 1 3 1 1ans =1 0 0 0 0 0 1 1 0 1 0 1 0 0 0 0 0 0 0 0PE_sim =
0.5500PE_theretical =
0.5200Elapsed time is 0.081161 seconds.
%% Simulation parameters% The number of symbols to be transmittedn = 20; % General DMC% Ex. 3.16 in lecture note% Channel Input S_X = [0 1]; S_Y = [1 2 3];p_X = [0.2 0.8];% Channel CharacteristicsQ = [0.5 0.2 0.3; 0.3 0.4 0.3];
%% DIY DecoderDecoder_Table = [0 1 0]; % The decoded values corresponding to the received Y
[DMC_decoder_DIY_demo.m]
X
Y
[Ex. 3.22]
DIY Decoder
26
%% DIY DecoderDecoder_Table = [0 1 0]; % The decoded values corresponding to the received Y
% Decode according to the decoder tablex_hat = y; % preallocationfor k = 1:length(S_Y)
I = (y==S_Y(k));x_hat(I) = Decoder_Table(k);
end
PE_sim = 1-sum(x==x_hat)/n % Error probability from the simulation
% Calculation of the theoretical error probabilityPC = 0;for k = 1:length(S_X)
I = (Decoder_Table == S_X(k));q = Q(k,:); PC = PC+ p_X(k)*sum(q(I));
endPE_theretical = 1-PC
[DMC_decoder_DIY_demo.m]
[Ex. 3.22]
DIY Decoder
27
>> DMC_decoder_DIY_demoPE_sim =
0.5213PE_theretical =
0.5200Elapsed time is 2.154024 seconds.
%% Simulation parameters% The number of symbols to be transmittedn = 1e4; % General DMC% Ex. 3.16 in lecture note% Channel Input S_X = [0 1]; S_Y = [1 2 3];p_X = [0.2 0.8];% Channel CharacteristicsQ = [0.5 0.2 0.3; 0.3 0.4 0.3];
%% DIY DecoderDecoder_Table = [0 1 0]; % The decoded values corresponding to the received Y
10 20 30 40 50 60 70 80 90 100
10
20
30
40
50
60
70
80
90
10010 20 30 40 50 60 70 80 90 100
10
20
30
40
50
60
70
80
90
10010 20 30 40 50 60 70 80 90 100
10
20
30
40
50
60
70
80
90
100
[DMC_decoder_DIY_demo.m]
[Ex. 3.22]
28
Digital Communication SystemsECS 452
Asst. Prof. Dr. Prapun [email protected]
3.3 Optimal Decoder
Searching for the Optimal Detector
29
>> DMC_decoder_ALL_demoans =
0 0 0 0.80000 0 1.0000 0.62000 1.0000 0 0.52000 1.0000 1.0000 0.3400
1.0000 0 0 0.66001.0000 0 1.0000 0.48001.0000 1.0000 0 0.38001.0000 1.0000 1.0000 0.2000
Min_PE =0.2000
Optimal_Detector =1 1 1
Elapsed time is 0.003351 seconds.
1 2 3
Ex. 3.22
Ex. 3.23
Review: ECS315 (2016)
30
Guessing Game 1
31
There are 15 cards. Each have a number on it. Here are the 15 cards:
1 2 2 3 3 3 4 4 4 4 5 5 5 5 5 One card is randomly selected from the 15 cards.
You need to guess the number on the card.
Have to pay 1 Baht for incorrect guess.
The game is to be repeated n = 10,000 times.
What should be your guess value?
32
close all; clear all;
n = 5; % number of time to play this game
D = [1 2 2 3 3 3 4 4 4 4 5 5 5 5 5];X = D(randi(length(D),1,n));
if n <= 10X
end
g = 1cost = sum(X ~= g)
if n > 1averageCostPerGame = cost/nend
>> GuessingGame_4_1_1X =
3 5 1 2 5g =
1cost =
4averageCostPerGame =
0.8000
33
close all; clear all;
n = 5; % number of time to play this game
D = [1 2 2 3 3 3 4 4 4 4 5 5 5 5 5];X = D(randi(length(D),1,n));
if n <= 10X
end
g = 3.3cost = sum(X ~= g)
if n > 1averageCostPerGame = cost/nend
>> GuessingGame_4_1_1X =
5 3 2 4 1g =
3.3000cost =
5averageCostPerGame =
1
34
close all; clear all;
n = 1e4; % number of time to play this game
D = [1 2 2 3 3 3 4 4 4 4 5 5 5 5 5];X = D(randi(length(D),1,n));
if n <= 10X
end
g = ?cost = sum(X ~= g)
if n > 1averageCostPerGame = cost/nend
Guessing Game 1
351 1.5 2 2.5 3 3.5 4 4.5 5
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
Guess value
AV
erag
e C
ost P
er G
ame
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 60.65
0.7
0.75
0.8
0.85
0.9
0.95
1
Guess value
AV
erag
e C
ost P
er G
ame
Guessing Game 1
36
Optimal Guess: The most-likely value
MAP Decoder
37
%% MAP DecoderP = diag(p_X)*Q; % Weight the channel transition probability by the
% corresponding prior probability.[V I] = max(P); % For I, the default MATLAB behavior is that when there are
% multiple max, the index of the first one is returned.Decoder_Table = S_X(I) % The decoded values corresponding to the received Y
%% Decode according to the decoder tablex_hat = y; % preallocationfor k = 1:length(S_Y)
I = (y==S_Y(k));x_hat(I) = Decoder_Table(k);
end
PE_sim = 1-sum(x==x_hat)/n % Error probability from the simulation
%% Calculation of the theoretical error probabilityPC = 0;for k = 1:length(S_X)
I = (Decoder_Table == S_X(k));Q_row = Q(k,:); PC = PC+ p_X(k)*sum(Q_row(I));
endPE_theretical = 1-PC
[DMC_decoder_MAP_demo.m]
ML Decoder
38
%% ML Decoder[V I] = max(Q); % For I, the default MATLAB behavior is that when there are
% multiple max, the index of the first one is returned.Decoder_Table = S_X(I) % The decoded values corresponding to the received Y
%% Decode according to the decoder tablex_hat = y; % preallocationfor k = 1:length(S_Y)
I = (y==S_Y(k));x_hat(I) = Decoder_Table(k);
end
PE_sim = 1-sum(x==x_hat)/n % Error probability from the simulation
%% Calculation of the theoretical error probabilityPC = 0;for k = 1:length(S_X)
I = (Decoder_Table == S_X(k));Q_row = Q(k,:); PC = PC+ p_X(k)*sum(Q_row(I));
endPE_theretical = 1-PC
[DMC_decoder_ML_demo.m]
1
Digital Communication SystemsECS 452
Asst. Prof. Dr. Prapun [email protected]
4. Mutual Information and Channel Capacity
Office Hours: BKD, 6th floor of Sirindhralai building
Tuesday 14:20-15:20Wednesday 14:20-15:20Friday 9:15-10:15
Reference for this chapter
2
Elements of Information Theory
By Thomas M. Cover and Joy A. Thomas
2nd Edition (Wiley)
Chapters 2, 7, and 8
1st Edition available at SIIT library: Q360 C68 1991
Asst. Prof. Dr. Prapun [email protected]
Operational Channel Capacity
3
Digital Communication SystemsECS 452
Asst. Prof. Dr. Prapun [email protected]
Information Channel Capacity
4
Digital Communication SystemsECS 452
Channel Capacity
5
Channel Capacity
“Operational”: max rate at which reliablecommunication is possible
“Information”: [bpcu]
Arbitrarily small error probability can be achieved.
Shannon [1948] shows that these two quantities are actually the same.
MATLAB
6
function H = entropy2s(p)% ENTROPY2 accepts probability mass function % as a row vector, calculate the corresponding % entropy in bits.p=p(find(abs(sort(p)-1)>1e-8)); % Eliminate 1p=p(find(abs(p)>1e-8)); % Eliminate 0if length(p)==0
H = 0;else
H = simplify(-sum(p.*log(p))/log(sym(2)));end
function I = informations(p,Q)X = length(p);q = p*Q;HY = entropy2s(q);temp = [];for i = 1:X
temp = [temp entropy2s(Q(i,:))];endHYgX = sum(p.*temp);I = HY-HYgX;
Capacity calculation for BAC
7
Capacity of 0.0918 bits is achieved by 0.5380, 0.4620p
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
p0I(X
;Y)
0
1
0
1
0.9
0.1
0.4
0.6
X Y
0.1 0.90.4 0.6
Q
Capacity calculation for BAC
8
close all; clear all;syms p0p = [p0 1-p0];Q = [1 9; 4 6]/sym(10);
I = simplify(informations(p,Q))
p0o = simplify(solve(diff(I)==0))
po = eval([p0o 1-p0o])
C = simplify(subs(I,p0,p0o))
eval(C)
>> Capacity_Ex_BACI =(log(2/5 - (3*p0)/10)*((3*p0)/10 - 2/5) - log((3*p0)/10 + 3/5)*((3*p0)/10 +
3/5))/log(2) + (log((5*2^(3/5)*3^(2/5))/6)*(p0 - 1))/log(2) +
(p0*log((3*3^(4/5))/10))/log(2)
p0o =(27648*2^(1/3))/109565 - (69984*2^(2/3))/109565 + 135164/109565
po =0.5376 0.4624
C =(log((3*3^(4/5))/10)*((27648*2^(1/3))/109565 - (69984*2^(2/3))/109565 + 135164/109565))/log(2) - (log((104976*2^(2/3))/547825 - (41472*2^(1/3))/547825 + 16384/547825)*((104976*2^(2/3))/547825 - (41472*2^(1/3))/547825 + 16384/547825) + log((41472*2^(1/3))/547825 - (104976*2^(2/3))/547825 + 531441/547825)*((41472*2^(1/3))/547825 - (104976*2^(2/3))/547825 + 531441/547825))/log(2) + (log((5*2^(3/5)*3^(2/5))/6)*((27648*2^(1/3))/109565 -(69984*2^(2/3))/109565 + 25599/109565))/log(2)
ans =0.0918
0
1
0
1
0.9
0.1
0.4
0.6
X Y0.1 0.90.4 0.6
Q
Same procedure applied to BSC
9
close all; clear all;syms p0p = [p0 1-p0];Q = [6 4; 4 6]/sym(10);
I = simplify(informations(p,Q))
p0o = simplify(solve(diff(I)==0))
po = eval([p0o 1-p0o])
C = simplify(subs(I,p0,p0o))
eval(C)
>> Capacity_Ex_BSCI =(log((5*2^(3/5)*3^(2/5))/6)*(p0 - 1))/log(2) -(p0*log((5*2^(3/5)*3^(2/5))/6))/log(2) - (log(p0/5 + 2/5)*(p0/5 + 2/5) - log(3/5 - p0/5)*(p0/5 -3/5))/log(2)p0o =1/2po =
0.5000 0.5000C =log((2*2^(2/5)*3^(3/5))/5)/log(2)ans =
0.0290
0
1
0
1
0.4
0.6
0.4
0.6
X Y0.6 0.40.4 0.6
Q
Blahut–Arimoto algorithm
10
function [ps C] = capacity_blahut(Q)% Input: Q = channel transition probability matrix% Output: C = channel capacity% ps = row vector containing pmf that achieves capacity
tl = 1e-8; % tolerance (for the stopping condition)n = 1000; % max number of iterations (in case the stopping condition
% is "never" reached") nx = size(Q,1); pT = ones(1,nx)/nx; % First, guess uniform X.for k = 1:n
qT = pT*Q;% Eliminate the case with 0% Column-division by qTtemp = Q.*(ones(nx,1)*(1./qT));%Eliminate the case of 0/0l2 = log2(temp); l2(find(isnan(l2) | (l2==-inf) | (l2==inf)))=0;logc = (sum(Q.*(l2),2))';CT = 2.^(logc);A = log2(sum(pT.*CT)); B = log2(max(CT));if((B-A)<tl)
breakend% For the next looppT = pT.*CT; % un-normalizedpT = pT/sum(pT); % normalizedif(k == n)
fprintf('\nNot converge within n loops\n')end
endps = pT;C = (A+B)/2; [capacity_blahut.m]
Capacity calculation for BAC: a revisit
11
close all; clear all;
Q = [1 9; 4 6]/10;
[ps C] = capacity_blahut(Q)
>> Capacity_Ex_BAC_blahutps =
0.5376 0.4624C =
0.0918
0
1
0
1
0.9
0.1
0.4
0.6
X Y0.1 0.90.4 0.6
Q
Berger plaque
12
Richard Blahut
13
Former chair of the Electrical and Computer Engineering Department at the University of Illinois at Urbana-Champaign
Best known for Blahut–Arimotoalgorithm (Iterative Calculation of C)
Raymond Yeung
14
BS, MEng and PhD degrees in electrical engineering from Cornell University in 1984, 1985, and 1988, respectively.