Exam #1 Reviewbazuinb/ECE3800/Exam1_Review.pdf1.3 Venn Diagrams 1.4 Random Variables 1.5 Basic Probability Rules 1.6 Probability Formalized 1.7 Simple Theorems 1.8 Compound Experiments

Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.

B.J. Bazuin, Spring 2020 1 of 32 ECE 3800

Exam #1 Review

What is on an exam? Read through the homework and class examples…

5 multipart questions. Points assigned based on complexity. (170 pts. Sp. 2019, 175 pts. Fa 2019)

Set theory based probability (switches - union, intersection, mutually exclusive events) Elementary probability determination (coin or coins flipped, die or dice rolled, etc.) Discrete probability pmf and CDF Expected Value Operator Moments, Variance, Moment Generating Function Selection or Failure related questions (combinatorial based questions)

Previous homework problem solutions as examples – Dr. Severance’s Skill Examples

Skills #1

Skills #2

Exam and homework like problems:

HW#1

HW#2

HW#3

HW#4

Read through previous exam ….

And now for a quick chapter review … the highlights …!



The Chapter Content: 1-4

1 PROBABILITY BASICS 1.1 What Is Probability? 1.2 Experiments, Outcomes, and Events 1.3 Venn Diagrams 1.4 Random Variables 1.5 Basic Probability Rules 1.6 Probability Formalized 1.7 Simple Theorems 1.8 Compound Experiments 1.9 Independence 1.10 Example: Can S Communicate With D?

1.10.1 List All Outcomes 1.10.2 Probability of a Union 1.10.3 Probability of the Complement

1.11 Example: Now Can S Communicate With D? 1.11.1 A Big Table 1.11.2 Break Into Pieces 1.11.3 Probability of the Complement

1.12 Computational Procedures

2 CONDITIONAL PROBABILITY 2.1 Definitions of Conditional Probability 2.2 Law of Total Probability and Bayes Theorem 2.3 Example: Urn Models 2.4 Example: A Binary Channel 2.5 Example: Drug Testing 2.6 Example: A Diamond Network

3 A LITTLE COMBINATORICS 3.1 Basics of Counting 3.2 Notes on Computation 3.3 Combinations and the Binomial Coefficients 3.4 The Binomial Theorem 3.5 Multinomial Coefficient and Theorem 3.6 The Birthday Paradox and Message Authentication 3.7 Hypergeometric Probabilities and Card Games



4 DISCRETE PROBABILITIES AND RANDOM VARIABLES 4.1 Discrete Random Variable and Probability Mass Functions 4.2 Cumulative Distribution Functions 4.3 Expected Values 4.4 Moment Generating Functions 4.5 Several Important Discrete PMFs

4.5.1 Uniform PMF 4.5.2 Geometric PMF 4.5.3 The Poisson Distribution

4.6 Gambling and Financial Decision Making3.4 The Binomial Theorem

Chapter 1: PROBABILITY BASICS

An understanding of Probability and Statistics is necessary in most if not all work related to science and engineering.

Statistics: the study of and the dealing with data.

Probability: the study of the likeliness of result, action or event occurring. Often based on prior knowledge or the statistics of similar or past events!

Definitions of Probability Experiment Possible Outcomes Trials Event Equally Likely Events/Outcomes Objects Attribute Sample Space With Replacement and Without Replacement



Probability as the Ratio of Favorable to Total Outcomes (Classical Theory)

N

NAr A

ArAN

limPr

Where APr is defined as the probability of event A.

1. 1Pr0 A 2. Pr(A)+Pr(B)+Pr(C)+ … =1, for mutually exclusive events 3. An impossible event, A, can be represented as 0Pr A .

4. A certain event, A, can be represented as 1Pr A .

Sets, Fields and Events Set Subset Space Null Set or Empty Set Venn Diagram Equality

ABandBAiffBA Sum or Union

ABBA

AAA

AA

SSA

ABifABA , Products or Intersection

ABBA

AAA

A

ASA

ABifBBA , Mutually Exclusive or Disjoint Sets

BA Complement



AA and SAA

S

S

AA

ABifBA ,

ABifBA , Differences

BAABABA

BBBA

AAA

AAAA

AA

SA

AAS Proofs of Set Algebra

DeMorgan’s Law

BABA

BABA

Union of two sets

GFGFGF PrPrPrPr



Equalities in Set Algebra

from: Robert M. Gray and Lee D. Davisson, An Introduction to Statistical Signal Processing, Cambridge University Press, 2004. Appendix A, Set Theory. Pdf file version found at http://www-ee.stanford.edu/~gray/sp.html

The Axiomatic Approach

For event A



1Pr0 A (Axiom 1 & 2, Theorem 1.3)

1Pr S (Axiom 2)

0Pr (Theroem 1.1)

Disjoint Sets

BABAthenBAIf PrPrPr,

Complement (complementary sets)

1PrPrPrPr, SAAAAthenAAIf

1Pr1Pr AA

Not a Disjoint Sets (solution)

BABABA PrPrPrPr

in general (three elements)

1

1

2

1

3

1

2

1

3

1

3

1321

Pr

Pr

PrPr

i ij jk kji

i ij ji

i i

EEE

EE

EEEE

Can you recognize a pattern …

“+”singles … “-“doubles … “+”triples … “-“quads … etc



Joint, Conditional, and Total Probabilities; Independence

BBABA Pr|PrPr , for 0Pr B

B

BABA

Pr

Pr|Pr

, for 0Pr B

Joint Probability

𝑃𝑟 𝐴|𝐵 𝑃𝑟 𝐴 when A follows B

Marginal Probabilities: Total Probability (for Ai disjoint sets)

nAAAAS 321

jiforAA ji ,

iii AABAB Pr|PrPr , for 0Pr iA

nn ABABABABAAAABSBB 321321

nn AABAABAABB Pr|PrPr|PrPr|PrPr 2211

Independence BAABBA PrPr,Pr,Pr

or BABA PrPrPr

Independence can be extended to more than two events, for example three, A, B, and C. The conditions for independence of three events is

BABA PrPrPr CBCB PrPrPr CACA PrPrPr

CBACBA PrPrPrPr

For multiple events, every set of events from n down must be verified. This implies that 12 nn equations must be verified for n independent events.

Bayes Theorem

B

AABBA ii

i Pr

Pr|Pr|Pr

, for 0Pr B

nn

iii AABAABAAB

AABBA

Pr|PrPr|PrPr|Pr

Pr|Pr|Pr

2211

BA,Pr

AABBBAABBA Pr|PrPr|Pr,Pr,Pr



Can S Communicate with D? (Switch problems)

Figure 1.4: Two-path Network

213 LLLDSComm , for all links independent!

32Pr pppDS

213213

213213

213

PrPrPrPrPrPr

PrPrPrPr

PrPr

LLLLLL

LLLLLL

LLLDS

32

213213 PrPrPrPrPrPrPr

ppp

pppppp

LLLLLLDS

Applying negative logic

DsDS Pr1Pr

213PrPr LLLDS

21213213 PrPrPrPrPrPrPrPr LLLLLLLLDS

2213 1111PrPrPr ppppLLLDS

2213 21221PrPrPr ppppLLLDS

322213 111PrPrPr pppppLLLDS

and

323211Pr1Pr ppppppDSDs



CBADSComm

CBACBCABACBA

CBACBCABACBA

CBADS

PrPrPrPrPrPrPrPrPrPrPrPr

PrPrPrPrPrPrPr

PrPr

Alternately consider

CBA

CBADS

PrPrPr

PrPr

Now consider unequal probabilities

Remember the initial derivation?

213 LLLDSComm , for all links independent!

213213

213213

213

PrPrPrPrPrPr

PrPrPrPr

PrPr

LLLLLL

LLLLLL

LLLDS

And finally,

213213

213213 PrPrPrPrPrPrPr

pppppp

LLLLLLDS



Experiment 1: A bag of marbles, draw 1

Experiment 2: A bag of marbles, draw 2

Experiment 3: A bag of marbles, draw 2 without replacement

Can you compute the probabilities related to bag of marbles questions?

Chapter 2: CONDITIONAL PROBABILITY

Defining the conditional probability of event A given that event B has occurred.

Using a Venn diagram, we know that B has occurred … then the probability that A has occurred given B must relate to the area of the intersection of A and B …


Therefore

B

BABA

Pr

Pr|Pr

, for 0Pr B

For elementary events,

B

BA

B

BABA

Pr

,Pr

Pr

Pr|Pr

, for 0Pr B

Special cases for BA , AB , and AB .



Special cases for BA , AB , and AB .

If A is a subset of B, then the conditional probability must be

B

A

B

BABA

Pr

Pr

Pr

Pr|Pr

, for BA

Therefore, it can be said that

AB

A

B

BABA Pr

Pr

Pr

Pr

Pr|Pr

, for BA

If B is a subset of A, then the conditional probability becomes

1

Pr

Pr

Pr

Pr|Pr

B

B

B

BABA , for AB

If A and B are mutually exclusive,

0

Pr

0

Pr

Pr|Pr

BB

BABA , for AB

Conditional probabilities are generally not symmetric!


AABBA Pr|PrPr , for 0Pr A

and

B

BABA

Pr

Pr|Pr

, for 0Pr B

A

BAAB

Pr

Pr|Pr

, for 0Pr A

A

BBA

A

B

B

BA

B

B

A

BAAB

Pr

Pr|Pr

Pr

Pr

Pr

Pr

Pr

Pr

Pr

Pr|Pr

Therefore

BAAB |Pr|Pr



Continuing Concepts

Conditional Probability

When the probability of an event depends upon prior events. If trials are performed without replacement and/or the initial conditions are not restored, you expect trial outcomes to be dependent on prior results or conditions.

ABA Pr|Pr when A follows B

The joint probability is.

AABBBAABBA Pr|PrPr|Pr,Pr,Pr

Applicable for objects that have multiple attributes and/or for trials performed without replacement.

Resistor Example: Joint and Conditional Probability

Assume we have a bunch of resistors (150) of various impedances and powers… Similar to old textbook problems (more realistic resistor values)

50 ohms 100 ohms 200 ohms Subtotal

¼ watt 40 20 10 70

½ watt 30 20 5 55

1 watt 10 10 5 25

Subtotal 80 50 20 150

Joint Probability (multiple dimensions)

Marginal Probability (for one of the dimensions only)

Conditional Probability (based on the condition, what is the probability)



A Priori and A Posteriori Probability (Sec. 2.2 Bayes Theorem)

The probabilities defined for the expected outcomes, iAPr , are referred to as a priori

probabilities (before the event). They describe the probability before the actual experiment or experimental results are known.

After an event has occurred, the outcome B is known. The probability of the event belonging to one of the expected outcomes can be defined as

BAi |Pr

or from before

BBAAABBA iiii Pr|PrPr|PrPr

B

AABBA ii

i Pr

Pr|Pr|Pr

, for 0Pr B

Using the concept of total probability

nn AABAABAABB Pr|PrPr|PrPr|PrPr 2211

We also have the following forms

nn

iii AABAABAAB

AABBA

Pr|PrPr|PrPr|Pr

Pr|Pr|Pr

2211

or

B

AAB

AAB

AABBA jj

n

i

jjj Pr

Pr|Pr

Pr|Pr

Pr|Pr|Pr

111

This probability is referred to as the a posteriori probability (after the event).

It is also referred to as Bayes Theorem.



2.6 Diamond Network

More Law of total Probability work (LTP)

The vertical link is a new consideration in out paths! By conditioning on the vertical link we can derive two separate problems and then find a general solution.

If we consider Link3 to be broken and connected, we can establish the total probability

𝑃𝑟 𝑆 → 𝐷 𝑃𝑟 𝑆 → 𝐷|𝐿 1 ∙ 𝑃𝑟 𝐿 1 𝑃𝑟 𝑆 → 𝐷|𝐿 0 ∙ 𝑃𝑟 𝐿 0

For equal probability of links

The diamond network solution then becomes

𝑃𝑟 𝑆 → 𝐷 𝑃𝑟 𝑆 → 𝐷|𝐿 1 ∙ 𝑃𝑟 𝐿 1 𝑃𝑟 𝑆 → 𝐷|𝐿 0 ∙ 𝑃𝑟 𝐿 0

𝑃𝑟 𝑆 → 𝐷 4 ∙ 𝑝 4 ∙ 𝑝 𝑝 ∙ 𝑝 2 ∙ 𝑝 𝑝 ∙ 1 𝑝

𝑃𝑟 𝑆 → 𝐷 4 ∙ 𝑝 4 ∙ 𝑝 𝑝 2 ∙ 𝑝 𝑝 2 ∙ 𝑝 𝑝

𝑃𝑟 𝑆 → 𝐷 2 ∙ 𝑝 2 ∙ 𝑝 5 ∙ 𝑝 2 ∙ 𝑝



Chapter 3: A LITTLE COMBINATORICS

(i) Sampling with replacement is the easy way …

Possible combinations rnnnnn

(ii) Sampling without replacement

Possible combinations !!

121rn

nrnnnn

Next considerations … how many ways can the r things be selected …

Possible selections !121 rrrr

Now we can consider the “unique” combinations

Unique combinations !!

!

rrn

n

lectionPossibleSe

mbinationPossibleCo

We have now defined an operator to determine unique values for “n choose r”

!!

!

rrn

n

r

nC n

r

also sometimes shown as !!

!,

rrn

nrnCCrn

Binomial Probability of Bernoulli Trials

The binomial distribution arises from the summation of Independent and Identically Distributed (IID) Bernoulli R.V.

𝑝 𝑘1 𝑝 𝑞, 𝑋 0, 𝑘 0

𝑝, 𝑋 1, 𝑘 1

Then, 𝑆 𝑋 𝑋 𝑋 ⋯ 𝑋

𝑝 𝑆 𝑘 𝑃𝑟 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒𝑠 𝑤𝑖𝑡ℎ 𝑘 1 𝑠 𝑎𝑛𝑑 𝑛 𝑘 0′𝑠. " "

As described previously, the number of sequences with exactly k ones in N trials is based on the combinatorial computation of the binomial coefficient.

𝑝 𝑆 𝑘 𝑝 𝑘𝑛𝑘 ∙ 𝑝 ∙ 𝑞



5-Card Draw Combinatorial

How many ways can 5 cards be drawn from a deck of 52 playing cards?

!!

!

rrn

n

r

nC n

r

!5!552

!52

5

522,598,960

3.1 Basics of Counting

1. Selections are ordered or in an arbitrary order.

2. Selections are made with replacement or not with replacement.

Generalized Formula:

a. Ordered, with replacement of n elements r time 𝑛 ∙ 𝑛 ∙ 𝑛 ⋯ 𝑛 𝑛

b. Ordered without replacement of n elements r times

𝑛 ∙ 𝑛 1 ∙ 𝑛 2 ⋯ 𝑛 𝑟 1𝑛!

𝑛 𝑟 !𝑛

where the textbook has defined new notations..

d. Unordered, without replacement 𝑛 ∙ 𝑛 1 ∙ 𝑛 2 ⋯ 𝑛 𝑟 1

𝑟!𝑛!

𝑛 𝑟 ! ∙ 𝑟!𝑛𝑟!

𝑛𝑟

c. Unordered, with replacement 𝑛 𝑟 1 !𝑛 1 ! ∙ 𝑟!

𝑛 𝑟 1𝑟



3.7 Hypergeometric Probabilities and Card Games

From: http://en.wikipedia.org/wiki/Hypergeometric_distribution

In probability theory and statistics, the hypergeometric distribution is a discrete probability distribution (probability mass function) that describes the number of successes in a sequence of n draws from a finite population without replacement.

A typical example is the following: There is a shipment of N objects in which D are defective. The hypergeometric distribution describes the probability that in a sample of n distinctive objects drawn from the shipment exactly x objects are defective.

n

N

xn

DN

x

D

nDNXx ,,,Pr

for DnxNnD ,min,0max

The equation is derived based on a non-replacement Bernoulli Trials …

Where the denominator term defines the number of trial possibilities, the 1st numerator term defines the number of ways to achieve the desired x, and the 2nd numerator term defines the filling of the remainder of the set.

The text extends this to multiple selections

𝑃𝑟 𝑘 , 2, ⋯ , 𝑘 ,

𝑛𝑘 ∙

𝑛𝑘 ∙ ⋯ ∙

𝑛𝑘

𝑛𝑘

where nj or subsets of n and kj selected from nj



Quality Control Example

A batch of 50 items contains 10 defective items. Suppose 10 items are selected at random and tested. What is the probability that exactly 5 of the items tested are defective?

The number of ways of selecting 10 items out of a batch of 50 is the number of combinations of size 10 from a set of 50 objects:

!40!10

!50

10

505010

C

The number of ways of selecting 5 defective and 5 nondefective items from the batch of 50 is the product N1 x N2 where N1 is the number of ways of selecting the 5 items from the set of 10 defective items, and N2 is the number of ways of selecting 5 items from the 40 nondefective items.

!35!5

!40

!5!5

!10405

105 CC

Thus the probability that exactly 5 tested items are defective is the desired ways the selection can be made divided by the total number of ways selection can be made, or

0.01614 01027227817

658008252

!40!10

!50!35!5

!40

!5!5

!10

5010

405

105

C

CC

Suppose 10 items are selected at random and tested. What is the probability that exactly 1 of the items tested are defective?

𝐶 ∙ 𝐶

𝐶

10!9! ∙ 1! ∙ 40!

31! ∙ 9!50!

40! ∙ 10!

10 ∙ 27343888010272278170

0.2662

nchoosek(10,1)*nchoosek(40,9)/nchoosek(50,10)



Textbook box with Marbles Example

A box contains 9 marbles (4 red and 5 blue). Six marbles are blindly selected from the box. What is the probability of two red marbles and four blue marbles?

𝐶 ∙ 𝐶𝐶

4!2! ∙ 2! ∙ 5!

4! ∙ 1!9!

6! ∙ 3!

6 ∙ 59 ∙ 8 ∙ 7/6

6 ∙ 53 ∙ 4 ∙ 7

514

0.3571

nchoosek(10,1)*nchoosek(40,9)/nchoosek(9,6)= 0.3571

Textbook Poker Full House in 5 cards

Draw 5 cards from a deck of 52. For s specific full house, assume 3 queens and 2 eights.

The probability becomes

𝐶 ∙ 𝐶

𝐶

4!3! ∙ 1! ∙ 4!

2! ∙ 2!52!

47! ∙ 5!

4 ∙ 4 ∙ 32

52 ∙ 51 ∙ 50 ∙ 49 ∙ 48120

𝐶 ∙ 𝐶

𝐶

242598960

1108290

9.2345𝑒 06

The number of possible unique full houses is … 13 3-card groups and 12 2 card groups for a total of 13 x 12 = 156.

Therefore the probability of being dealt a full house of any kind is

13 ∙ 12 ∙𝐶 ∙ 𝐶

𝐶

156108290

1.4406𝑒 03



Chapter 4: DISCRETE PROBABILITIES AND RANDOM VARIABLES

Probability Mass Functions (pmf)

From: http://en.wikipedia.org/wiki/Probability_mass_function

In probability theory, a probability mass function (abbreviated pmf) gives the probability that a discrete random variable is exactly equal to some value.

Cumulative Distribution Function (CDF)

Probability mass functions are related discrete countable outcomes of an experiment and the probability each outcomes has at the discrete values.

Generalized properties of pmf (from Stark and Wood)

Properties of the pmf include 1. xforxf X ,0

2. 1

u

X uf

3.

x

u

XX ufxF

4.

2

1

21Prx

xuX ufxXx

Generalized Properties of CDF (from Stark and Wood)

Cumulative Distribution Function (CDF):The probability of the event that the observed random variable X is less than or equal to the allowed value x.

xXxFX Pr

The defined function can be discrete or continuous along the x-axis. Constraints on the cumulative distribution function are:

xforxFX ,10

0XF and 1XF (property #1 and #3 in the textbook)

XF is non-decreasing as x increases (property #2 in the textbook)

1221Pr xFxFxXx XX



4.3 Expected Values, Moments, Central Moments and Variance

For random variables, the expected value operation produces a “probabilistic average” of the particular probability based function of interest.

𝐸 𝑔 𝑋 𝑔 𝑥 ∙ 𝑝 𝑘

https://en.wikipedia.org/wiki/Expected_value

where g(X) is a function of the random variable X and p(k) is the pmf. (Chapter 3 ROI calculations are expected values!)

Mean or 1st Moment

For example the 1st moment or mean value of the random variable is defined by

𝜇 ≡ 𝐸 𝑋 𝑋 𝑥 ∙ 𝑝 𝑘

The mean square value or second moment is

𝐸 𝑋 𝑋 𝑥 ∙ 𝑝 𝑘

Other “moments” ( the nth moment) are defined as

𝐸 𝑋 𝑋 𝑥 ∙ 𝑝 𝑘

Central Moments

𝐸 𝑋 𝜇 𝑋 𝜇 ∙ 𝑝 𝑘

The 2nd central moment or variance

𝜎 ≡ 𝐸 𝑋 𝜇 𝑋 𝜇 ∙ 𝑝 𝑘

The standard deviation is

𝜎 𝜎



Importance of mean and standard deviation

Often when we talk about values we say

𝜇 𝜎

That is to say that we expect the result x to be

𝜇 𝜎 𝑥 𝜇 𝜎

A useful variance – moments proof (Theorem 4.1)

𝜎 𝐸 𝑋 𝜇 𝑋 𝜇 ∙ 𝑝 𝑘

𝜎 𝐸 𝑋 𝜇 𝑋 2 ∙ 𝜇 ∙ 𝑋 𝜇 ∙ 𝑝 𝑘

𝜎 𝐸 𝑋 𝜇 𝑋 ∙ 𝑝 𝑘 2 ∙ 𝜇 ∙ 𝑋 ∙ 𝑝 𝑘 𝜇 ∙ 𝑝 𝑘

𝜎 𝐸 𝑋 𝜇 𝐸 𝑋 2 ∙ 𝜇 𝜇

𝜎 𝐸 𝑋 𝜇 𝐸 𝑋 𝜇

Note that the second moment is related to signal power/energy!

𝐸 𝑋 𝜇 𝜎



More basics about moments – the expected value operator

A constant, non-random variable

𝐸 𝑐 𝑐 ∙ 𝑝 𝑘 𝑐 ∙ 𝑝 𝑘 𝑐

A constant multiplication

𝐸 𝑐 ∙ 𝑔 𝑋 𝑐 ∙ 𝑔 𝑋 ∙ 𝑝 𝑘 𝑐 ∙ 𝑔 𝑋 ∙ 𝑝 𝑘 𝑐 ∙ 𝐸 𝑔 𝑋

Therefore

𝐸 𝑎 ∙ 𝑋 𝑏 𝑎 ∙ 𝑋 𝑏 ∙ 𝑝 𝑘 𝑏 𝑎 ∙ 𝑋 ∙ 𝑝 𝑘 𝑎 ∙ 𝜇 𝑏

Note: 𝐸 𝑌 𝐸 𝑎 ∙ 𝑋 𝑏 𝑎 ∙ 𝜇 𝑏 𝜇

Summations (superposition)

𝐸 𝑔 𝑋 𝑔 𝑋 𝑔 𝑋 𝑔 𝑋 ∙ 𝑝 𝑘 𝑔 𝑋 ∙ 𝑝 𝑘 𝑔 𝑋 ∙ 𝑝 𝑘

𝐸 𝑔 𝑋 𝑔 𝑋 𝐸 𝑔 𝑋 𝐸 𝑔 𝑋

Multiplication does not work!

Integration and differentiation

𝑑𝑑𝑣

𝐸 𝑔 𝑋, 𝑣𝑑

𝑑𝑣𝑔 𝑋, 𝑣 ∙ 𝑝 𝑘

𝑑𝑑𝑣

𝑔 𝑋, 𝑣 ∙ 𝑝 𝑘 𝐸𝑑

𝑑𝑣𝑔 𝑋, 𝑣

𝐸 𝑔 𝑋, 𝑣 ∙ 𝑑𝑣 𝑔 𝑋, 𝑣 ∙ 𝑝 𝑘 ∙ 𝑑𝑣 𝑔 𝑋, 𝑣 ∙ 𝑑𝑣 ∙ 𝑝 𝑘

𝐸 𝑔 𝑋, 𝑣 ∙ 𝑑𝑣 𝐸 𝑔 𝑋, 𝑣 ∙ 𝑑𝑣



Moment-Generating Functions

The moment generation function (MGF) is the two sided Laplace transform of the probability mass function (pmf) or the probability density function (pdf). If the MGF exists, there is a forward and inverse relationship between the MGF and the pmf/pdf. The MGF is defined based on the expected value as

𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋

𝑀 𝑠 𝑒𝑥𝑝 𝑠 ∙ 𝑋 ∙ 𝑝 𝑘

Why do we do this?

1. It enables a convenient computation of the higher order moments

2. It can be used to estimate fx(x) from experimental measurements of the moments

3. It can be used to solve problems involving the computation of the sums of R.V.

4. It is an important analytical instrument that can be used to demonstrate results and establish additional bounds (the Chernoff Bound and the Central Limit Theorem).

Determining the moments

Perform the Taylor series expansion of the exponential

𝑒𝑥𝑝 𝑥 1𝑥1!

𝑥2!

⋯𝑥𝑛!

⋯

𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋 𝐸 1𝑠 ∙ 𝑋

1!𝑠 ∙ 𝑋

2!⋯

𝑠 ∙ 𝑋𝑛!

⋯

or

𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋 1𝑠 ∙ 𝑚

1!𝑠 ∙ 𝑚

2!⋯

𝑠 ∙ 𝑚𝑛!

⋯

The mi are the ith moments of the density function!

So how would we solve for the moments? By taking derivatives and setting s=0!

Taking the 1st derivative … 𝜕

𝜕𝑠𝑀 𝑠

𝜕𝜕𝑠

𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋 0𝑚1!

2 ∙ 𝑠 ∙ 𝑚2!

⋯𝑛 ∙ 𝑠 ∙ 𝑚

𝑛!⋯



Setting s=0

𝜕𝜕𝑠

𝑀 𝑠 0𝑚1!

0 ⋯ 0 ⋯𝑚1!

Taking the nth derivative …

𝜕𝜕𝑠

𝑀 𝑠 0 ⋯ 0𝑛! ∙ 𝑚

𝑛!𝑛! ∙ 𝑠 ∙ 𝑚

𝑛 1!⋯

Setting s=0

𝜕𝜕𝑠

𝑀 𝑠 0 ⋯ 0𝑛! ∙ 𝑚

𝑛!0 ⋯

𝑛! ∙ 𝑚𝑛!

Therefore, all moments can be determined if the moment generation function exists!

If given a simple pmf, can you provide the exponential terms that form the MGF?!

𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋

𝑀 𝑠 𝑒𝑥𝑝 𝑠 ∙ 𝑋 ∙ 𝑝 𝑘



Chebyshev Inequality Textbook description

Defining a bound based on the variance of a random variable.

𝑃𝑟 |𝑋 𝜇 | 𝜖𝑉𝑎𝑟 𝑋

𝜖

For the Chebyshev Inequality the region of interested is defined based on the equation

𝐴 |𝑋 𝜇 | 𝜖

Note that the Chebyshev Inequality is a bound that applies in all cases. There is no judgment or determination if it is a good or even useful bound!

Note that for any R.V. where the variance tends to zero, you would have

𝑃𝑟 |𝑋 𝜇 | 𝜖 → 0

and the “zero variance random variable must equal the mean value!

Extension, relating epsilon to sigma …

𝜖 𝑘 ∙ 𝜎 with 𝑉𝑎𝑟 𝑋 𝜎

𝑃𝑟 |𝑋 𝜇 | 𝑘 ∙ 𝜎𝜎

𝑘 ∙ 𝜎

𝑃𝑟 |𝑋 𝜇 | 𝑘 ∙ 𝜎1

𝑘

Considering the negative of the probability function

11

𝑘1 𝑃𝑟 |𝑋 𝜇 | 𝑘 ∙ 𝜎

or

11

𝑘𝑃𝑟 |𝑋 𝜇 | 𝑘 ∙ 𝜎



Discrete Uniform pmf

The same probability for m different values from 1 to m.

𝑃𝑟 𝑋 𝑘 𝑝 𝑘

0, 𝑘 11𝑚

, 1 𝑘 𝑚

0, 𝑚 𝑘

Computing the mean or first moment

𝐸 𝑋 𝜇 𝑘 ∙1𝑚

𝐸 𝑋 𝜇1𝑚

∙ 𝑘1𝑚

∙𝑚 ∙ 𝑚 1

2𝑚 1

2

Computing the second moment

𝐸 𝑋 𝑘 ∙1𝑚

𝐸 𝑋1𝑚

∙ 𝑘1𝑚

∙2 ∙ 𝑚 1 ∙ 𝑚 1 ∙ 𝑚

3 ∙ 22 ∙ 𝑚 1 ∙ 𝑚 1

6

Computing the variance or second central moment

𝐸 𝑋 𝜇 𝑉𝑎𝑟 𝑋 𝜎 𝐸 𝑋 𝜇

𝑉𝑎𝑟 𝑋 𝜎𝑚 1

12𝑚 1 ∙ 𝑚 1

12

Computing the Moment Generating Function

𝑀 𝑢 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙ 𝑝 𝑘 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙1𝑚

𝑀 𝑢1𝑚

∙ 𝑒𝑥𝑝 𝑢 ∙1 𝑒𝑥𝑝 𝑢 ∙ 𝑚

1 𝑒𝑥𝑝 𝑢1𝑚

∙𝑒𝑥𝑝 𝑢 ∙ 𝑚 1 𝑒𝑥𝑝 𝑢

𝑒𝑥𝑝 𝑢 1

Note that the computation of the moments is not straight forward and requires L’Hospital’s rule instead of direct derivation! (Note that at u=0 you always get 0/0.)



Binomial pmf

𝑃𝑟 𝑋 𝑘 𝑝 𝑘𝑛𝑘 ∙ 𝑝 ∙ 𝑞

Determine the expected value

𝐸 𝑋 𝜇 𝑘 ∙𝑛𝑘 ∙ 𝑝 ∙ 𝑞

𝐸 𝑋 𝜇 𝑛 ∙ 𝑝

Determine the 2nd moment

𝐸 𝑋 𝑘 ∙𝑛𝑘 ∙ 𝑝 ∙ 𝑞

𝐸 𝑋 𝑛 ∙ 𝑛 1 ∙ 𝑝 𝑛 ∙ 𝑝

Determine the variance


𝑉𝑎𝑟 𝑋 𝜎 𝑛 ∙ 𝑝 ∙ 𝑝 1 𝑛 ∙ 𝑝 ∙ 𝑞

Determine the Moment Generating Function

𝑀 𝑢 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙ 𝑝 𝑘 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙𝑛𝑘 ∙ 𝑝 ∙ 𝑞

nX qtptM exp



Geometric Distribution pmf

𝑃𝑟 𝑋 𝑘 𝑝 𝑘0, 𝑘 0

𝑝 ∙ 1 𝑝 , 𝑘 1,2, ⋯

The 1st Moment

𝐸 𝑋 𝜇 𝑘 ∙ 𝑝 ∙ 1 𝑝

𝐸 𝑋 𝜇 𝑝 ∙1

1 1 𝑝𝑝 ∙

1𝑝

1𝑝

The 2nd Moment

𝐸 𝑋 𝑘 ∙ 𝑝 ∙ 1 𝑝

𝐸 𝑋2 2 ∙ 𝑝

𝑝1𝑝

2 𝑝𝑝

The Variance


𝑉𝑎𝑟 𝑋 𝜎2 𝑝

𝑝1

𝑝1 𝑝

𝑝

The Moment Generating Function (MGF)

𝑀 𝑢 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙ 𝑝 𝑘 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙ 𝑝 ∙ 1 𝑝

𝑀 𝑢𝑝 ∙ 𝑒𝑥𝑝 𝑢

1 𝑒𝑥𝑝 𝑢 ∙ 1 𝑝𝑝

𝑒𝑥𝑝 𝑢 1 𝑝



Textbook Poisson Distribution (4.5.2)

𝑃𝑟 𝑋 𝑘 𝑝 𝑘0, 𝑘 0

𝜆𝑘!

∙ 𝑒𝑥𝑝 𝜆 , 𝑘 0,1,2, ⋯

The 1st Moment

𝐸 𝑋 𝜇 𝑘 ∙𝜆𝑘!

∙ 𝑒𝑥𝑝 𝜆

𝐸 𝑋 𝜇 𝜆

The 2nd Moment

𝐸 𝑋 𝑘 ∙𝜆𝑘!


𝐸 𝑋 𝜆 𝜆

The Variance


𝑉𝑎𝑟 𝑋 𝜎 𝜆 𝜆 𝜆 𝜆

The Moment Generating Function (MGF)

𝑀 𝑢 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙ 𝑝 𝑘 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙𝜆𝑘!


𝑀 𝑢 𝑒𝑥𝑝 𝜆 ∙ 𝑒𝑥𝑝 𝜆 ∙ 𝑒𝑥𝑝 𝑢 𝑒𝑥𝑝 𝜆 ∙ 𝑒𝑥𝑝 𝑢 1



Risk Taking Decision Making (A function of the random variable)

𝑌 𝑔 𝑋1, 𝑃𝑟 𝑋 0 1 𝑝𝑤, 𝑃𝑟 𝑋 1 𝑝

Examples:

Win-lose propositions (X=1 win, X=0 lose) Gambling bets with odds Buying a stock as an investment

The 1st Moment

𝐸 𝑌 𝜇 𝑔 𝑋 ∙ 𝑃𝑟 𝑋

𝐸 𝑌 𝜇 1 ∙ 1 𝑝 𝑤 ∙ 𝑝

𝐸 𝑌 𝜇 𝑤 ∙ 𝑝 𝑝 1

The 2nd Moment

𝐸 𝑌 𝑔 𝑋 ∙ 𝑃𝑟 𝑋

𝐸 𝑌 𝑤 ∙ 𝑝 1 𝑝

The Variance

𝐸 𝑌 𝜇 𝑉𝑎𝑟 𝑌 𝜎 𝐸 𝑌 𝜇

𝑉𝑎𝑟 𝑌 𝜎 𝑝 ∙ 1 𝑝 ∙ 𝑤 1

To have a chance at winning ….

𝐸 𝑌 𝜇 0

𝑤 or 𝑝

Notice that the variance goes up by the square of the amount wagered!

Documents

Exam #1 Reviewbazuinb/ECE3800/Exam1_Review.pdf1.3 Venn Diagrams 1.4 Random Variables 1.5 Basic Probability Rules 1.6 Probability Formalized 1.7 Simple Theorems 1.8 Compound Experiments