Multiple Discrete Random Variables

Introduction• Consider the choice of a student at random from a population of

college students. We wish to know his/her height, weight, blood pressure, pulse rate, etc.

• The mapping from sample space of students to measurements of height and weight, would be H(si) = hi, W(si) = wi, of the student selected.

• The table is a two-dimensional array that lists the probability P[H = hi and W =wj].

Introduction• The information can also be displayed in a three-dimensional format.

• These probabilities were termed join probabilities. The height and weight could be represented as 2 x 1 random vector.

• We will study dependencies between the multiple RV. For example: “Can we predict a person’s height from his weight?”

Jointly Distributed RVs• Consider two discrete RVs X and Y. They represent the functions that

map an outcome of an experiment si to a value in the plane.

for all

• The experiment consists of the simultaneous tossing of a penny and a nickel.

• Two random variable that are defined on the same sample space S are said to be jointly distributed.

Jointly Distributed RVs• There are four vectors that comprise the sample space

• The values of the random vector (multiple random variables) are denoted either by (x,y) which is an ordered pair/point in the plane or [x y]T a 2D vector.

• The size of the sample space for discrete RV can be finite of countably infinite.

• If X can take on 2 values Nx = 2, and Y can take on 2 values NY =2, the total number of elements in SX,Y is NXNY = 4.

• Generally, if SX = {x1, x2,…,xNx} and SY = {y1, y2,…,yNy}, then the random vector can take on values in

Jointly Distributed RVs• The notation A × B, where A and B are sets, denotes a Cartesian

product set.

• The joint PMF (bivariate PMF) as

Properties of joint PMF• Property 1. Range of values of joint PMF

• Property 2. Sum of values of joint PMF

Similarly for a countably infinite sample space.

• For two fair coins that do not interact as they are tossed we might assign pX,Y[i,j] = ¼.

The procedure to determine the joint PMF from the probabilities defined on S• The procedure depends on whether the RV mapping is one-to-

one or many-to-one. • For a one-to-one mapping from S to SX,Y we have

It is assumed that sk is the only solution to X(s) = xi and X(s) = yj.• For a many-to-one transformation the joint PMF is found as

Two dice toss with different colored dice• A red die and a blue die are tossed. The die that yields the

larger number of dots is chosen.• If both dice display the same number of dots, the red die is

chosen. The numerical outcome of the experiment is defined to be 0 if the blue die is chosen and 1 if the red die is chosen, along with its corresponding number of dots.

• What is pX,Y[1,3] for example?

Two dice toss with different colored dice• To determine the desired value of the PMF, we assume that each

outcome in S is equally likely and therefore is equal to 1/36.

• Since there are three outcomes that map into (1,3).

• In general, we can use the joint PMF, to find probability of event A defined on SX,Y = SX × SY.

Marginal PMFs and CDFs• If pX,Y[x,y] is known, then marginal probabilities pX[xi] and pY[yi]

can be determined. • Consider the general case find calculating the probability of an

event of interest A on countably infinite sample space .

• Let A = {xk} × SY. Then,

with i = k only

with j = k only

Example: Two coin toss• A penny (RV X) and a nickel (RV Y) are tossed and the outcomes

are mapped into a 1 for a head and a 0 for a tail. Consider the joint PMF

• The marginal PMFs are given as

=1

=1

Joint PMF cannot be determined from marginal PMFs• It is not possible in general to obtain joint PMF from marginal

PMFs.• Consider the following joint PMF

The marginal PMFs are the same as the ones before.• There are an infinite number of joint PMFs that have the same

marginal PMFs.

joint PMF marginal PMFs

marginal PMFs joint PMF

Joint cumulative distribution function • A joint cumulative distribution function (CDF) can be defined

for a random vector as

• and can be found explicitly by summing the joint PMFs as

• The PMF can be recovered as

Properties of Cumulative distribution functions• The marginal CDFs can be easily found from the joint CDF as

• Property 1. Range of values

• Property 2. Values of “endpoints”

• Property 3. Monotonically increasingMonotonically increases as x and/or y increases.

• Property 4. “Right” continuous• The joint CDF takes the value after the jump.

Independence of Multiple RV• Consider the experiment of tossing a coin and then a die.• The outcome of the coin X = {0,1} and the outcome of a die

Y = {1,2,3,4,5,6} are independent, hence the probability of the random vector (X,Y) taking on a value Y = yi does not depend on X = xi.

• X and Y are independent random variables if all the joint events on SX,Y are independent.

• The probability of joint events may be reduced to probabilities of “marginal events”.

• If A = {xi} and B = {yj}, then

and

Independence of Multiple RV• If the joint PMF factors, then X and Y are independent. • To prove it, assume the joint PMF factors, the for all A and B

• Example: Two coin toss – independence• Assume we toss a penny and nickel. If all outcomes

are equivalently the joint PMF is given by

marginal probability

Independence of Multiple RV• Example: Two coin toss – dependence• Consider the same experiment but with a joint PMF given by

Then pX,Y[0,0] = 1/8 ≠ (1/4)(3/8) = pX[0]pY[0] and hence X and Y cannot be independent.

• If two random variables are not independent, they are said to be dependent.

Independence of Multiple RV• Example: Two coin toss – dependent but fair coins• Consider the same experiment again but with joint PMF given by

• Since pX,Y[0,0] = 3/8 ≠(1/2)(1/2), X and Y are dependent. However, by examining the marginal PMFs we see that the coins are in some sense fair since P[heads] = 1/2, thus we might conclude that the RVs were independent. This is incorrect.

• If the RVs are independent, the joint CDF factors as well.

Transformations of Multiple Random Variables• The PMF of Y = g(x) if the PMF of X is known is given by

• In the case of two discrete RVs X and Y that are transformed into W = g(X, Y) and Z = h(X, Y), we have

• Sometimes we wish to determine the PMF of Z = h(X, Y) only. Then we can use auxiliary RV W = X, so that pZ is the marginal PMF and can be found form the formula above as

Example: Independent Poisson RVs• Assume that the joint PMF is give as the product of the marginal

PMFs, and each PMF is Poisson PMF.

• Consider the transformation

• We need to determine all (k,l) so that

• But xk, yl and wi, zj can be replaced by k,l and i,j each with 0,1,….

Example: Independent Poisson RVs• Apply the given transformation we get

• Solving for (k, l) for the given (i, j), we have

• We must have l ≥ 0 so that l = j – I ≥ 0.

discrete unit step

Use the discrete unit step sequence to avoid mistakes• The discrete unit step sequence was introduced to designate

the region of w-z plane over which pW,Z[i,j] is nonzero.• The transformation will generally change the region over

which the new joint PMF is nonzero.• A common mistake is to disregard this region and assert that

the joint PMG is nonzero over i = 0,1,…; j = 0,1,…. • To avoid possible errors unit steps are applied

Example: Independent Poisson RVs• To find the PMF of Z = X + Y from the joint PMF obtained

earlier we set W = X so we have SW = SX = {0,1,…} and

• Since u[i] = 1 for I = 0,1,… and u[j - i] = 1 for i = 0,1,…,j and u[j - i] = 0 for I > j, we drop u[i]u[j – i] multipliers.

• Not that Z can take on values j = 0,1,… since Z = X + Y.

Connection to characteristic function

• Generally the formula for the PMF of the sum of any two discrete RV X and Y, dependent or independent is given by

• If the RV are independent, then since the joint PMF must factor, we have the result

• This summation is a discrete convolution. Taking the Fourier transformation (defined with a +j) of both sides produces

Example: Independent Poisson RVs using CF approach• We showed that if X ~ Pois(λ), then

• Thus using the above Fourier property we have

• But the CF in the braces is that of a Poisson RV and corresponds to

• The use of CF for determination of PMF for a sum of independent RV has considerably simplified the derivation.

• In summary, if X and Y are independent RV with integer values, then the PMF of Z = X + Y is given by

Transformation of a fine sample space• It is possible to obtain the PMF of Z = g(X,Y) by a direct calculation if

the sample SX,Y is finite.

• To first obtain the transformed joint PMF pw,z we 1. determine the finite sample space SZ.2. determine which sample points (xi,yj) in SX,Y map into each 3. sum the probabilities of those (xi,yj) sample points to yield pz[zk].

• Mathematically this is equivalent to

Direct computation of PMF for transformed RV• Consider the transformation of the RV (X,Y) into the scalar RV

Z = X2 + Y2. The joint PMF is given by

• To find the PMF for Z first note that (X,Y) takes on the values (i,j) = (0,0),(1,0),(0,1),(1,1). Therefore, Z must take on the values zk = i2 + j2 = 0,1,2. Then

Expected Values• If Z = g(X, Y), then by definition its expected value

Or using a more direct approach

EX. Expected value of a sum of random variables: Z = g(X,Y) = X + Y

Expected value of a product of RV

• If Z = g(X, Y) = XY, then

• If X and Y are independent, then since the joint PMF factors, we have

• More generally,

Variance of a sum of RVs• Consider the calculation of var(X + Y). Then, letting Z = g(X,Y) =

(X + Y – EX,Y[X + Y])2, we have

• The last term is called the covariance and defined as

• or alternatively

Joint moments• The questions of interest

• If the outcomes of one RV is a given value, what can we say about the outcome of the other RV?

• Will it be about the same or have the same have no relationship to other RV?

There is clearly a relationship between height and weight.

Joint moments

• To quantify these relationships we form the product XY, which can take on the values +1, -1, and ±1 for the joint PMF above.

• To determine the value of XY on the average we define the joint moment as EX,Y[XY].

For the case (a)

Joint moments

• In previous example EX[X] = EY[Y] = 0. If they are not zero, the joint moment will depend on the values of the means.

• To nullify this effect it is convenient to use the joint central moments.

That will produce the desired +1 for the joint PMF above.

Independence implies zero covariance but zero covariance does not imply independence

• For the joint PMF the covariance is zero since

• Consider the joint PMF which assigns equal probability ½ to each of the four points point.

and thus• However, X and Y are dependent because pX,Y[1,0] = 1/4 but

pX[1]pY[0]=(1/4)(1/2) = 1/8.

The joint k-lth moment

More generally the joint k-lth moment is defined as

For k = 1,2,…; l =1,2,…, when it exists. The joint k-lth moment central moment is defined as

For k = 1,2,…; l =1,2,…, when it exists.

Prediction of a RV outcome• The covariance between two RVs is useful for predicting Y

based on knowledge of the outcome of X.• We seek a predictor Y that is linear in X or

• The constants a and b are to be chose so that “on the average” the observed value of aX + b is close to the observed value of Y.

• The solution is given by

Prediction of a RV outcome• The the optimal linear prediction of Y given the outcome X = x is

• Example: Predicting one RV outcome from knowledge of second RV outcome.

We found from marginals

Regression line

Practice problems• Two coins are tossed in succession with a head being mapped

into a +1 and a tail mapped into a -1. If a RV is defined as (X,Y) with X representing the mapping of the first toss and Y representing the mapping of the second toss, draw the mapping. Also, what is SX,Y? (Hint: see slides 4,5).

• Two dice are tossed. The number of dots observed on the dice are added together to form the random variable X and also difference to form Y. Determine the possible outcomes of the random vector (X,Y) and plot them I the plane. How many possible outcomes are there?

• Is a valid joint PMF?• .

Find the marginal probability

Homework1

2) The values of a joint PMF are given below. Determine the marginal PMFs

3)