Exercises Chapter 2 Maximum Likelihood · PDF fileExercises Chapter 2 Maximum Likelihood Estimation Advanced Econometrics - HEC Lausanne Christophe Hurlin University of OrlØans November

Exercises Chapter 2

Maximum Likelihood Estimation

Advanced Econometrics - HEC Lausanne

Christophe Hurlin

University of Orléans

November 2013

Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 2013 1 / 74

Exercise 1

MLE and Geometric Distribution


Problem (MLE and geometric distribution)We consider a sample X1,X2, ..,XN of i .i .d . discrete random variables,where Xi has a geometric distribution with a pmf given by:

fX (x , θ) = Pr (X = x) = θ � (1� θ)x�1 8x 2 f1, 2, 3, ..g

where the success probability θ satis�es 0 < θ < 1 and is unknown. Weassume that:

E (X ) =1θ

V (X ) =1� θ

θ2

Question 1: Write the log-likelihood function of the sample fx1, x2, ..xNg .


Solution

fX (x , θ) = Pr (X = x) = θ � (1� θ)x�1 8x 2 f1, 2, 3, ..gSince the X1,X2, ..,XN are i .i .d . then

LN (θ; x1.., xN ) =N

∏i=1fX (xi ; θ) = θN � (1� θ)∑

Ni=1(xi�1)

`N (θ; x1, .., xn) =N

∑i=1ln fX (xi ; θ) = N ln (θ) + ln (1� θ)

N

∑i=1(xi � 1) �


Problem (MLE and geometric distribution)Question 2: Determine the maximum likelihood estimator of the successprobability θ.


Solution

The maximum likelihood estimate of the success probability θ is de�ned by:

bθ = argmax0<θ<1

`N (θ; x) = argmax0<θ<1

N ln (θ) + ln (1� θ)N

∑i=1(xi � 1)

The gradient and the hessian (deterministic) are de�ned by:

∂`N (θ; x)∂θ

=Nθ� 11� θ

N

∑i=1(xi � 1)

∂2`N (θ; x)

∂θ2= �N

θ2��

11� θ

�2 N

∑i=1(xi � 1)


Solution (cont�d)

So, the FOC (likelihood equation) is:

∂`N (θ; x)∂θ

��bθ = Nbθ � 1

1� bθN

∑i=1(xi � 1) = 0

() 1� bθbθ =1N

N

∑i=1xi � 1

() 1bθ = 1N

N

∑i=1xi

So we have: bθ = 1xn

where xn denotes the realisation of the sample mean XN = N�1 ∑Ni=1 Xi .


Solution (cont�d)

The SOC is:

∂2`N (θ; x)

∂θ2

��bθ = �Nbθ2 ��

1

1� bθ�2 N

∑i=1(xi � 1)

Since bθ = 1/xn, we have:

N

∑i=1(xi � 1) =

N

∑i=1xi �N = Nxn �N = N

�1bθ � 1

�= N

1� bθbθ

!

So, we have:

∂2`N (θ; x)

∂θ2

��bθ = �Nbθ2 ��

1

1� bθ�2N

1� bθbθ

!

= �N

0@ 1bθ2 + 1bθ �1� bθ�1A


Solution (cont�d)

∂2`N (θ; x)

∂θ2

��bθ = �N

0@bθ�1� bθ�+ bθ2bθ3 �1� bθ�

1A= � Nbθ2 �1� bθ� < 0

we have a maximum since 0 < bθ < 1.Conclusion: the ML estimator of θ is equal to the inverse of the samplemean: bθ = 1

XN�


Problem (MLE and geometric distribution)Question 3: Show that the maximum likelihood estimator of the successprobability θ is weakly consistent.


Solution

In two lines...

1 Since the X1,X2, ..,XN are i .i .d . then according to the Khinchin�stheorem (WLLN), we have:

XNp! E (Xi ) =

1θ

2 Given that bθ = 1/XN , by using the continuous mapping theorem(CMP) for a function g (x) = 1/x , we get:

bθ = g �XN � p! g�1θ

�or equivalently bθ p! θ

The estimator bθ is (weakly) consistent. �


Problem (MLE and geometric distribution)Question 4: By using the asymptotic properties of the MLE, derive theasymptotic distribution of the ML estimator bθ = 1/XN .


Solution

1 The log-likelihood function ln fX (θ; xi ) satis�es the regularityconditions.

2 So, the ML estimator is asymptotically normally distributed with

pN�bθ � θ0

�d! N

�0, I�1 (θ0)

�where θ0 denotes the true value of the parameter and I (θ0) the(average) Fisher information number for one observation.


Solution (cont�d)

3. Compute the Fisher information number for one observation. Sincewe consider a marginal log-likelihood, the Fisher information numberassociated to Xi is the same for the observations i . We have threede�nition for I (θ)

I (θ) = Vθ

�∂ì (θ;Xi )

∂θ

�= Eθ

�∂ì (θ;Xi )

∂θ

∂`>i (θ;Xi )∂θ

�= Eθ

��∂2ì (θ;Xi )

∂θ2

�


Solution (cont�d)

Let us consider the third one:

I (θ) = Eθ

��∂2ì (θ;Xi )

∂θ2

�= Eθ

1

θ2+

�1

1� θ

�2(Xi � 1)

!

=1

θ2+

�1

1� θ

�2(Eθ (Xi )� 1)

=1

θ2+

�1

1� θ

�2��1θ� 1�

=1

θ2 (1� θ)


Solution (cont�d)

The asymptotic distribution of the ML estimator is:

pN�bθ � θ0

�d�!

N!∞N�0, θ20 (1� θ0)

�where θ0 denotes the true value of the parameter. Or equivalently:

bθ asy� N

θ0,

θ20 (1� θ0)

N

!�


Problem (MLE and geometric distribution)Question 5: By using the central limit theorem and the delta method,�nd the asymptotic distribution of the ML estimator bθ = 1/XN .


Solution

1 Since the X1,X2, ..,XN are i .i .d . with E (X ) = 1/θ0 andV (X ) = (1� θ0) /θ20, according to the Lindberg-Levy�s CLT we getimmediately

pN�XN �

1θ0

�d! N

�0,1� θ0

θ20

�2 Our MLE estimator is de�ned by bθ = 1/XN . Let us consider afunction g (z) = 1/z . So, g (.) is a continuous and continuouslydi¤erentiable function with g (1/θ) = θ 6= 0 and not involving N,then the delta method implies

pN�g�XN�� g

�1θ0

��d! N

0,�

∂g (z)∂z

��1/θ

�2 �1� θ0

θ20

�!


Solution (cont�d)

pN�g�XN�� g

�1θ0

��d! N

0,�

∂g (z)∂z

��1/θ

�2 �1� θ0

θ20

�!

We known that g (z) = 1/z and ∂g (z) /∂z = �1/z2, so we have

pN�bθ � θ0

�d! N

�0, θ40 �

�1� θ0

θ20

��Finally, we get the same result as in the previous question:

pN�bθ � θ0

�d! N

�0, θ20 (1� θ0)

��


Problem (MLE and geometric distribution)Question 6: Determine the FDCR or Cramer-Rao bound. Is the MLestimator bθ e¢ cient and/or asymptotically e¢ cient?


Solution

The FDCR or Cramer-Rao bound is de�ned by:

FDCR = I�1N (θ0)

where IN (θ0) denotes the Fisher information number for the sampleevaluated at the true value θ0. There are three alternative de�nitions forIN (θ0) .

IN (θ0) = Vθ

∂`N (θ;X )

∂θ

��θ0

!

IN (θ0) = Eθ

∂`N (θ;X )

∂θ

��θ0

∂`N (θ;X )>

∂θ

��θ0

!

IN (θ0) = Eθ

� ∂2`N (θ;X )

∂θ∂θ>

��θ0

!


Solution (cont�d)

Let us consider the third one:

IN (θ0) = Eθ

� ∂2`N (θ;X )

∂θ∂θ>

��θ0

!

= Eθ

N

θ20+

�1

1� θ0

�2 N

∑i=1(Xi � 1)

!

=N

θ20+

�1

1� θ0

�2 N

∑i=1(Eθ (Xi )� 1)

=N

θ20+

�1

1� θ0

�2�N �

�1θ0� 1�

=N

θ20 (1� θ0)


Solution (cont�d)

So, the FDCR or Cramer-Rao bound is de�ned by:

FDCR = I�1N (θ0) =θ20 (1� θ0)

N

1 We don�t know if bθ is e¢ cient... For that we need to compute thevariance V

�bθ� = V�1/XN

�.

2 Since the log-likelihood function ln fX (θ; xi ) satis�es the regularityconditions, the MLE is asymptotically e¢ cient. �

Remark: we shown that for N large:

Vasy

�bθ� = I�1N (θ0) =θ20 (1� θ0)

N


Remark

How to get the Fisher information number for the sample (and as aconsequence the FDCR or Cramer-Rao bound) in one line from thequestion 4? Since the sample is i .i .d ., we have:

IN (θ0) = N � I (θ0) =N

θ20 (1� θ0)


Problem (MLE and geometric distribution)Question 7: Propose a consistent estimator for the asymptotic varianceof the ML estimator bθ.


Solution

We have:

Vasy

�bθ� = θ20 (1� θ0)

N

and we know that the ML estimator bθ is a (weakly) consistent estimator ofθ0: bθ p! θ0

A natural estimator for the asymptotic variance is given by:

bVasy

�bθ� = bθ20�1� bθ0�N

Given the CMP and Slutsky�s theorem, it is easy to show that:

bVasy

�bθ� p! Vasy

�bθ� �


Problem (MLE and geometric distribution)Question 8: Write a Matlab code in order to

(1) Generate a sample of size N = 1, 000 of i .i .d . random variabledistributed according to a geometric distribution with a success probabilityθ = 0.3 by using the function geornd.

(2) Estimate by MLE the parameter θ. Compare your estimate with thesample mean.

Remak: There are two de�nitions of geometric distribution:

Pr (X = x) = θ � (1� θ)x�1 8x 2 f1, 2, ..g used in this exercice

Pr (X = x) = θ� (1� θ)x 8x 2 f0, 1, 2, ..g used by Matlab for geornd.




Exercise 2

MLE and AR(p) processes


De�nition (AR(1) process)

A stationary Gaussian AR(1) process takes the form

Yt = c + ρYt�1 + εt

with εt i .i .d . N�0, σ2

�, jρj < 1 and:

E (Yt ) =c

1� ρV (Yt ) =

σ2

1� ρ2


Problem (MLE and AR processes)

Question 1: Denote θ =�c ; ρ; σ2

�> the 3� 1 vector of parameters andwrite the likelihood and the log-likelihood of the �rst observation y1.


Solution

Since the variable Y1 is gaussian with

E (Yt ) =c

1� ρV (Yt ) =

σ2

1� ρ2

The (unconditional) likelihood of y1 is equal to:

L1 (θ; y1) =1p

2πp

σ2/ (1� ρ2)exp

�12(y1 � c/ (1� ρ))2

σ2/ (1� ρ2)

!

The (unconditional) log-likelihood of y1 is equal to:

`1 (θ; y1) = �12ln (2π)� 1

2ln�

σ2

1� ρ2

�� 12(y1 � c/ (1� ρ))2

σ2/ (1� ρ2)�


Problem (MLE and AR processes)Question 2: What is the conditional distribution of Y2 given Y1 = y1.Write the (conditional) likelihood and the (conditional) log-likelihood ofthe second observation y2.


Solution

For t = 2, we have:Y2 = c + ρY1 + ε2

where ε2 � N�0, σ2

�. As a consequence, the conditional distribution of

Y2 given Y1 = y1 is also normal:

Y2jY1 = y1 � N�c + ρy1, σ2

�


Solution (cont�d)

GivenY2jY1 = y1 � N

�c + ρy1, σ2

�The conditional likelihood of y2 is equal to:

L2 (θ; y2j y1) =1

σp2π

exp

�12(y2 � c � ρy1)

2

σ2

!

The conditional log-likelihood of y2 is equal to:

`2 (θ; y2j y1) = �12ln (2π)� 1

2ln�σ2�� 12(y2 � c � ρy1)

2

σ2�



Question 3: Consider a sample of fy1, y2g of size T = 2. Write theexact likelihood (or full likelihood) and the exact log-likelihood of theAR (1) model for the sample fy1, y2g .Note that for two continuous random variables X and Y , the pdf of thejoint distribution (X ,Y ) can be written as:

fX ,Y (x , y) = fX jY=y (x j y)� fY (y)


Solution

The exact (or full) likelihood of the sample fy1, y2g corresponds to the pdfof the joint distribution of (Y1,Y2) :

LT (θ; y1, y2) = fY1,Y2 (y1, y2)

This joint density can be rewritten as the product of the marginal densityof Y1 by the conditional density of Y2 given Y1 = y1 :

LT (θ; y1, y2) = fY2 jY1=y1 (y2j y1; θ)� fY1 (y1; θ)

or equivalently:

LT (θ; y1, y2) = L2 (θ; y2j y1)� L1 (θ; y1)


Solution (cont�d)

The exact (or full) likelihood of the sample fy1, y2g is equal to:

LT (θ; y1, y2) =1p

2πp

σ2/ (1� ρ2)exp

�12(y1 � c/ (1� ρ))2

σ2/ (1� ρ2)

!

� 1

σp2π

exp

�12(y2 � c � ρy1)

2

σ2

!


Solution (cont�d)

Similarly the exact (or full) log-likelihood of the sample fy1, y2g is equalto:

`T (θ; y1, y2) = `2 (θ; y2j y1) + `1 (θ; y1)Then, we get:

`T (θ; y1, y2) = �12ln (2π)� 1

2ln�

σ2

1� ρ2

�� 12(y1 � c/ (1� ρ))2

σ2/ (1� ρ2)

�12ln (2π)� 1

2ln�σ2�� 12(y2 � c � ρy1)

2

σ2�



Question 4: Write the exact likelihood (or full likelihood) and theexact log-likelihood of the AR (1) model for a sample fy1, y2, .., yT g ofsize T .


Solution

More generally, we have:

LT (θ; y1, ., yT ) = L1 (θ; y1)�T

∏t=2Lt (θ; yt j yt�1)

`T (θ; y1, .., yT ) = `1 (θ; y1) +T

∑t=2`t (θ; yt j yt�1)


Solution (cont�d)

LT (θ; y) =1p

2πp

σ2/ (1� ρ2)exp

�12(y1 � c/ (1� ρ))2

σ2/ (1� ρ2)

!

�T

∏t=2

1

σp2π

exp

�12(yt � c � ρyt�1)

2

σ2

!

`T (θ; y) = �12ln (2π)� 1

2ln�

σ2

1� ρ2

�� 12(y1 � c/ (1� ρ))2

σ2/ (1� ρ2)

+T

∑t=2

�12ln (2π)� 1

2ln�σ2�� 12(yt � c � ρyt�1)

2

σ2

!�


Problem (MLE and AR processes)Question 5: The exact log-likelihood function is a non-linear function ofthe parameters θ, and so there is no closed form solution for the exactmles. The exact MLE bθ = (bc ;bρ; bσ2)> must be determined by numericallymaximizing the exact log-likelihood function. Write a Matlab code to

(1) to generate a sample of size T = 1, 000 from an AR (1) process withc = 1, ρ = 0.5 and σ2 = 1. Remark : for the initial condition, generate anormal random variable.

(2) to compute the exact MLE.





Problem (MLE and AR processes)Question 6: Now we consider the �rst observation y1 as given(deterministic). Then, we have fY1 (y1; θ) = 1. Write the conditionallog-likelihood of the AR (1) model for a sample fy1, y2, .., yT g of size T .


Solution

The conditional likelihood is de�ned by:

LT (θ; y2, .., yT j y1) =T

∏t=2fYt jYt�1,Y1=y1 (yt j yt�1, y1; θ)� fY1 (y1; θ)

=T

∏t=2fYt jYt�1 (yt j yt�1; θ)

The conditional log-likelihood is de�ned by:

`T (θ; y1, .., yT j y1) = `1 (θ; y1) +T

∑t=2`t (θ; yt j yt�1, y1)

=T

∑t=2`t (θ; yt j yt�1)

where `t (θ; yt j yt�1) = ln�fYt jYt�1 (yt j yt�1; θ)

�.


Solution (cont�d)

The conditional log-likelihood is then equal to:

`T (θ; y) =T

∑t=2

�12ln (2π)� 1

2ln�σ2�� 12(yt � c � ρyt�1)

2

σ2

!

or equivalently

`T (θ; y) = � (T � 1)2

ln (2π)� (T � 1)2

ln�σ2�

� 12σ2

T

∑t=2(yt � c � ρyt�1)

2�


Problem (MLE and AR processes)Question 7: Write the likelihood equations associated to the conditionallog-likelihood.


Solution

The ML estimator bθ = (bc ;bρ; bσ2)> of θ is de�ned by:

bθ = argmaxθ2Θ

`T (θ; y1, .., yT )

The log-likelihood equations are:

∂`T (θ; y)∂θ

��bθ =0BBBBB@

∂`T (θ;y )∂c

��bθ∂`T (θ;y )

∂ρ

��bθ∂`T (θ;y )

∂σ2

��bθ

1CCCCCA =

0@ 000

1A


Solution (cont�d)

`T (θ; y) = � (T � 1)2

ln (2π)� (T � 1)2

ln�σ2�

� 12σ2

T

∑t=2(yt � c � ρyt�1)

2

∂`T (θ; y)∂c

��bθ = 1bσ2T

∑t=2(yt � bc � bρyt�1) = 0

∂`T (θ; y)∂ρ

��bθ = 1bσ2T

∑t=2(yt � bc � bρyt�1) yt�1 = 0

∂`T (θ; y)∂σ2

��bθ = � (T � 1)2bσ2 +1

2bσ4T

∑t=2(yt � bc � bρyt�1)2 = 0 �



Question 8: Show that the conditional ML estimators bc and bρ correspondto the OLS estimator. Give the the estimator of bσ2. Remark: do not verifythe SOC at this step.


Solution

The maximisation of `T (θ; y) with respect to c and ρ

`T (θ; y) = � (T � 1)2

ln (2π)� (T � 1)2

ln�σ2�

� 12σ2

T

∑t=2(yt � c � ρyt�1)

2

is equivalent to the minimisation of

T

∑t=2(yt � c � ρyt�1)

2 = (y�Xβ)> (y�Xβ)

with y = (y2; ..; yN )> , β = (c ; ρ)> and X = (1 : y�1) with

y�1 = (y1; ..; yN�1)> .


Solution

The conditional ML estimators of c and ρ are equivalent to the ordinaryleast square (OLS) estimators obtained in regression of yt on a constantand its own lagged value:

yt = c + ρyt�1 + εt� bcbρ�=

T � 1 ∑T

t=2 yt�1

∑Tt=2 yt�1 ∑T

t=2 y2t�1

!�1 ∑Tt=2 yt

∑Tt=2 yt�1yt

!


Solution

The ML estimator bσ2 is de�ned by:∂`T (θ; y)

∂σ2

��bθ = � (T � 1)2bσ2 +1

2bσ4T

∑t=2(yt � bc � bρyt�1)2 = 0

Then, we get:

bσ2 = 1T � 1

T

∑t=2(yt � bc � bρyt�1)2 = 1

T � 1T

∑t=2

bεt 2 �


Problem (MLE and AR processes)Question 9: Write a Matlab code to compute the conditional maximumlikelihood estimator bθ = (bc ;bρ; bσ2)>.(1) Generate a sample of size T = 1, 000 from an AR (1) process withc = 1, ρ = 0.5 and σ2 = 1. Remark : for the initial condition, generate anormal random variable.

(2) Compute the conditional MLE.

(3) Compare the ML estimators bc and bρ to the OLS ones.





Problem (MLE and AR processes)Question 10: Write the average Fisher information matrix associated tothe conditonal likelihood.


Solution

In general for a conditional model, in order to compute the averageinformation matrix I (θ) for one observation:

Step 1: Compute the Hessian matrix or the score vector for oneobservation

Hi (θ; Yi j xi ) =∂2ì (θ; Yi j xi )

∂θ∂θ>si (θ; Yi j xi ) =

∂ì (θ; Yi j xi )∂θ

Step 2: Take the expectation (or the variance) with respect to theconditional distribution Yi jXi = xi

I i (θ) = Vθ (si (θ; Yi j xi )) = Eθ (�Hi (θ; Yi j xi ))

Step 3: Then the expectation with respect to the conditioning variable X

I (θ) = EX (I i (θ))


Solution (cont�d)

Step 1:∂`t (θ; yt )

∂c=1

σ2(yt � c � ρyt�1)

∂`t (θ; yt )∂ρ

=1

σ2(yt � c � ρyt�1) yt�1

∂`t (θ; yt )∂σ2

= � 12σ2

+12σ4

(yt � c � ρyt�1)2

The Hessian matrix for one observation is de�ned by:

Ht (θ; Yt j yt�1) =

0B@ �1/σ2 �yt�1/σ2 �εt/σ4

�yt�1/σ2 �y2t�1/σ2 �εtyt�1/σ4

�εt/σ4 �εtyt�1/σ4 1/2σ4 � ε2t /σ6

1CA


Solution (cont�d)

Ht (θ; Yt j yt�1) =

0B@ �1/σ2 �yt�1/σ2 �εt/σ4

�yt�1/σ2 �y2t�1/σ2 �εtyt�1/σ4

�εt/σ4 �εtyt�1/σ4 1/2σ4 � ε2t /σ6

1CAStep 2: Take the expectation (or the variance) with respect to theconditional distribution Yt jYt�1 = yt�1

I t (θ) = Eθ (�Ht (θ; Yt j yt�1))

I t (θ) =

0B@ 1/σ2 yt�1/σ2 0

yt�1/σ2 y2t�1/σ2 0

0 0 1/2σ4

1CAsince Eθ (εt ) = 0, Eθ (εtyt�1) = yt�1Eθ (εt ) = 0 and Eθ

�ε1t�= σ2.


Solution (cont�d)

I t (θ) =

0B@ 1/σ2 yt�1/σ2 0

yt�1/σ2 y2t�1/σ2 0

0 0 1/2σ4

1CAStep 3: Then take the expectation with respect to the conditioningvariable xt = (1 : yt�1)

I (θ) = EX (I i (θ))

I (θ) =

0B@ 1/σ2 EX (yt�1) /σ2 0

EX (yt�1) /σ2 EX�y2t�1

�/σ2 0

0 0 1/2σ4

1CA �


Problem (MLE and AR processes)Question 11: What is the asymptotic distribution of the conditionalMLE? Propose an estimator for the asymptotic variance covariance matrixof bθ = (bc ;bρ; bσ2)>.


Solution

Since the log-likelihood is regular, we have:

pT � 1

�bθ� θ0�

d! N�0, I�1 (θ0)

�or equivalently bθ � N �

θ0,1

T � 1 I�1 (θ0)

�with

I (θ) =

0B@ 1/σ2 EX (yt�1) /σ2 0


�/σ2 0

0 0 1/2σ4

1CA


Solution (cont�d)

I (θ) =

0B@ 1/σ2 EX (yt�1) /σ2 0


�/σ2 0

0 0 1/2σ4

1CAAn estimator of the asymptotic variance covariance matrix can be derivedfrom:

bI (θ) =0B@ 1/bσ2 (T � 1)�1 ∑T

t=2 yt�1/bσ2 0

(T � 1)�1 ∑Tt=2 yt�1/bσ2 (T � 1)�1 ∑T

t=2 y2t�1/bσ2 0

0 0 1/2bσ41CA

where bσ2 is the ML estimator of σ2.

bVasy

�bθ� = 1T � 1

bI�1 (θ)Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 2013 69 / 74

Solution (cont�d)

If we denote by X = (1 : y�1) , then we have:

bI (θ) = 1T�1X

>X/bσ2 02�101�2 1/2bσ4

!

since

X>X =�

T � 1 ∑Tt=2 yt�1

∑Tt=2 yt�1 ∑T

t=2 y2t�1

�


Problem (MLE and AR processes)Question 12: Write a Matlab code to compute the asymptotic variancecovariance matrix associated to the conditional maximum likelihoodestimator bθ = (bc ;bρ; bσ2)>.(1) Import the data from the excel �le Chapter2_Exercice2.xls

(2) Compute the asymptotic variance covariance matrix of theconditional MLE.

(3) Compare your results with the results reported in Eviews.



Perfect....


End of Exercices - Chapter 2

Christophe Hurlin (University of Orléans)


Documents

Exercises Chapter 2 Maximum Likelihood · PDF fileExercises Chapter 2 Maximum Likelihood Estimation Advanced Econometrics - HEC Lausanne Christophe Hurlin University of OrlØans November