Chapter 4: Statistical Hypothesis Testing · 2. Statistical hypothesis testing Objectives The objective of this section is to de–ne the following concepts: 1 Null and alternative

Chapter 4: Statistical Hypothesis Testing

Christophe Hurlin

November 20, 2015

Christophe Hurlin () Advanced Econometrics - Master ESA November 20, 2015 1 / 225

Section 1

Introduction

Christophe Hurlin (University of Orléans) Advanced Econometrics - Master ESA November 20, 2015 2 / 225

1. Introduction

The outline of this chapter is the following:

Section 2. Statistical hypothesis testing

Section 3. Tests in the multiple linear regression model

Subsection 3.1. The Student test

Subsection 3.2. The Fisher test

Section 4. MLE and Inference

Subsection 4.1. The Likelihood Ratio (LR) test

Subsection 4.2.The Wald test

Subsection 4.3. The Lagrange Multiplier (LM) test


1. Introduction

References

Amemiya T. (1985), Advanced Econometrics. Harvard University Press.

Greene W. (2007), Econometric Analysis, sixth edition, Pearson - PrenticeHil (recommended)

Ruud P., (2000) An introduction to Classical Econometric Theory, OxfordUniversity Press.


1. Introduction

Notations: In this chapter, I will (try to...) follow some conventions ofnotation.

fY (y) probability density or mass function

FY (y) cumulative distribution function

Pr () probability

y vector

Y matrix

Be careful: in this chapter, I don�t distinguish between a random vector(matrix) and a vector (matrix) of deterministic elements (except in section2). For more appropriate notations, see:

Abadir and Magnus (2002), Notation in econometrics: a proposal for astandard, Econometrics Journal.


Section 2

Statistical hypothesis testing


2. Statistical hypothesis testingObjectives

The objective of this section is to de�ne the following concepts:

1 Null and alternative hypotheses

2 One-sided and two-sided tests

3 Rejection region, test statistic and critical value

4 Size, power and power function

5 Uniformly most powerful (UMP) test

6 Neyman Pearson lemma

7 Consistent test and unbiased test

8 p-value


2. Statistical hypothesis testing

Introduction

1 A statistical hypothesis test is a method of making decisions or arule of decision (as concerned a statement about a populationparameter) using the data of sample.

2 Statistical hypothesis tests de�ne a procedure that controls (�xes)the probability of incorrectly deciding that a default position (nullhypothesis) is incorrect based on how likely it would be for a set ofobservations to occur if the null hypothesis were true.



Introduction (cont�d)

In general we distinguish two types of tests:

1 The parametric tests assume that the data have come from a typeof probability distribution and makes inferences about the parametersof the distribution

2 The non-parametric tests refer to tests that do not assume the dataor population have any characteristic structure or parameters.

In this course, we only consider the parametric tests.




A statistical test is based on three elements:

1 A null hypothesis and an alternative hypothesis

2 A rejection region based on a test statistic and a critical value

3 A type I error and a type II error










De�nition (Hypothesis)A hypothesis is a statement about a population parameter. The formaltesting procedure involves a statement of the hypothesis, usually in termsof a �null�or maintained hypothesis and an �alternative,� conventionallydenoted H0 and H1, respectively.



Introduction

1 The null hypothesis refers to a general or default position: that thereis no relationship between two measured phenomena or that apotential medical treatment has no e¤ect.

2 The costs associated to the violation of the null must be higher thanthe cost of a violation of the alternative.

Example (Choice of the null hypothesis)In a credit scoring problem, in general we have: H0 : the client is notrisky(acceptance of the loan) versus H1 : the client is risky (refusal of theloan).



De�nition (Simple and composite hypotheses)A simple hypothesis speci�es the population distribution completely. Acomposite hypothesis does not specify the population distributioncompletely.

Example (Simple and composite hypotheses)

If X � t (θ) , H0 : θ = θ0 is a simple hypothesis. H1 : θ > θ0, H1 : θ < θ0,and H1 : θ 6= θ0 are composite hypotheses.



De�nition (One-sided test)A one-sided test has the general form:

H0 : θ = θ0 or H0 : θ = θ0

H1 : θ < θ0 H1 : θ > θ0



De�nition (Two-sided test)A two-sided test has the general form:

H0 : θ = θ0

H1 : θ 6= θ0










De�nition (Rejection region)

The rejection region is the set of values of the test statistic (orequivalently the set of samples) for which the null hypothesis is rejected.The rejection region is denoted W. For example, a standard rejectionregion W is of the form:

W = fx : T (x) > cg

or equivalentlyW = fx1, .., xN : T (x1, .., xN ) > cg

where x denotes a sample fx1, .., xNg , T (x) the realisation of a teststatistic and c the critical value.



Remarks

1 A (hypothesis) test is thus a rule that speci�es:

1 For which sample values the decision is made to �fail to reject H0�astrue;

2 For which sample values the decision is made to �reject H0�.

3 Never say "Accept H1", "fail to reject H1" etc..

2 The complement of the rejection region is the non-rejection region.



Remark

The rejection region is de�ned as to be:

W = fx : T (x)| {z }test statistic

7 c|{z}gcritical value

T (x) is the realisation of the statistic (random variable):

T (X ) = T (X1, ..,XN )

The test statistic T (X ) has an exact or an asymptotic distribution Dunder the null H0.

T (X ) �H0D or T (X )

d!H0D










Decision

Fail to reject H0 Reject H0

Truth H0 Correct decision Type I error

H1 Type II error Correct decision



De�nition (Size)

The probability of a type I error is the (nominal) size of the test. This isconventionally denoted α and is also called the signi�cance level.

α = Pr (WjH0)



Remark

For a simple null hypothesis:

α = Pr (WjH0)

For a composite null hypothesis:

α = supθ02H0

Pr (WjH0)

A test is said to have level if its size is less than or equal to α.



De�nition (Power)The power of a test is the probability that it will correctly lead torejection of a false null hypothesis:

power = Pr (WjH1) = 1� β

where β denotes the probability of type II error, i.e. β = Pr�W��H1� and

W denotes the non-rejection region.



Example (Test on the mean)Consider a sequence X1, ..,XN of i .i .d . continuous random variables withXi � N

�m, σ2

�where σ2 is known. We want to test

H0 : m = m0H1 : m = m1

with m1 < m0. An econometrician propose the following rule of decision:

W = fx : xN < cg

where XN = N�1 ∑Ni=1 Xi denotes the sample mean and c is a constant

(critical value). Question: calculate the size and the power of this test.



Solution

The rejection region is W= fx : xN < cg . Under the null H0 : m = m0 :

XN �H0N�m0,

σ2

N

�So, the size of the test is equal to:

α = Pr (WjH0)= Pr

�XN < c

��H0�= Pr

�XN �m0

σ/pN

<c �m0σ/pN

��H0�= Φ

�c �m0σ/pN

�



Solution (cont�d)

The rejection region is W= fx : xN < cg . Under the alternativeH1 : m = m1 :

XN �H1N�m1,

σ2

N

�So, the power of the test is equal to:

power = Pr (WjH1)

= Pr�XN �m1

σ/pN

<c �m1σ/pN

��H1�= Φ

�c �m1σ/pN

��



Solution (cont�d)

In conclusion:

α = Φ�c �m0σ/pN

�β = 1� power = 1�Φ

�c �m1σ/pN

�We have a system of two equations with three parameters: α, β (or power)and the critical value c .

1 There is a trade-o¤ between the probabilities of the errors of type Iand II, i.e. α and β : if c decreases, α decreases but β increases.

2 A solution is to impose a size α and determine the critical value andthe power.



Solution (cont�d)

In order to illustrate the tradeo¤ between α and β given the critical valuec , take an example with σ2 = 1 and N = 100:

H0 : m = m0 = 1.2 H1 : m = m1 = 1

XN �H0N�m0,

σ2

N

�XN �

H1N�m1,

σ2

N

�We have

W = fx : xN < cg

α = Pr (WjH0) = Φ�c �m0σ/pN

�= Φ (10 (c � 1.2))

β = Pr�W��H1� = 1�Φ

�c �m1σ/pN

�= 1�Φ (10 (c � 1))



0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.60

0.5

1

1.5

2

2.5

3

3.5

4

4.5Density under H0Density under H1

α=Pr(W[H0)=5% β=1Pr(W|H1)=36.12%



Click me!



Fact (Critical value)

The (nominal) size α is �xed by the analyst and the critical value isdeduced from α.




�m, σ2

�, N = 100 and σ2 = 1 . We want to test

H0 : m = 1.2 H1 : m = 1

An econometrician propose the following rule of decision:

W = fx : xN < cg

where XN = N�1 ∑Ni=1 Xi denotes the sample mean and c is a constant

(critical value). Questions: (1) what is the critical value of the test ofsize α = 5%? (2) what is the power of the test?



Solution

We know that:

α = Pr (WjH0) = Φ�c �m0σ/pN

�So, the critical value that corresponds to a signi�cance level of α is:

c = m0 +σpN

Φ�1 (α)

NA: if m0 = 1.2, m1 = 1, N = 100, σ2 = 1 and α = 5%, then therejection region is

W = fx : xN < 1.0355g


2. Statistical hypothesis testingSolution (cont�d)

W =

�x : xN < m0 +

σpN

Φ�1 (α)

�The power of the test is:

power = Pr (WjH1) = Φ�c �m1σ/pN

�Given the critical value, we have:

power = Φ�m0 �m1σ/pN+Φ�1 (α)

��

NA: if m0 = 1.2, m1 = 1, N = 100, σ2 = 1 and α = 5%:

power = Φ�1.2� 11/p100

+Φ�1 (0.05)�= 0.6388 �




�m, σ2

�with σ2 = 1 and N = 100. We want to test

H0 : m = 1.2 H1 : m = 1

The rejection region for a signi�cance level α = 5% is:

W = fx : xN < 1.0355g

where XN = N�1 ∑Ni=1 Xi denotes the sample mean. Question: if the

realisation of the sample mean is equal to 1.13, what is the conclusion ofthe test?



Solution (cont�d)

For a nominal size α = 5%, the rejection region is:

W = fx : xN < 1.0355g

If we observexN = 1.13

This realisation does not belong to the rejection region:

xN /2 W

For a level of 5%, we do not reject the null hypothesis H0 : m = 1.2. �



De�nition (Power function)In general, the alternative hypothesis is composite. In this case, the poweris a function of the value of the parameter under the alternative.

power = P (θ) 8θ 2 H1




�m, σ2


H0 : m = m0H1 : m < m0

Consider the following rule of decision:

W =

�x : xN < m0 +

σpN

Φ�1 (α)

�Questions: What is the power function of the test?



Solution

As in the previous case, we have:

power = P (m) = Φ�m0 �mσ/pN+Φ�1 (α)

�with m < m0

NA: if m0 = 1.2, N = 100, σ2 = 1 and α = 5%.

P (m) = Φ�1.2�m1/10

� 1.6449�

with m < m0



Power function P (m)

0.7 0.8 0.9 1 1.1 1.20

0.2

0.4

0.6

0.8

1

1.2



Example (Power function)

Consider a test H0 : θ = θ0 versus H1 : θ 6= θ0, the power function has thisgeneral form:



De�nition (Most powerful test)

A test (denoted A) is uniformly most powerful (UMP) if it has greaterpower than any other test of the same size for all admissible values of theparameter.

αA = αB = α

βA � βB

for any test B of size α.



UMP tests

How to derive the rejection region of the UMP test of size α ?

=> the Neyman�Pearson lemma



Lemma (Neyman Pearson)

Consider a hypothesis test between two point hypotheses H0 : θ = θ0 andH1 : θ = θ1. The uniformly most powerful (UMP) test has a rejectionregion de�ned by:

W =

�x j LN (θ0; x)LN (θ1; x)

< K�

where LN (θ0; x) denotes the likelihood of the sample x and K is aconstant determined by the size α such that:

Pr�LN (θ0;X )LN (θ1;X )

< K

��H0� = α




�m, σ2


H0 : m = m0H1 : m = m1

with m1 > m0. Question: What is the rejection region of the UMP test ofsize α?



Solution

Since X1, ..,XN are N .i .d .�m , σ2

�, the likelihood of the sample

fx1, .., xNg is de�ned as to be (cf. chapter 2):

LN (θ; x) =1

σN (2π)N/2 exp

� 12σ2

N

∑i=1(xi �m)2

!

Given the Neyman Pearson lemma the rejection region of the UMP test ofsize α is given by:

LN (θ0; x)LN (θ1; x)

< K

where K is a constant determined by the size α.



Solution (cont�d)

1σN (2π)N/2 exp

�� 12σ2 ∑N

i=1 (xi �m0)2�

1σN (2π)N/2 exp

�� 12σ2 ∑N

i=1 (xi �m1)2� < K

This expression can rewritten as:

exp�12σ2

�∑Ni=1 (xi �m1)

2 �∑Ni=1 (xi �m0)

2��

< K

() ∑Ni=1 (xi �m1)

2 �∑Ni=1 (xi �m0)

2 < K1

where K1 = 2σ2 ln (K ) is a constant.



Solution (cont�d)

∑Ni=1 (xi �m1)

2 �∑Ni=1 (xi �m0)

2 < K1

() 2 (m0 �m1)∑Ni=1 xi +N

�m21 �m20

�< K1

() (m0 �m1)∑Ni=1 xi < K2

where K2 =�K1 �N

�m21 �m20

��/2 is a constant.



Solution (cont�d)

(m0 �m1)∑Ni=1 xi < K2

Since m1 > m0, we have

1N

∑Ni=1 xi > K3

where K3 = K2/ (N (m0 �m1)) is a constant.The rejection region of the UMP test for H0 : m = m0 againstH0 : m = m1 with m1 > m0 has the general form:

W = fx : xN > Ag

where A is a constant.



Solution (cont�d)

W = fx : xN > AgDetermine the critical value A from the nominal size:

α = Pr (WjH0)= Pr (xN > AjH0)

= 1� Pr�XN �m0

σ/pN

<A�m0σ/pN

��H0�= 1�Φ

�A�m0σ/pN

�



Solution (cont�d)

α = 1�Φ�A�m0σ/pN

�So, we have

A = m0 +σpN

Φ�1 (1� α)

The rejection region of the UMP test of size α for H0 : m = m0 againstH0 : m = m1 with m1 > m0 is:

W =

�x : xN > m0 +

σpN

Φ�1 (1� α)

��



Fact (UMP one-sided test)For a one-sided test

H0 : θ = θ0 against H1 : θ > θ0 (or H1 : θ < θ1)

the rejection region W of the UMP test is equivalent to the rejectionregion obtained for the test

H0 : θ = θ0 against H1 : θ = θ1

with for θ1 > θ0 (or θ1 < θ0) if this region does not depend on the valueof θ1.




�m, σ2


H0 : m = m0H1 : m > m0

Question: What is the rejection region of the UMP test of size α?


2. Statistical hypothesis testingSolution

Consider the test:

H0 : m = m0H1 : m = m1

with m1 > m0. The rejection region of the UMP test of size α is:

W =

�x : xN > m0 +

σpN

Φ�1 (1� α)

�W does not depend on m1. It is also the rejection region of the UMPone-sided test for

H0 : m = m0H1 : m > m0 �



Fact (Two-sided test)For a two-sided test

H0 : θ = θ0 against H1 : θ 6= θ0

the non rejection region W of the test of size α is the intersection of thenon rejection regions of the corresponding one-sided UMP tests of sizeα/2

Test A: H0 : θ = θ0 against H1 : θ > θ0

Test B: H0 : θ = θ0 against H1 : θ < θ0

So, we have:W = W A \W B




�m, σ2


H0 : m = m0H1 : m 6= m0

Question: What is the rejection region of the test of size α?



Solution

Consider the one-sided tests:

Test A: H0 : m = m0 against H1 : m < m0

Test B: H0 : m = m0 against H1 : m > m0

The rejection regions of the UMP test of size α/2 are:

WA =

�x : xN < m0 +

σpN

Φ�1 (α/2)�

WB =

�x : xN > m0 +

σpN

Φ�1 (1� α/2)�



Solution (cont�d)

The non-rejection regions of the UMP test of size α/2 are:

WA =

�x : xN � m0 +

σpN

Φ�1 (α/2)�

WB =

�x : xN � m0 +

σpN

Φ�1 (1� α/2)�

The non rejection region of the two-sided test corresponds to theintersection of these two regions:

W = W A \W B



So, non rejection region of the two-sided test of size α is:

W =

�x : m0 +

σpN

Φ�1 (α/2) � xN � m0 +σpN

Φ�1 (1� α/2)�

Since, Φ�1 (α/2) = �Φ�1 (1� α/2) , this region can be rewritten as:

W =

�x : jxN �m0j �

σpN

Φ�1 (1� α/2)�



W =

�x : jxN �m0j �

σpN

Φ�1 (1� α/2)�

Finally, the rejection region of the two-sided test of size α is:

W =

�x : jxN �m0j >

σpN

Φ�1 (1� α/2)�

�



Solution (cont�d)

W =

�x : jxN �m0j >

σpN

Φ�1 (1� α/2)�

NA: if m0 = 1.2, N = 100, σ2 = 1 and α = 5%:

W =

�x : jxN � 1.2j >

110

Φ�1 (0.975)�

W = fx : jxN � 1.2j > 0.1960gIf the realisation of jxN � 1.2j is larger than 0.1960, we reject the nullH0 : m = 1.2 for a signi�cance level of 5%.



De�nition (Unbiased Test)

A test is unbiased if its power P (θ) is greater than or equal to its size αfor all values of the parameter θ.

P (θ) � α 8θ 2 H1

By construction, we have P (θ0) = Pr (Wj H j0) = α.



De�nition (Consistent Test)A test is consistent if its power goes to one as the sample size grows toin�nity.

limN!∞

P (θ) = 1 8θ 2 H1




�m, σ2


H0 : m = m0H1 : m < m0

The rejection region of the UMP test of size α is

W =

�x : xN < m0 +

σpN

Φ�1 (α)

�Question: show that this test is (1) unbiased and (2) consistent.



Solution

W =

�x : xN < m0 +

σpN

Φ�1 (α)

�The power function of the test is de�ned as to be:

P (m) = Pr (WjH1)

= Pr�XN < m0 +

σpN

Φ�1 (α)

��m < m0�= Φ

�m0 �mσ/pN+Φ�1 (α)

�



Solution

P (m) = Φ�m0 �mσ/pN+Φ�1 (α)

�8m < m0

The test is consistent since:

limN!∞

P (m) = 1

The test is unbiased since

P (m) � α 8m < m0

limm!m0

P (m) = Φ�Φ�1 (α)

�= α �



Solution

The decision �Reject H0�or �fail to reject H0� is not so informative!

Indeed, there is some �arbitrariness� to the choice of α (level).

Another strategy is to ask, for every α, whether the test rejects atthat level.

Another alternative is to use the so-called p-value� the smallest levelof signi�cance at which H0 would be rejected given the value of thetest-statistic.



De�nition (p-value)

Suppose that for every α 2 [0, 1], one has a size α test with rejectionregion Wα. Then, the p-value is de�ned to be:

p-value = inf fα : T (y) 2 Wαg

The p-value is the smallest level at which one can reject H0.



The p-value is a measure of evidence against H0:

p-value evidence

< 0.01 Very strong evidence against H0

0.01� 0.05 Strong evidence against H0

0.05� 0.10 Weak evidence against H0

> 0.10 Little or no evidence against H0



Remarks

1 A large p-value does not mean �strong evidence in favor of H0�.

2 A large p-value can occur for two reasons:

1 H0 is true;

2 H0 is false but the test has low power.

3 The p-value is not the probability that the null hypothesis is true!



For a nominal size of 5%, we reject the null H0 : βSP500 = 0.

For a nominal size of 5%, we fail to reject the null H0 : βC = 0.



Summary

Hypothesis testing is de�ned by the following general procedure

Step 1: State the relevant null and alternative hypotheses (misstating thehypotheses muddies the rest of the procedure!);

Step 2: Consider the statistical assumptions being made about the samplein doing the test (independence, distributions, etc.)� incorrectassumptions mean that the test is invalid!

Step 3: Choose the appropriate test (exact or asymptotic tests) and thusstate the relevant test statistic (say, T).



Summary (cont�d)

Step 4: Derive the distribution of the test statistic under the nullhypothesis (sometimes it is well-known, sometimes it is more tedious!)� forexample, the Student t-distribution or the Fisher distribution.

Step 5: Determine the critical value (and thus the critical region).

Step 6: Compute (using the observations!) the observed value of the teststatistic T , say tobs .

Step 7: Decide to either fail to reject the null hypothesis or reject in favorof the alternative assumption� the decision rule is to reject the nullhypothesis H0 if the observed value of the test statistic, tobs is in thecritical region, and to �fail to reject� the null hypothesis otherwise



Key concepts

1 Null and alternative hypotheses2 Simple and composite hypotheses3 One-sided and two-sided tests4 Rejection region, test statistic and critical value5 Type I and type II errors6 Size, power and power function7 Uniformly most powerful (UMP) test8 Neyman Pearson lemma9 Consistent test and unbiased test10 p-value


Section 3

Tests in the multiple linear regression model


3. Tests in the multiple linear regression model

Objectives

In the context of the multiple linear regression model (cf. chapter 3), theobjective of this section is to present :

1 the Student test

2 the t-statistic and the z-statistic

3 the Fisher test

4 the global F-test

5 To distinguish the case with normality assumption and the casewithout any assumption on the distribution of the error term(semi-parametric speci�cation)



Be careful: in this section, I don�t distinguish between a random vector(matrix) and a vector (matrix) of deterministic elements. For moreappropriate notations, see:

Abadir and Magnus (2002), Notation in econometrics: a proposal for astandard, Econometrics Journal.



Model

Consider the (population) multiple linear regression model:

y = Xβ+ ε

where (cf. chapter 3):

y is a N � 1 vector of observations yi for i = 1, ..,N

X is a N �K matrix of K explicative variables xik for k = 1, .,K andi = 1, ..,N

ε is a N � 1 vector of error terms εi .

β = (β1..βK )> is a K � 1 vector of parameters



Assumptions

Fact (Assumptions)We assume that the multiple linear regression model satisfy theassumptions A1-A5 (cf. chapter 3)

We distinguish two cases:

1 Case 1: assumption A6 (Normality) holds and ε � N�0, σ2IN

�2 Case 2: the distribution of ε is unknown (semi-parametricspeci�cation) and ε �??



Parametric tests

The βk are unknown features of the population, but:

1 One can formulate a hypothesis about their value;

2 One can construct a test statistic with a known �nite sampledistribution (case 1) or an asymptotic distribution (case 2);

3 One can take a �decision�meaning �reject H0� if the value of thetest statistic is too unlikely.



Three tests of interest:

H0 : βk = ak or H0 : βk = akH1 : βk < ak H1 : βk > ak

H0 : βk = akH1 : βk 6= ak

H0 : Rβ = qH1 : Rβ 6= q

where ak = 0 or ak 6= 0.



For that, we introduce two types of test

1 The Student test or t-test

2 The Fisher test of F-test


Subsection 3.1

The Student test


3.1. The Student test

Case 1: Normality assumption A6



Assumption 6 (normality): the disturbances are normally distributed.

εjX � N�0N�1, σ2IN

�



Reminder (cf. chapter 3)

Fact (Linear regression model)

Under the assumption A6 (normality), the estimators bβ and bσ2 have a�nite sample distribution given by:

bβ � N �β,σ2

�X>X

��1�bσ2σ2(N �K ) � χ2 (N �K )

Moreover, bβ and bσ2 are independent. This result holds whether or not thematrix X is considered as random. In this last case, the distribution of bβ isconditional to X.



Remarks

1 Any linear combination of bβ is also normally distributed:Abβ � N �

Aβ,σ2A�X>X

��1A>�

2 Any subset of bβ has a joint normal distribution.bβk � N �

βk , σ2mkk

�where mkk is k th diagonal element of

�X>X

��1.



Reminder

If X and Y are two independent random variables such that

X � N (0, 1)

Y � χ2 (θ)

then the variable Z de�ned as to be

Z =XpY /θ

has a Student�s t-distribution with θ degrees of freedom

Z � t(θ)



Student test statistic

Consider a test with the null:

H0 : βk = ak

Under the null H0 : bβk � akσpmkk

�H0N (0, 1)

bσ2σ2(N �K ) �

H0χ2 (N �K )

and these two variables are independent...



Student test statistic (cont�d)

bβk � akσpmkk

�H0N (0, 1)

bσ2σ2(N �K ) �

H0χ2 (N �K )

So, under the null H0 we have:

bβk�akσpmkkq bσ2

σ2(N�K )(N�K )

=bβk � akbσpmkk �H0 t(N�K )



De�nition (Student t-statistic)

Under the null H0 : βk = ak , the Student test-statistic or t-statistic isde�ned to be:

Tk =bβk � akbse �bβk� �H0 t(N�K )

where N is the number of observations, K is the number of explanatoryvariables (including the constant term), t(N�K ) is the Studentt-distribution with N �K degrees of freedom and

bse �bβk� = bσpmkkwith mkk is k th diagonal element of

�X>X

��1.



Remarks

1 Under the assumption A6 (normality) and under the nullH0 : βk = ak , the Student test-statistic has an exact (�nite sample)distribution.

Tk �H0t(N�K )

2 The term bse �bβk� denotes the estimator of the standard error of theOLS estimator bβk and it corresponds to the square root of the k thdiagonal element of bV �bβ� (cf. chapter 3):

bV �bβ� = bσ2 �X>X��1



Consider the one-sided test:

H0 : βk = akH1 : βk < ak


W = fy : Tk (y) < Ag

where A is a constant determined by the nominal size α.

α = Pr (WjH0) = Pr�Tk (y) < AjTk �

H0t(N�K )

�



α = Pr�Tk (y) < AjTk �

H0t(N�K )

�= FN�K (A)

where FN�K (.) denotes the cdf of the Student�s t-distribution with N �Kdegrees of freedom. Denote cα the α-quantile of this distribution:.

A = F�1N�K (α) = cα

The rejection region of the test of size α is de�ned as to be:

W = fy : Tk (y) < cαg



De�nition (One-sided Student test)

The critical region of the Student test is that H0 : βk = ak is rejected infavor of H1 : βk < ak at the α (say, 5%) signi�cance level if:

W = fy : Tk (y) < cαg

where cα is the α (say, 5%) critical value of a Student t-distribution withN �K degrees of freedom and Tk (y) is the realisation of the Studenttest-statistic.



Example (One-sided test)

Consider the CAPM model (cf. chapter 1) and the following results(Eviews). We want to test the beta of MSFT as

H0 : βMSFT = 1 against H1 : βMSFT < 1

Question: give a conclusion for a nominal size of 5%.



Solution

Step 1: compute the t-statistic

TMSFT (y) =bβMSFT � 1bse �bβMSFT � =

1.9898� 10.3142

= 3.1501

Step 2: Determine the rejection region for a nominal size α = 5%.

TMSFT �H0t(20�2)

W = fy : Tk (y) < �1.7341gConclusion: for a signi�cance level of 5%, we fail to reject the nullH0 : βMSFT = 1 against H1 : βMSFT < 1 �



Solution (cont�d)

4 3 2 1 0 1 2 3 40

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Density of Ts under H0

DF = 18

Critical value = 1.7341

α=5%



Consider the one-sided test

H0 : βk = akH1 : βk > ak


W = fy : Tk (y) > Ag

where A is a constant determined by the nominal size α.

α = Pr (WjH0) = Pr�Tk (y) > AjTk �

H0t(N�K )

�



α = 1� Pr�Tk (y) < AjTk �

H0t(N�K )

�or equivalently

1� α = FN�K (A)

where FN�K (.) denotes the cdf of the Student�s t-distribution with N �Kdegrees of freedom. Denote c1�α the 1� α quantile of this distribution:

A = F�1N�K (1� α) = c1�α


W = fy : Tk (y) > c1�αg



De�nition (One-sided Student test)

The critical region of the Student test is that H0 : βk = ak is rejected infavor of H1 : βk > ak at the α (say, 5%) signi�cance level if:

W = fy : Tk (y) > c1�αg

where c1�α is the 1� α (say, 95%) critical value of a Studentt-distribution with N �K degrees of freedom and Tk (y) is the realisationof the Student test-statistic.





H0 : βMSFT = 1 against H1 : βMSFT > 1




Solution



1.9898� 10.3142

= 3.1501



W = fy : Tk (y) > 1.7341gConclusion: for a signi�cance level of 5%, we reject the nullH0 : βMSFT = 1 against H1 : βMSFT > 1 �



Solution (cont�d)

4 3 2 1 0 1 2 3 40

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5


DF = 18

Critical value = 1.7341

α=5%



Consider the two-sided test

H0 : βk = akH1 : βk 6= ak

The non-rejection region is de�ned as the intersection of the twonon-rejection regions of the one-sided test of level α/2:

W = WA \WB

H0 : βk = ak against H1 : βk < ak WA = fy : Tk (y) > cα/2gH0 : βk = ak against H1 : βk > ak WB = fy : Tk (y) < c1�α/2g



W = fy : cα/2 < Tk (y) < c1�α/2gSince the Student�s t-distribution is symmetric, cα/2 = �c1�α/2

W = fy : �c1�α/2 < Tk (y) < c1�α/2g

The rejection region is then de�ned as to be:

W = fy : jTk (y)j > c1�α/2g



De�nition (Two-sided Student test)

The critical region of the Student test is that H0 : βk = ak is rejected infavor of H1 : βk 6= ak at the α (say, 5%) signi�cance level if:

W = fy : jTk (y)j > c1�α/2g

where c1�α/2 is the 1� α/2 (say, 97.5%) critical value of a Studentt-distribution with N �K degrees of freedom and Tk (y) is the realisationof the Student test-statistic.





H0 : βMSFT = 1 against H1 : βMSFT 6= 1




Solution



1.9898� 10.3142

= 3.1501



W = fy : jTk (y)j > 2.1009gConclusion: for a signi�cance level of 5%, we reject the nullH0 : βMSFT = 1 against H1 : βMSFT 6= 1 �



4 3 2 1 0 1 2 3 40

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5


DF = 18

|Critical value| = 2.1009

α=2.5%α=2.5%



Rejection regions

H0 H1 Rejection region

βk = ak βk > ak W = fy : Tk (y) > c1�αgβk = ak βk < ak W = fy : Tk (y) < cαgβk = ak βk 6= ak W = fy : jTk (y)j > c1�α/2g

where cβ denotes the β-quantile (critical value) of the Studentt-distribution with N �K degrees of freedom.



De�nition (P-values)The p-values of Student tests are equal to:

Two-sided test: p-value = 2� FN�K (� jTk (y)j)

Right tailed test: p-value = 1� FN�K (Tk (y))Left tailed test: p-value = FN�K (�Tk (y))

where Tk (y) is the realisation of the Student test-statistic and FN�K (.)the cdf of the Student�s t-distribution with N �K degrees of freedom.


3.1. The Student testExample (One-sided test)Consider the previous CAPM model. We want to test:

H0 : c = 0 against H1 : c 6= 0

H0 : βMSFT = 0 against H1 : βMSFT 6= 0Question: �nd the corresponding p-values.



Solution

Since we consider two-sided tests with N = 20 and K = 2:

p-valuec = 2� F18 (� jTc (y)j) = 2� F18 (�0.9868) = 0.3368

p-valuec = 2� F18 (� jTMSFT (y)j) = 2� F18 (�6.3326) = 5.7e�006



Fact (Student test with large sample)For a large sample size N

Tk �H0t(N�K ) � N (0, 1)

Then, the rejection region for a Student two-sided test becomes

W =�y : jTk (y)j > Φ�1 (1� α/2)

where Φ (.) denotes the cdf of the standard normal distribution. Forα = 5%, Φ�1 (0.975) = 1.96, so we have:

W = fy : jTk (y)j > 1.96g



Case 2: Semi-parametric model



Assumption 6 (normality): the distribution of the disturbances isunknown, but satisfy (assumptions A1-A5):

E (εjX) = 0N�1

V (εjX) = σ2IN



Problem

1 The exact (�nite sample) distribution of bβk and bσ2 are unknown.2 As a consequence the �nite sample distribution of Tk (y) is alsounknown.

3 But, we can use the asymptotic properties of the OLS estimators (cf.chapter 3). In particular, we have:

pN�bβ� β

�d! N

�0, σ2Q�1

�where

Q = p lim1NX>X = EX

�xix>i

�



De�nition (Z-statistic)

Under the null H0 : βk = ak , if the assumptions A1-A5 hold (cf. chapter3), the z-statistic de�ned by

Zk =bβk � akbseasy �bβk�

d!H0N (0, 1)

where bseasy �bβk� = bσpmkk denotes the estimator of the asymptoticstandard error of the estimator bβk and mkk is k th diagonal element of�X>X

��1.



Rejection regions

The rejection regions have the same form as for the t-test (except for thedistribution)

H0 H1 Rejection region

βk = ak βk > ak W =�y : Zk (y) > Φ�1 (1� α)

βk = ak βk < ak W =

�y : Zk (y) < Φ�1 (α)

βk = ak βk 6= ak W =

�y : jZk (y)j > Φ�1 (1� α/2)

where Φ (.) denotes the cdf of the standard normal distribution.



De�nition (P-values)The p-values of the Z-tests are equal to:

Two-sided test: p-value = 2�Φ (� jZk (y)j)

right tailed test: p-value = 1�Φ (Zk (y))

left tailed test: p-value = Φ (�Zk (y))where Zk (y) is the realisation of the Z-statistic and Φ (.) the cdf of thestandard normal distribution.



Summary

Normality Assumption Non Assumption

Test-statistic t-statistic z-statistic

De�nition Tk =bβk�akbσpmkk Zk =

bβk�akbσpmkkExact distribution Tk �

H0t(N�K ) �

Asymptotic distribution � ZKd!H0N (0, 1)




Subsection 3.2

The Fisher test


3.2. The Fisher test

Consider the two-sided test associated to p linear constraints on theparameters βk :

H0 : Rβ = qH1 : Rβ 6= q

where R is a p �K matrix and q is a p � 1 vector.



Example (Linear constraints)

If K = 4 and if we want to test H0 : β1 + β2 = 0 and β2 � 3β3 = 4, thenwe have p = 2 linear constraints with:

R(2�4)

β(4,1)

= q(2�1)

�1 1 0 00 1 �3 0

�0BB@β1β2β3β4

1CCA =

�04

�




If K = 4 and if we want to test H0 : β2 = β3 = β4 = 0, then we havep = 3 linear constraints with:

R(3�4)

β(4,1)

= q(3�1)

0@ 0 1 0 00 0 1 00 0 0 1

1A0BB@

β1β2β3β4

1CCA =

0@ 000

1A



Case 1: Normality assumption A6



De�nition (Fisher test-statistic)

Under assumptions A1-A6 (cf. chapter 3), the Fisher test-statistic isde�ned as to be:

F =1p

�Rbβ� q�> �bσ2R�X>X��1 R>��1 �Rbβ� q�

where bβ denotes the OLS estimator. Under the null H0 : Rβ = q, theF -statistic has a Fisher exact (�nite sample) distribution

F �H0F(p,N�K )



Reminder

If X and Y are two independent random variables such that

X � χ2 (θ1)

Y � χ2 (θ2)

then the variable Z de�ned by

Z =X/θ1Y /θ2

has a Fisher distribution with θ1 and θ2 degrees of freedom

Z � F(θ1,θ2)



Proof

Under assumption A6, we have the following (conditional to X)distribution bβ � N �

β,σ2�X>X

��1�bσ2σ2(N �K ) � χ2 (N �K )



Proof (cont�d)

Consider the vector m = Rbβ� q. Under the nullH0 : Rβ = q

We haveE (m) = RE

�bβ�� q = Rβ� q = 0

V (m) = E

��Rbβ� q� �Rbβ� q�>�

= RV�bβ�R>

= σ2R�X>X

��1R>



Proof (cont�d)

We can base the test of H0 on the Wald criterion:

W(1�1)

= m>(1�p)

(V (m))�1p�p

mp�1

=�Rbβ� q�> �σ2R

�X>X

��1R>��1 �

Rbβ� q�Under assumption A6 (normality)

W �H0

χ2 (p)

bσ2σ2(N �K ) � χ2 (N �K )

These two variables are independent.



Proof (cont�d)

W �H0

χ2 (p)

bσ2σ2(N �K ) � χ2 (N �K )

So, the ratio of these two variables has a Fisher distribution

F =Wpbσ2

σ2(N�K )(N�K )

�H0F(p,N�K )



Proof (cont�d)

F =

�Rbβ� q�> �σ2R

�X>X

��1R>��1 �

Rbβ� q� /p

bσ2σ2(N �K ) / (N �K )

After simpli�cation, the F-statistic is de�ned by:

F =1p

�Rbβ� q�> �bσ2R�X>X��1 R>��1 �Rbβ� q�

Under the null H0 : Rβ =q :

F �H0F(p,N�K ) �




Under assumptions A1-A6 (cf. chapter 3), the Fisher test-statistic canbe de�ned as a function of the SSR of the constrained (H0) andunconstrained model (H1):

F =�SSR0 � SSR1

SSR1

��N �Kp

�where SSR0 denotes the sum of squared residuals of the constrained modelestimated under H0 and SSR1 denotes the sum of squared residuals of theunconstrained model estimated under H1.




Under assumptions A1-A6 (cf. chapter 3), the Fisher test-statistic canbe de�ned as to be:

F =1bσ2p�bβH1 � bβH0�> �X>X� �bβH1 � bβH0�

where bβH0 denotes the OLS estimator obtained in the constrained model(under H0) and bβH1 denotes the OLS estimator obtained in theunconstrained model (under H1).



De�nition (Constrained OLS estimator)

Under suitable regularity conditions, the constrained OLS estimator bβC ofβ, obtained under the constraint Rβ = q, is given by:

bβC = bβUC � �X>X��1 R> �R�X>X��1 R>��1 �RbβUC � q�where bβUC is the unconstrained OLS estimator.



Example (Fisher test and CAPM model)

Consider the extended CAPM model (�le: Chapter4_data.xls):

rMSFT ,t = β1 + β2rSP500,t + β3rFord ,t + β4rGE ,t + εt

where rMSFT ,t is the excess return for Microsoft, rSP500,t for the SP500,rFord ,t for Ford and rGE ,t for general electric. We want to test thefollowing linear constraints:

H0 : β2 = 1 and β3 = β4

Question: write a Matlab code to compute the F-statistic according tothe three alternative de�nitions.



Solution

In this problem, the null H0 : β2 = 1 and β3 = β4 can be written as:

R(2�4)

β(4,1)

= q(2�1)

�0 1 0 00 0 1 �1

�0BB@β1β2β3β4

1CCA =

�10

�









Consider the Fisher test

H0 : Rβ = qH1 : Rβ 6= q

Since the Fisher test-statistic is always positive, the rejection region isde�ned as to be:

W = fy : F (y) > Agwhere A is a constant determined by the nominal size α.

α = Pr (WjH0) = Pr�F (y) > Aj F �

H0F(p,N�K )

�



α = Pr (WjH0) = Pr�F (y) > Aj F �

H0F(p,N�K )

�or equivalently

α = 1� Pr�F (y) < Aj F �

H0F(p,N�K )

�Denote d1�α the 1� α quantile of the Fisher distribution with p andN �K degrees of freedom.

A = d1�α


W = fy : F (y) > d1�αg



De�nition (Rejection region of a Fisher test)

The critical region of the Fisher test is that H0 : Rβ = q is rejected infavor of H1 : Rβ 6= q at the α (say, 5%) signi�cance level if:

W = fy : F (y) > d1�αg

where d1�α is the 1� α critical value (say 95%) of the Fisher distributionwith p and N �K degrees of freedom and Fk (y) is the realisation of theFisher test-statistic.



Example (Fisher test and CAPM model)



where rMSFT ,t is the excess return for Microsoft, rSP500,t for the SP500,rFord ,t for Ford and rGE ,t for general electric. We want to test thefollowing linear constraints:

H0 : β2 = 1 and β3 = β4

Question: given the realisation of the Fisher test-statistic (cf. previousexample), conclude for a signi�cance level α = 5%.



Solution

Step 1: compute the F-statistic (cf. Matlab code)

F (y) = 4.3406

Step 2: Determine the rejection region for a nominal size α = 5% forN = 24, K = 4 and p = 2

F �H0F(2,20)

W = fy : F (y) > 3.4928gConclusion: for a signi�cance level of 5%, we reject the null H0 : Rβ = qagainst H1 : Rβ 6= q �



0 1 2 3 4 5 60

0.2

0.4

0.6

0.8

1

1.2

Density of F under H0

α=5%

DF1 = 2 DF2 = 20 Critical value = 3.4928



De�nition (Student test-statistic and Fisher test-statistic )Consider the test

H0 : βk = ak versus H1 : βk 6= ak

the Fisher test-statistic corresponds to the squared of the correspondingStudent�s test-statistic

F = T2k



Proof

Consider the test H0 : βk = ak against H1 : βk 6= ak , then we have:

R =�0 0 .. 1

k th position0 0

�q = ak

As a consequence :Rbβ� q = bβk � ak

bσ2R�X>X��1 R> = bV �bβk�



Proof (cont�d)

So, for a test H0 : βk = ak against H1 : βk 6= ak , the Fisher test-statisticbecomes

F =�Rbβ� q�> �bσ2R�X>X��1 R>��1 �Rbβ� q�

So, we have:

F =

�bβk � ak�2bV �bβk�and the F test-statistic is equal to the squared t-statistic:

F = T2k �



De�nition (P-values)The p-value of the F-test is equal to:

p-value = 1� Fp,N�K (F (y))

where F(y) is the realisation of the F-statistic and Fp,N�K (.) the cdf ofthe Fisher distribution with p and N �K degrees of freedom.



De�nition (Global F-test)In a multiple linear regression model with a constant term

yi = β1 +∑Kk=2 βkxik + εi

the global F-test corresponds to the test of signi�cance of all theexplicative variables:

H0 : β2 = .. = βK = 0

Under the assumption A6 (normality), the global F-test-statistic satis�es:

F �H0F(K�1,N�K )



Remarks

1 The global F-test is a test designed to see if the model is usefuloverall.

2 The null H0 : β2 = .. = βK = 0 can be written as:

R(K�1�K )

β(K ,1)

= q(K�1�1)0BBBB@

0 1 0 0 .. 00 0 1 0 .. 0.. .. 0 1 .. .... .. .. .. .. ..0 .. 0 0 .. 1

1CCCCA0BB@

β1....

βK

1CCA =

[email protected]

1CCA



Corollary (Global F-test)In a multiple linear regression model with a constant term

yi = β1 +∑Kk=2 βkxik + εi

the global F-test-statistic can also be de�ned as:

F =�

R2

1� R2��

N �KK � 1

�where R2 denotes the (unadjusted) coe¢ cient of determination.



Example (Global F-test and CAPM model)



Question: write a Matlab code to compute the global F-test, the criticalvalue for α = 5% and the p-value. Compare your results with Eviews.







Case 2: Semi-parametric model



Assumption 6 (normality): the distribution of the disturbances isunknown, but satisfy (assumptions A1-A5):

E (εjX) = 0N�1

V (εjX) = σ2IN



Problem

1 The exact (�nite sample) distribution of bβk and bσ2 are unknown. As aconsequence the �nite sample distribution of F(y) is also unknown.

2 But, we can express the F-statistic as a linear function of the Waldstatistic.

3 The Wald statistic has a chi-squared asymptotic distribution (cf. nextsection)



De�nition (F-test-statistic and Wald statistic)The Fisher test-statistic can expressed as a linear function of the Waldtest-statistic as

F =1pWald

Wald =1p

�Rbβ� q�> �R�Vasy

�bβ��1 R>��1 �Rbβ� q�Under assumptions A1-A5, the Wald test-statistic converges to achi-squared distribution

Waldd!H0

χ2 (p)



Key concepts of Section 3

1 Student test

2 Fisher test

3 t-statistic and z-statistic

4 Global F-test

5 Exact (�nite sample) distribution under the normality assumption

6 Asymptotic distribution


Section 4

MLE and Inference


4. MLE and inference

Introduction

Consider a parametric model, linear or nonlinear (GARCH, probit,logit, etc.), with a vector of parameters θ = (θ1 : .. : θK )

>

We assume that the problem is regular (cf. chapter 2) and weconsider a ML estimator bθThe �nite sample distribution of bθ is unknown, but bθ isasymptotically normally distributed (cf. chapter 2).

We want to test a set of linear or nonlinear constraints on the trueparameters (population) θ1, .., θK .



De�nition (Null hypothesis)

Consider a null hypothesis of p linear and/or nonlinear constraints

H0 : c (θ)|{z}p�1

= 0p�1

where c (θ) is a vectorial function de�ned as:

c : RK ! Rp

θ 7! c (θ)



Notations

1 c (θ) is a p � 1 vector of functions c1 (θ) , .., cp (θ):

c (θ) =

0BB@c1 (θ)c2 (θ)..

cp (θ)

1CCA2 In the case of p linear constraints, we have:

H0 : c (θ) = Rθ� q = 0




Consider the two linear constraints θ1 = θ2 + θ3 and θ2 + θ4 = 1. Wehave p = 2 constraints such that:

H0 : c (θ)(2,1)

=

�θ1 � θ2 � θ3θ2 + θ4 � 1

�=

�00

�The function c (θ) can be written as Rθ� q. For instance if K = 4 and θ

= (θ1 θ2 θ3 θ4)> , we have

c (θ) = Rθ� q =�1 �1 �1 00 1 0 1

�0BB@θ1θ2θ3θ4

1CCA�� 01

�



Example (Nonlinear constraints)Consider the linear and nonlinear constraints

θ1 � θ2 = 0 θ21 � θ3 = 0

We have p = 2 constraints such that:

H0 : c (θ)(2,1)

=

�θ1 � θ2θ21 � θ3

�=

�00

�



Assumptions

1 The functions c1 (θ) , .., cp (θ) are di¤erentiable.

2 There is no redundant constraint (identi�cation assumption).Formally, we have

(row) rank�

∂c (θ)∂θ>

�= p 8θ 2 Θ

with

∂c (θ)∂θ>(p,K )

=

0BBBBB@∂c1(θ)

∂θ1

∂c1(θ)∂θ2

..∂c1(θ)

∂θK∂c2(θ)

∂θ1

∂c2(θ)∂θ2

..∂c2(θ)

∂θK

.. .. .. ..∂cp (θ)

∂θ1

∂cp (θ)∂θ2

..∂cp (θ)

∂θK

1CCCCCAChristophe Hurlin (University of Orléans) Advanced Econometrics - Master ESA November 20, 2015 173 / 225


Consider the two-sided test

H0 : c (θ) = 0 versus H1 : c (θ) 6= 0We introduce three di¤erent asymptotic tests (the trilogy..)

1 The Likelihood Ratio (LR) test

2 The Wald test

3 The Lagrance Multiplier (LM) test



For each of the three tests, we will present:

1 the test-statistic

2 its asymptotic distribution under the null

3 the (asymptotic) rejection region

4 the (asymptotic) p-value


Subsection 4.1

The Likelihood Ratio (LR) test


4.1. The Likelihood Ratio (LR) test

De�nition (Likelihood Ratio (LR) test statistic)

The likelihood ratio (LR) test-statistic is de�ned by as to be:

LR = �2�`N�bθH0 ; y j x�� `N �bθH1 ; y j x��

where `N (θ; y j x) denotes the (conditional) log-likelihood of the sample y ,bθH0 and bθH1 are respectively the maximum likelihood estimator of θ underthe alternative and the null hypothesis.



Comments

Consider the ratio of likelihoods under H1 (no constraint) and under H0(with c (θ) = 0).

λ =LN�bθH0 ; y j x�

LN�bθH1 ; y j x�

1 λ > 0 since both likelihood are positive.2 λ < 1 since LN (H0) cannot be larger than LN (H1) . A restrictedoptimum is never superior to an unrestricted one.

3 If λ is too small, then doubt is cast on the restrictions c (θ) = 0.4 Consider the statistic LR= 2 ln (λ): if λ is "too small", then LR islarge (rejection of the null)...



De�nition (Asymptotic distribution and critical region)

Under some regularity conditions (cf. chapter 2) and under the nullH0 : c (θ) = 0, the LR test-statistic converges to a chi-squareddistribution with p degrees of freedom (the number of restrictionsimposed):

LRd!H0

χ2 (p)

The (asymptotic) critical region for a signi�cance level of α is:

W =�y : LR (y) > χ21�α (p)

where χ21�α (p) is the 1� α critical value of the chi-squared distributionwith p degrees of freedom and LR(y) is the realisation of the LRtest-statistic.



De�nition (p-value of the LRT test)The p-value of the LR test is equal to:

p-value = 1� Gp (LR (y))

where LR(y) is the realisation of the LR test-statistic and Gp (.) is the cdfof the chi-squared distribution with p degrees of freedom.



Example (LRT and Poisson distribution)Suppose that X1,X2,� � � ,XN are i.i.d. discrete random variables, such thatXi � Pois (θ) with a pmf (probability mass function) de�ned as:

Pr (Xi = xi ) =exp (�θ) θxi

xi !

where θ is an unknown parameter to estimate. We have a sample(realisation) of size N = 10 given by f5, 0, 1, 1, 0, 3, 2, 3, 4, 1g . Question:use a LR test to test the null H0 : θ = 1.8 against H1 : θ 6= 1.8 and give aconclusion for signi�cance level of 5%.



Solution

The log-likelihood function is de�ned as to be:

`N (θ; x) = �θN + ln (θ)N

∑i=1xi � ln

�N∏i=1xi !�

In the chapter 2, we found that the ML estimator of θ is the sample mean:

bθ = 1N

N

∑i=1Xi

Given the sample f5, 0, 1, 1, 0, 3, 2, 3, 4, 1g , the estimate of θ (under H1,with non constraint) is bθH1 = 2, and the corresponding log-likelihood isequal to:

`N�bθH1 ; x� = ln (0.104)



Solution (cont�d)

Under the null H0 : θ = 1.8, we don�t need to estimate θ and thelog-likelihood is equal to:

`N (θH0 ; x) = �1.8N + ln (1.8)N

∑i=1xi � ln

�N∏i=1xi !�= ln (0.0936)

The LR test-statistic is equal to:

LR (y) = �2 ln�0.09360.104

�= 0.21072



Solution (cont�d)

LR (y) = 0.21072

For N = 10, p = 1 (one restriction) and α = 0.05, the critical region is:

W =�y : LR (y) > χ20.95 (1) = 3.8415

and the p-value is

pvalue = 1� G1 (0.21072) = 0.6462

where G1 (.) is the cdf of the χ2 (1) distribution.

Conclusion: for a signi�cance level of 5%, we fail to reject the nullH0 : θ = 1.8. �


Subsection 4.2

The Wald test


4.2. The Wald test

De�nition (Wald test-statistic)

The Wald test-statistic associated to the test of H0 : c (θ) = 0 is de�nedas to be:

Wald = c�bθH1�> � ∂c

∂θ>

�bθH1� bVasy

�bθH1� ∂c∂θ>

�bθH1�>��1 c�bθH1�where bθH1 is the maximum likelihood estimator of θ under the alternative

hypothesis (unconstrained model) and bVasy

�bθH1� is an estimator of itsasymptotic variance covariance matrix.


4.2. The Wald test

Remark

Wald = c�bθH1�>| {z }1�p

0BBB@ ∂c∂θ>

�bθH1�| {z }p�K

bVasy

�bθH1�| {z }K�K

∂c∂θ>

�bθH1�>| {z }K�p

1CCCA�1

c�bθH1�| {z }p�1


4.2. The Wald test

Example (Wald test-statistic)

Consider a model with K = 3 parameters θ = (θ1 : θ2 : θ3)> with

θ1 � θ2 = 0 θ21 � θ3 = 0

We have two constraints (p = 2) and:

H0 : c (θ)(2,1)

=

�θ1 � θ2θ21 � θ3

�=

�00

�

Denote bθH1 = (θ1 : θ2 : θ3)> the ML estimator of θ under the alternative

hypothesis and bVasy

�bθH1� the estimator of its asymptotic variancecovariance matrix. Question: write the Wald test-statistic.


4.2. The Wald test

Solution

Here we have K = 3 and p = 2

c�bθH1� =

bθ1 � bθ2bθ21 � bθ3!

∂c∂θ>

�bθH1� = � 1 �1 02bθ1 0 �1

�


∂θ>

�bθH1� bVasy


�bθH1�>��1 c�bθH1�


4.2. The Wald test

Remark

In the case of linear constraints

H0 : Rθ� q = 0

we haveH0 : c (θ) = 0

withc (θ) = Rθ� q

∂c∂θ>

(θ) = R


4.2. The Wald test

De�nition (Wald test-statistic and linear constraints)

Consider the test of linear constraints H0 : c (θ) = Rθ� q = 0. TheWald test-statistic is de�ned as to be:

Wald =�RbθH1 � q�> �R bVasy

�bθH1� R>��1 �RbθH1 � q�where bθH1 is the maximum likelihood estimator of θ under the alternative

hypothesis (unconstrained model) and bVasy

�bθH1� is an estimator of itsasymptotic variance covariance matrix.


4.2. The Wald test


Under some regularity conditions (cf. chapter 2) and under the nullH0 : c (θ) = 0, the Wald test-statistic converges to a chi-squareddistribution with p degrees of freedom (the number of restrictionsimposed):

Waldd!H0

χ2 (p)


W =�y : Wald (y) > χ21�α (p)

where χ21�α (p) is the 1� α critical value of the chi-squared distributionwith p degrees of freedom and Wald(y) is the realisation of the Waldtest-statistic.


4.2. The Wald testProof

Under some regularity conditions, we have

pN�bθH1 � θ0

�d! N

�0, I�1 (θ0)

�We use the delta method for the function c (.) . The function c (.) is acontinuous and continuously di¤erentiable function not involving N, then

pN�c�bθH1�� c (θ0)� d! N

�0,

∂c∂θ>

(θ0) I�1 (θ0)∂c

∂θ>(θ0)

>�

Under the null H0 : c (θ0) = 0, we have�∂c

∂θ>(θ0) I�1 (θ0)

∂c∂θ>

(θ0)>��1/2p

Nc�bθH1� d! N (0, Ip)

where Ip is the identity matrix of size p.


4.2. The Wald testProof (cont�d)

The Wald criteria is de�ned as to be:

Wald criteria

= N � c�bθH1�>

�∂c

∂θ>(θ0) I�1 (θ0)

∂c∂θ>

(θ0)>��1/2

!>

��

∂c∂θ>

(θ0) I�1 (θ0)∂c

∂θ>(θ0)

>��1/2

� c�bθH1�

= N � c�bθH1�> �� ∂c

∂θ>(θ0) I�1 (θ0)

∂c∂θ>

(θ0)>��1

� c�bθH1�

So, under the null H0 : c (θ0) = 0,, we have

Wald criteriad!H0

χ2 (p)


4.2. The Wald test

Proof (cont�d)

Wald Criteria = N � c�bθH1�>

��

∂c∂θ>

(θ0) I�1 (θ0)∂c

∂θ>(θ0)

>��1

� c�bθH1�

A feasible Wald test-statistic is given by

Wald = N � c�bθH1�>

��

∂c∂θ>

�bθH1�bI�1 �bθH1� ∂c∂θ>

�bθH1�>��1 � c�bθH1�


4.2. The Wald test

Proof (cont�d)

Since bVasy

�bθH1� = N�1bI�1 �bθH1�We have �nally

Wald = c�bθH1�>�� ∂c

∂θ>

�bθH1� bVasy


�bθH1�>��1� c�bθH1�and

Waldd!H0

χ2 (p) �


4.2. The Wald test

De�nition (p-value of the Wald test)The p-value of the Wald test is equal to:

p-value = 1� Gp (Wald (y))

where Wald(y) is the realisation of the Wald test-statistic and Gp (.) isthe cdf of the chi-squared distribution with p degrees of freedom.


4.2. The Wald test

De�nition (z-statistic)

Consider the test H0 : θk = ak versus H1 : θk 6= ak . The z-statisticcorresponds to the square root of the Wald test-statistic and satis�es

Zk =

�bθk � ak�rbVasy

�bθk�d!H0N (0, 1)

where bθk is the ML estimator of θk obtained under H1 (unconstrainedmodel). The critical region for a signi�cance level of α is:

W =ny : jZk (y)j > Φ�1

�1� α

2

�owhere Φ (.) denotes the cdf of the standard normal distribution.


4.2. The Wald test

Computational issues

The Wald test-statistic depends on the estimator of the asymptoticvariance covariance matrix:


∂θ>

�bθH1� bVasy


�bθH1�>��1 c�bθH1�bVasy

�bθH1� = N�1bI�1 �bθH1�where I

�bθH1� denotes the average Fisher information matrix.


4.2. The Wald test

Computational issues (cont�d)

Three estimators are available for the average Fisher information matrix:

Actual Average Fisher Matrix: bIA �bθ� = 1N

N

∑i=1

bI i �bθ�

BHHH estimator: bIB �bθ� = 1N

N

∑i=1

∂ì (θ; yi j xi )

∂θ

��bθ ∂ì (θ; yi j xi )∂θ

��>bθ!

Hessian based estimator: bI c �bθ� = 1N

N

∑i=1

�� ∂2ì (θ; yi j xi )

∂θ∂θ>

��bθ�


4.2. The Wald test


1 These estimators are asymptotically equivalent, but the correspondingestimates may be very di¤erent in small samples.

2 Thus, we can obtain three di¤erent values for the Wald statisticgiven the choice of the estimator for Vasy

�bθH1� (cf. exercises).3 In general, the estimator A is rarely available and the estimator B(BHHH) gives erratic results.

4 Most of the software use the estimator C (Hessian based estimator).


4.2. The Wald test



Subsection 4.3

The Lagrange Multiplier (LM) test


4.3. The Lagrange Multiplier (LM) test

Introduction

Consider the set of constraints c (θ) = 0. Let λ be a vector of Lagrangemultipliers and de�ne the Lagrangian function

`N (θ�; y j x) = `N (θ; y j x) + λc (θ)

The solution to the constrained maximization problem is the root of

∂`N (θ�; y j x)

∂θ=

∂`N (θ; y j x)∂θ

+

�∂c (θ)∂θ>

�>λ

∂`N (θ�; y j x)

∂λ= c (θ)




∂`N (θ�; y j x)

∂θ=

∂`N (θ; y j x)∂θ

+

�∂c (θ)∂θ>

�>λ

1 If the restrictions are valid, then imposing them will not lead to asigni�cant di¤erence in the maximized value of the likelihood function.In the �rst-order conditions, the meaning is that the second term inthe derivative vector will be small. In particular, λ will be small.

2 We could test this directly, that is, test

H0 : λ = 0

which leads to the Lagrange multiplier test.




There is an equivalent simpler formulation, however. If the restrictionsc (θ) = 0 are valid, the derivatives of the log-likelihood of theunconstrained model evaluated at the restricted parameter vector willbe approximately zero.

∂`N (θ; y j x)∂θ

��bθH0 = 0The vector of �rst derivatives of the log-likelihood is the vector of(e¢ cient) scores.



De�nition (LM or score test)For these reasons, this test is called the score test as well as theLagrange multiplier test.



Guess

Let us assume that θ is scalar, i.e. K = 1, then the LM statistic is simplyde�ned as:

LM =sN�bθH0 ; Y j x�2

V�sN�bθH0 ; Y j x��

Since bIN �bθH0� = V�sN�bθH0 ; Y j x�� , we have:LM =

sN�bθH0 ; Y j x�2bIN �bθH0�



De�nition (LM or score test)The LM test-statistic or score test associated to the test of H0 :c (θ) = 0 is de�ned as to be:

LM = sN�bθH0 ; Y j x�>bI�1N �bθH0� sN �bθH0 ; Y j x�

where bθH0 is the maximum likelihood estimator of θ under the nullhypothesis (constrained model), sN (θ; Y j x) is the score vector of theunconstrained model and bIN �bθH0� is an estimator of the Fisherinformation matrix of the sample evaluated at bθH0 .



Remark

Since: bVasy

�bθH0� = bI�1N �bθH0�there is another expression for the LM statistic.



De�nition (LM or score test)The LM test-statistic or score test associated to the test of H0 :c (θ) = 0 is de�ned as to be:

LM = sN�bθH0 ; Y j x�> bVasy

�bθH0� sN �bθH0 ; Y j x�where bθH0 is the maximum likelihood estimator of θ under the nullhypothesis (constrained model), sN (θ; Y j x) is the score vector of theunconstrained model and bVasy

�bθH0� is an estimator of the asymptoticvariance covariance matrix of bθH0 .



Remark

The LM test-statistic can also be de�ned by:

LM = λ>∂c

∂θ>

�bθH0� bVasy

�bθH0�� ∂c∂θ>

�bθH0��> λ

where λ denotes the Lagrange Multiplier associated to the constraintsc (θ) = 0.



The LM test-statistic can be obtained from the following auxiliaryprocedure:

Step 1: Estimate the constrained model and obtain bθH0 .Step 2: Form the gradients for each observation of the unrestrictedmodel evaluated at bθH0

gi�bθH0 ; yi j xi� 8i = 1, ..N

Step 3: Run the regression of a vector of 1 on the variablesgi�bθH0 ; yi j xi� 8i = 1, ..N, then

LM = N � R2

where R2 denotes the (unadjusted) coe¢ cient of determination of thisauxiliary regression.



Computational issues

1 The LM test-statistic depends on the estimator of the asymptoticvariance covariance matrix:

LM = sN�bθH0 ; Y j x�> bVasy

�bθH0� sN �bθH0 ; Y j x�bVasy

�bθH0� = N�1bI�1 �bθH0�where I

�bθH0� denotes the average Fisher information matrix.2 Thus, we can obtain three di¤erent values for the LM statisticgiven the choice of the estimator for Vasy

�bθH0� (cf. exercises).




Under some regularity conditions (cf. chapter 2) and under the nullH0 : c (θ) = 0, the LM test-statistic converges to a chi-squareddistribution with p degrees of freedom (the number of restrictionsimposed):

LMd!H0

χ2 (p)


W =�y : LM (y) > χ21�α (p)

where χ21�α (p) is the 1� α critical value of the chi-squared distributionwith p degrees of freedom and LM(y) is the realisation of the LMtest-statistic.



De�nition (p-value of the LM test)The p-value of the LM test is equal to:

p-value = 1� Gp (LM (y))

where LM(y) is the realisation of the LM test-statistic and Gp (.) is thecdf of the chi-squared distribution with p degrees of freedom.


Subsection 4.4

A comparison of the three tests


4.4. A comparison of the three tests

Source: Pelgrin (2010), Lecture notes, Advanced Econometrics



Summary

Test Requires estimation under

LRT H0 and H1Wald H1LM H0



Computational problems

If the ML maximisation problem is complex (with local extrema) andif it has no closed form solution (nonlinear models: GARCH, MarkovSwitching models etc.), it may be particularly di¢ cult to get a MLestimates bθ through a numerical optimisation of the log-likelihood.If the constraints c (θ) = 0 are not valid in the data, the (numerical)convergence of the optimisation algorithm may be very problematicunder the null H0.



Asymptotic comparison

The three tests have the same asymptotic distribution under the nullH0 : c (θ) = 0:

LRTd!H0

χ2 (p)

Waldd!H0

χ2 (p)

LMd!H0

χ2 (p)



Theorem (Asymptotic comparison)The three tests are asymptotically equivalent. Under some regularityconditions and under the null H0 : c (θ) = 0, the di¤erences between thethree test statistics converge to 0 as N tends to in�nity:

LRT� LM p!H00

LRT�Wald p!H00

LM�Wald p!H00



Fact (Finite sample properties)The �nite sample properties of the three tests may be di¤erent,especially in small samples. For small sample size, they can lead toopposite conclusion about the rejection of the null hypothesis.



Key concepts of Section 4

1 Likelihood Ratio (LR) test

2 Wald test

3 Lagrange Multiplier (LM) test

4 Computational issues

5 Comparison of the three tests (the trilogy) in �nite samples


End of Chapter 4

Christophe Hurlin (University of Orléans)


Documents

Chapter 4: Statistical Hypothesis Testing · 2. Statistical hypothesis testing Objectives The objective of this section is to de–ne the following concepts: 1 Null and alternative