OLS Assumptions

8/8/2019 OLS Assumptions

1/41

The Simple Regression Model

y = F0 +F1x +u


2/41

Some Terminology

In the simple linear regression model,

wherey = F0 +F1x +u, we typically refer

to y as the

Dependent Variable, or

Left-Hand Side Variable, or

Explained Variable, or Regressand


3/41

Some Terminology, cont.

In the simple linear regression of y on x,we typically refer to x as the

Independent Variable, or Right-Hand Side Variable, or

Explanatory Variable, or

Regressor, or

Covariate, or

Control Variables


4/41

A Simple Assumption

The average value ofu, the error term, in

the population is 0. That is,

E(u) = 0

This is not a restrictive assumption, since

we can always use F0 to normalize E(u) to 0


5/41

Zero Conditional Mean

We need to make a crucial assumptionabout how u andx are related

We want it to be the case that knowingsomething about x does not give us anyinformation about u, so that they arecompletely unrelated. That is, that

E(u|x) = E(u) = 0, which implies

E(y|x) = F0 +F1x


6/41

.

.

x1

x2

E(y|x) as a linear function ofx, where for anyx

the distribution ofy is centered about E(y|x)

E(y|x) = F0 + F1x

y

f(y)


7/41

Ordinary Least Squares

Basic idea of regression is to estimate the

population parameters from a sample

Let {(xi,yi): i=1, ,n} denote a random

sample of size n from the population

For each observation in this sample, it will

be the case that

yi = F0 + F1xi + ui


8/41

.

.

.

.

y4

y1

y2

y3

x1 x2 x3 x4

}

}

{

{

u1

u2

u3

u4

x

y

Population regression line, sample data points

and the associated error terms

E(y|x) =F0 + F1x


9/41

Deriving OLS Estimates

To derive the OLS estimates we need torealize that our main assumption of E(u|x) =

E(u) = 0 also implies that

Cov(x,u) = E(xu) = 0

Why? Remember from basic probabilitythat Cov(X,Y) = E(XY) E(X)E(Y)


10/41

Deriving OLS continued

We can write our2 restrictions just in terms

ofx,y, F0 and F , since u =y F0 F1x

E(y F0 F1x) = 0

E[x(y F0 F1x)] = 0

These are called moment restrictions


11/41

Deriving OLS using M.O.M.

The method of moments approach toestimation implies imposing the population

moment restrictions on the sample moments

What does this mean? Recall that for E(X),the mean of a population distribution, asample estimator of E(X) is simply thearithmetic mean of the sample


12/41

More Derivation ofOLS

We want to choose values of the parameters that

will ensure that the sample versions of our

moment restrictions are trueThe sample versions are as follows:

1

1

1

1

1

1

!

!

!

!

n

i

iii

n

i

ii

xyxn

xyn

FF

FF


13/41


Given the definition of a sample mean, and

properties of summation, we can rewrite the first

condition as follows

xy

xy

10

10

or

,

FF

FF

!

!


14/41


!!

!!

!

!

!

!

n

i

ii

n

i

i

n

i

ii

n

i

ii

n

i

iii

xxyyxx

xxxyyx

xxyyx

1

2

1

1

1

1

1

1

11

0

F

F

FF


15/41

So the OLS estimated slope is

0thatprovided

1

2

1

2

11

"

!

!

!

!

n

i

i

n

i

i

n

i

ii

xx

xx

yyxx

F


16/41

Summary ofOLS slope estimate

The slope estimate is the sample covariancebetweenx andy divided by the sample

variance ofxIfx andy are positively correlated, theslope will be positive

Ifx andy are negatively correlated, theslope will be negative

Only needx to vary in our sample


17/41

More OLS

Intuitively, OLS is fitting a line through the

sample points such that the sum of squared

residuals is as small as possible, hence theterm least squares

The residual, , is an estimate of the error

term, u, and is the difference between thefitted line (sample regression function) and

the sample point


18/41

.

.

.

.

y4

y1

y2

y3

x1 x2 x3 x4

}

}

{

{

1

2

3

4

x

y

Sample regression line, sample data points

and the associated estimated error terms

xy1

0

FF !


19/41

Alternate approach to derivation

Given the intuitive idea of fitting a line, we can

set up a formal minimization problem

That is, we want to choose our parameters suchthat we minimize the following:

!! !n

i

ii

n

i

i xyu1

2

10

1

FF


20/41

Alternate approach, continued

If one uses calculus to solve the minimization

problem for the two parameters you obtain the

following first order conditions, which are thesame as we obtained before, multiplied by n

0

0

1

10

1

10

!

!

!

!

n

i

iii

n

i

ii

xyx

xy

FF

FF


21/41

Algebraic Properties ofOLS

The sum of the OLS residuals is zero

Thus, the sample average of the OLS

residuals is zero as well

The sample covariance between the

regressors and the OLS residuals is zero

The OLS regression line always goes

through the mean of the sample


22/41

Algebraic Properties (precise)

xy

ux

n

u

u

n

iii

n

i

in

i

i

10

1

1

1

0

0

thus,and0

FF !

!

!!

!

!

!


23/41

More terminology

TThen

( )squaresosumresidualtheis

( )squaresosumexplainedtheis

( T)squaresosumtotaltheis

:ollo ingthede inethenWe

part,dunexplaineanandpart,explainedanoup

madebeingasnobservatioeachocan thinkWe

2

2

2

!

!

i

i

i

iii

u

yy

yy

uyy


24/41

Proof that SST = SSE + SSR

? A

? A

!!

!

!

!

0thatknoeand

SSE2SSR

2

22

2

22

yyu

yyu

yyyyuu

yyu

yyyyyy

ii

ii

iiii

ii

iiii


25/41

Goodness-of-Fit

How do we think about how well oursample regression line fits our sample data?

Can compute the fraction of the total sumof squares (SST) that is explained by themodel, call this the R-squared of regression

R2 = SSE/SST = 1 SSR/SST


26/41

Using Stata forOLS regressions

Now that weve derived the formula for

calculating the OLS estimates of our

parameters, youll be happy to know youdont have to compute them by hand

Regressions in Stata are very simple, to run

the regression of y on x, just typereg y x


27/41

Unbiasedness ofOLS

Assume the population model is linear inparameters asy = F0 + F1x + u

Assume we can use a random sample ofsize n, {(xi, yi): i=1, 2, , n}, from thepopulation model. Thus we can write thesample modelyi = F0 + F1xi + uiAssume E(u|x) = 0 and thus E(ui|xi) = 0

Assume there is variation in thexi


28/41

Unbiasedness ofOLS (cont)

In order to think about unbiasedness, we need to

rewrite our estimator in terms of the population

parameterStart with a simple rewrite of the formula as

|

!22

2 where,

xxs

s

yxx

ix

x

ii

F


29/41


ii

iii

ii

iii

iiiii

uxxxxxxx

uxx

xxxxx

uxxxyxx

!

!!

10

10

10

FF

FF

FF


30/41


211

2

1

2

thusand,

asrewrittenbecannumeratorthe,so

,0

x

ii

iix

iii

i

s

uxx

uxxs

xxxxx

xx

!

!

!

FF

F


31/41


1211

21

1

then,1

thatso,let

FFF

FF

!

!

!

!

iix

iix

i

ii

uEds

E

uds

xxd


32/41

Unbiasedness Summary

The OLS estimates ofF1 and F0 areunbiased

Proof of unbiasedness depends on our 4assumptions if any assumption fails, thenOLS is not necessarily unbiased

Remember unbiasedness is a description ofthe estimator in a given sample we may benear or far from the true parameter


33/41

Variance of the OLS Estimators

Now we know that the samplingdistribution of our estimate is centered

around the true parameterWant to think about how spread out thisdistribution is

Much easier to think about this varianceunder an additional assumption, so

Assume Var(u|x) = W2 (Homoskedasticity)


34/41


35/41

.

.

x1

x2

Homoskedastic Case

E(y|x) = F0 + F1x

y

f(y|x)


36/41

.

xx1 x2

f(y|x)

Heteroskedastic Case

x3

.

.

E(y|x) = F0 + F1x


37/41

Variance ofOLS (cont)

12

22

2

2

2

2

2

2

222

2

2

2

2

2

2

2

211

1

11

11

1

FWW

WW

FF

Vars

ss

d

s

d

s

uVards

udVars

uds

VarVar

xx

x

i

x

i

x

iix

iix

iix

!!

!

!

!

!

!

!


38/41

Variance ofOLS Summary

The larger the error variance, W2, the larger

the variance of the slope estimate

The larger the variability in thexi, thesmaller the variance of the slope estimate

As a result, a larger sample size should

decrease the variance of the slope estimateProblem that the error variance is unknown


39/41

Estimating the Error Variance

We dont know what the error variance, W2,

is, because we dont observe the errors, ui

What we observe are the residuals, i

We can use the residuals to form anestimate of the error variance


40/41

Error Variance Estimate (cont)

2/

2

1

isofestimatorunbiasedanThen,

22

2

1100

1010

10

!

!

!!

!

nSSRun

u

xux

xyu

i

i

iii

iii

W

W

FFFFFFFF

FF


41/41

Error Variance Estimate (cont)

2121

1

2

/se

,

oerrorstandardthe

havethen eorsubstituteei

sdthatrecall

regressiontheoerrortandard

!

!

!!

xx

s

i

x

WFF

WW

WF

WW

Documents

OLS Assumptions