6-1

Lecture 6

Regression Diagnostics

STAT 512

Spring 2011

Background Reading

KNNL: 3.1-3.6

6-2

Topic Overview

Chapter 3 – Diagnostics & Remedial Measures

for Simple Linear Regression

Diagnostics: Look at the data to diagnose

situations where the assumptions of our model are

violated. (Lecture 6)

Remedies: Changes in analytic strategy to fix

these problems. (Lecture 7)

6-3

What Do We Need To Check?

Main Assumptions: Errors are independent,

normal random variables with common

variance 2

Does the assumption of linearity make

sense?

Were any important predictors excluded

from the model?

6-4

What Do We Need To Check?

Are there “outlying” values for the predictor

variables (X) that could unduly influence

the regression model?

Are there outliers? (Generally the term

outlier refers to a response that is vastly

different from other responses (Y) – see

KNNL pg 108)

How to Get Started? - Look at the Data!

6-5

Diagnostics for Predictors (X)

We do not make any specific assumptions

about X. However, understanding what is

going on with X is necessary to

interpreting what is going on with the

response (Y).

So, we can look at some basic summaries of

the X variables to get oriented.

However, we are not checking our

assumptions at this point.

6-6

Diagnostics for Predictors (X)

Dot plots, Stem-and-leaf plots, Box plots,

and Histograms can be useful in

identifying potential outlying observations

in X. Note that just because it is an

outlying observation does not mean it will

create a problem in the analysis. However

it is a data point that will probably have

higher influence over the regression

estimates.

Sequence plots can be useful for identifying

potential problems with independence.

6-7

SAS Procedures

PROC UNIVARIATE for getting basic

statistics and creating histograms for both

response and predictor variables. Check

for outliers, unusual skewness, clumping.

PROC GPLOT to create a scatter plot of X

against Y. Assess linearity visually.

6-8

Reminder – Scatterplot

6-9

UNIVARIATE Procedure (1)

(06_misc.sas)

PROC UNIVARIATE data=muscle plot;

var age;

histogram age / normal (mu=est sigma=est);

title 'Histogram for Age';

RUN; Basic Statistical Measures

Location Variability

Mean 59.98333 Std Deviation 11.79700

Median 60.00000 Variance 139.16921

Mode 78.00000 Range 37.00000

Interquartile Range 20.50000

6-10

UNIVARIATE Procedure (2)

Stem Leaf # Boxplot

78 00000 5 |

76 0000 4 |

74 0 1 |

72 000 3 |

70 000 3 +-----+

68 000 3 | |

66 0 1 | |

64 0000 4 | |

62 000 3 | |

60 0000 4 *--+--*

58 000 3 | |

56 0000 4 | |

54 000 3 | |

52 000 3 | |

50 0 1 | |

48 00 2 +-----+

46 0000 4 |

44 00 2 |

42 0000 4 |

40 000

----+----+----+----+

6-11

UNIVARIATE Procedure (3)

6-12

UNIVARIATE Procedure (4)

What if we add in a data point for:

age=100, mmass =40?

Stem Leaf # Boxplot

10 0 1 0

9

9

8

8

7 5666788888 10 |

7 001223 6 +-----+

6 5556889 7 | |

6 00013334 8 *--+--*

5 56777999 8 | |

5 123344 6 +-----+

4 5677788 7 |

4 11122334 8 |

----+----+----+----+

Multiply Stem.Leaf by 10**+1

6-13

UNIVARIATE Procedure (5)

6-14

Diagnostics for Residuals (1)

Basic Distributional Assumptions on Errors

Model: Yi = β0 + β1Xi + εi

o Where 2~ 0,iid

i N (i.e., the εi are

independent, normal, and have constant

variance).

The ei (residuals) should be similar to the εi

How do we check this? Plot the Residuals!

6-15

Diagnostics for Residuals (2)

Basic Questions addressed by diagnostics

for residuals

o Is the relationship linear?

o Does the variance depend on X?

o Are the errors normal?

o Are the errors independent?

o Are their outliers?

o Are any important predictors omitted?

6-16

Checking Linearity

Plot Y vs. X (scatterplot)

Plot e vs X (or Y ) - residual plot

Generally can see from a scatter plot when a

relationship is nonlinear

Patterns in residual plots can emphasize

deviations from linear pattern

6-17

Checking Constant Variance

Plot e vs X (or Y ) - residual plot

Patterns suggest issues!

Megaphone shape indicates

increasing/decreasing variance with X

Other shapes can indicate non-linearity

Outliers show up in obvious way

6-18

SAS Code PROC REG data=muscle;

model mmass=age;

output out=diag p=pred r=resid;

RUN;

*Plot residuals vs age;

symbol1 v=dot i=none;

PROC GPLOT data=diag;

plot resid*age;

title 'Residuals for Muscle Mass Data';

run;

6-19

6-20

6-21

6-22

6-23

Checking for Normality Plot residuals in a Normal Probability Plot

o Compare residuals to their expected value

under normality (normal quantiles)

o Should be linear IF normal

Plot residuals in a Histogram

PROC UNIVARIATE is used for both of

these

Book shows method to do this by hand –

you do not need to worry about having to

do that.

6-24

SAS Code

PROC REG data=muscle;

model mmass=age;

output out=diag p=pred r=resid;

RUN;

*Check normality assumption;

PROC UNIVARIATE data=diag normal;

var resid;

histogram resid /normal(mu=est sigma=est);

qqplot resid /normal;

title 'Check for Normality';

RUN;

6-25

6-26

6-27

Normality Plot

Outliers show up in a quite obvious way.

Non-normal distributions can look very

wacky.

Symmetric / Heavy tailed distributions show

an “S” shape.

Skewed distributions show exponential

looking curves (see figure 3.9)

6-28

6-29

6-30

- 4 - 3 - 2 - 1 0 1 2 3 4

- 100000

- 50000

0

50000

100000

150000

R

e

s

i

d

u

a

l

Nor mal Quant i l es

6-31

Checking Independence

Sequence Plot: Residuals against time/order

Patterns suggest non-independence

See figure 3.8 in KNNL.

6-32

Additional Predictors

Plot residuals against other potential

predictors (not predictors from the model)

Patterns indicate an important predictor that

maybe should be in the model.

Example: Suppose we use a muscle mass

dataset that includes both men and women.

6-33

6-34

Residuals vs Age

Plot looks great, right?

But what happens if we separate male and

female?

PROC GPLOT data=diag;

plot resid*age=gender /overlay;

RUN;

6-35

6-36

Additional Predictors

Seems like gender is also an important

predictor of muscle mass (note that gender

is categorical, so we’ll have to wait until

later in the semester for further analysis)

For continuous variables, you look for a

linear pattern with a non-zero slope.

6-37

Summary of Diagnostic Plots You will have noticed that the same plots are

used for checking more than one assumption.

These are your basic tools.

o Plot Y vs. X (check for linearity, outliers)

o Plot Residuals vs. X (check for constant

variance, outliers, linearity)

o Normal Probability Plot and/or

Histogram of residuals (normality, outliers)

If it makes sense, consider also doing a

sequence plot of the residuals (independence)

6-38

Plots vs. Significance Tests If you are uncertain what to conclude after

examining the plots, you may additionally wish

to perform hypothesis tests for model

assumptions (normality, homogeneity of

variance, independence).

These tests are not a replacement for the plots,

but rather a supplement to them.

Note of caution: Plots are more likely to

suggest a remedy and significance test results

are very dependent on sample size.

6-39

Significance Tests for

Model Assumptions

Constancy of Variance:

o Brown-Forsythe (modified Levene)

o Breusch-Pagan

Normality

o Kolmogorov-Smirnov, etc.

Independence of Errors:

o Durbin-Watson Test

6-40

Tests for Normality

PROC UNIVARIATE data=diag normal;

var resid;

Tests for Normality

Test --Statistic--- -----p Value------

Shapiro-Wilk W 0.979585 Pr < W 0.4112

Kolmogorov-Smirnov D 0.079433 Pr > D >0.1500

Cramer-von Mises W-Sq 0.057805 Pr > W-Sq >0.2500

Anderson-Darling A-Sq 0.383556 Pr > A-Sq >0.2500

Small p-values indicate non-normality

6-41

Upcoming in Lecture 7...

Remedial Measures: What to do when there

is a problem with your model assumptions

(KNNL: 3.8-3.11)