OPIM 303-Lecture #8

Jose M. Cruz

Assistant Professor

Session 8 - Overview

• Simple Regression Model• Determining the best fit• “Goodness of Fit”

– R2

– Confidence Intervals– Hypothesis tests– Residual Analysis

Purpose of Regression Analysis

• Regression analysis is used primarily to model causality and provide prediction– Predicts the value of a dependent (response)

variable based on the value of at least one independent (explanatory) variable

– Explains the effect of the independent variables on the dependent variable

Types of Regression Models

Positive Linear Relationship

Negative Linear Relationship

Relationship NOT Linear

No Relationship

Simple Linear Regression Model

• Relationship between variables is described by a linear function

• The change of one variable causes the change in the other variable

• A dependency of one variable on the other

PopulationRegressionLine (conditional mean)

Population Linear Regression

Population regression line is a straight line that describes the dependence of the average value (conditional mean)average value (conditional mean) of one variable on the other

Population Y intercept

Population SlopeCoefficient

Random Error

Dependent (Response) Variable

Independent (Explanatory) Variable

ii iY X

YX

Population Linear Regression

(continued)

ii iY X

= Random Error

Y

X

(Observed Value of Y) =

Observed Value of Y

YX iX

i

(Conditional Mean)

Sample regression line provides an estimateestimate of the population regression line as well as a predicted value of Y

Sample Linear Regression

Sample Y Intercept

SampleSlopeCoefficient

Residual0 1i iib bY X e

0 1Y b b X Sample Regression Line (Fitted Regression Line, Predicted Value)

Sample Linear Regression

• and are obtained by finding the values of and that minimizes the sum of the squared residuals

• provides an estimateestimate of • provides and estimateestimate of

0b 1b 0b1b

0b

1b

(continued)

22

1 1

ˆn n

i i ii i

Y Y e

Sample Linear Regression

(continued)

Y

XObserved Value

YX iX

i

ii iY X

0 1i iY b b X

ie

0 1i iib bY X e 1b

0b

Interpretation of the Slope and the Intercept

• is the average value of Y when

the value of X is zero.

• measures the change in the

average value of Y as a result of a one-unit

change in X.

| 0E Y X

1

|E Y X

X

• is the estimatedestimated average

value of Y when the value of X is zero.

• is the estimatedestimated change in

the average value of Y as a result of a one-unit

change in X.

(continued)

ˆ | 0b E Y X

1

ˆ |E Y Xb

X

Interpretation of the Slope and the Intercept

Simple Linear Regression: Example

You want to examine the linear dependency of the annual sales of produce stores on their size in square footage. Sample data for seven stores were obtained. Find the equation of the straight line that fits the data best.

Annual Store Square Sales

Feet ($1000)

1 1,726 3,681

2 1,542 3,395

3 2,816 6,653

4 5,555 9,543

5 1,292 3,318

6 2,208 5,563

7 1,313 3,760

Scatter Diagram: Example

0

2000

4000

6000

8000

10000

12000

0 1000 2000 3000 4000 5000 6000

Square Feet

An

nu

al

Sa

les

($00

0)

Excel Output

Equation for the Sample Regression Line: Example

0 1ˆ

1636.415 1.487i i

i

Y b b X

X

From Excel Printout:

CoefficientsIntercept 1636.414726X Variable 1 1.486633657

Excel Output

Regression Statistics

Multiple R 0.970557

R Square 0.941981

Adjusted R Square 0.930378

Standard Error 611.7515

Observations 7

ANOVA

  df SS MS FSignificance

F

Regression 1 30380456 30380456 81.17909 0.000281

Residual 5 1871200 374239.9

Total 6 32251656      

 Coefficient

sStandard

Error t Stat P-value Lower 95% Upper 95%Intercept 1636.415 451.4953 3.624433 0.015149 475.8109 2797.019X Variable 1 1.486634 0.164999 9.009944 0.000281 1.06249 1.910777

Graph of the Sample Regression Line: Example

0

2000

4000

6000

8000

10000

12000

0 1000 2000 3000 4000 5000 6000

Square Feet

An

nu

al

Sa

les

($00

0)

Y i = 1636.415 +1.487X i

Interpretation of Results: Example

The slope of 1.487 means that for each increase of one unit in X, we predict the average of Y to increase by an estimated 1.487 units.

The model estimates that for each increase of one square foot in the size of the store, the expected annual sales are predicted to increase by $1487.

ˆ 1636.415 1.487i iY X

How Good is the regression?

• R2

• Residual Plots• Analysis of Variance• Confidence Intervals• Hypothesis (t) tests

Coefficient of Correlation

• Measures the strength of the linear relationship between two quantitative variables

1

2 2

1 1

n

i ii

n n

i ii i

X X Y Yr

X X Y Y

The Coefficient of Determination

• Denoted by R2

• Measures the proportion of variation in Y that is explained by the independent variable X in the regression model

Coefficients of Determination (r 2) and Correlation (r)

r2 = 1, r2 = 1,

r2 = .8, r2 = 0,Y

Yi = b0 + b1Xi

X

^

YYi = b0 + b1Xi

X

^Y

Yi = b0 + b1Xi

X

^

Y

Yi = b0 + b1Xi

X

^

r = +1 r = -1

r = +0.9 r = 0

Linear Regression Assumptions

1. Linearity

2. Normality– Y values are normally distributed for each X– Probability distribution of error is normal

2. Homoscedasticity (Constant Variance)

3. Independence of Errors

Residual Analysis

• Purposes– Examine linearity – Evaluate violations of assumptions

• Graphical Analysis of Residuals– Plot residuals vs. Xi , Yi and time

Residual Analysis for Linearity

Not Linear Linear

X

e eX

Y

X

Y

X

• Y values are normally distributed around the regression line.

• For each X value, the “spread” or variance around the regression line is the same.

Variation of Errors around the Regression Line

X1

X2

X

Y

f(e)

Sample Regression Line

Residual Analysis for Homoscedasticity

Heteroscedasticity Homoscedasticity

SR

X

SR

X

Y

X X

Y

Residual Plot

0 1000 2000 3000 4000 5000 6000

Square Feet

Residual Analysis:Excel Output for Produce Stores Example

Excel Output

Observation Predicted Y Residuals1 4202.344417 -521.34441732 3928.803824 -533.80382453 5822.775103 830.22489714 9894.664688 -351.66468825 3557.14541 -239.14541036 4918.90184 644.09816037 3588.364717 171.6352829

Residual Analysis for Independence

Not Independent Independente e

TimeTime

Residual is plotted against time to detect any autocorrelation

No Particular PatternCyclical Pattern

Graphical Approach

The ANOVA Table in Excel

ANOVA

df SS MS FSignificance

F

Regression p SSRMSR

=SSR/pMSR/MSE

P-value of

the F Test

Residuals n-p-1 SSEMSE

=SSE/(n-p-1)

Total n-1 SST

Measures of VariationThe Sum of Squares: Example

Excel Output for Produce Stores

ANOVA

df SS MS F Significance F

Regression 1 30380456.12 30380456 81.17909 0.000281201

Residual 5 1871199.595 374239.92

Total 6 32251655.71

Measures of Variation: Produce Store Example

Regression StatisticsMultiple R 0.9705572R Square 0.94198129Adjusted R Square 0.93037754Standard Error 611.751517Observations 7

Excel Output for Produce Stores

r2 = .94

94% of the variation in annual sales can be explained by the variability in the size of the store as measured by square footage

Inference about the Slope: t Test

• t test for a population slope– Is there a linear dependency of Y on X ?

• Null and alternative hypotheses– H0: 1 = 0 (no linear dependency)– H1: 1 0 (linear dependency)

• Test statistic–

–

1

1

1 1

2

1

where

( )

YXb n

bi

i

b St S

SX X

. . 2d f n

Example: Produce Store

Data for Seven Stores: Estimated Regression Equation:

The slope of this model is 1.487.

Is square footage of the store affecting its annual sales?

Annual Store Square Sales

Feet ($000)

1 1,726 3,681

2 1,542 3,395

3 2,816 6,653

4 5,555 9,543

5 1,292 3,318

6 2,208 5,563

7 1,313 3,760

Yi = 1636.415 +1.487Xi

Inferences about the Slope: t Test Example

H0: 1 = 0

H1: 1 0

.05

df 7 - 2 = 5

Critical Value(s):

Test Statistic:

Decision:

Conclusion:There is evidence that square footage affects annual sales.

t0 2.5706-2.5706

.025

Reject Reject

.025

From Excel Printout

Reject H0

Coefficients Standard Error t Stat P-valueIntercept 1636.4147 451.4953 3.6244 0.01515Footage 1.4866 0.1650 9.0099 0.00028

1b 1bS t

Inferences about the Slope: Confidence Interval Example

Confidence Interval Estimate of the Slope:

11 2n bb t S Excel Printout for Produce Stores

At 95% level of confidence, the confidence interval for the slope is (1.062, 1.911). Does not include 0.

Conclusion: There is a significant linear dependency of annual sales on the size of the store.

Lower 95% Upper 95%Intercept 475.810926 2797.01853X Variable 11.06249037 1.91077694

Confidence Intervals for Estimators

Regression Statistics

Multiple R 0.970557

R Square 0.941981

Adjusted R Square 0.930378

Standard Error 611.7515

Observations 7

ANOVA

  df SS MS FSignificance

F

Regression 1 30380456 30380456 81.17909 0.000281

Residual 5 1871200 374239.9

Total 6 32251656      

 Coefficient

sStandard

Error t Stat P-value Lower 95% Upper 95%Intercept 1636.415 451.4953 3.624433 0.015149 475.8109 2797.019X Variable 1 1.486634 0.164999 9.009944 0.000281 1.06249 1.910777