Upload
nicholas-horn
View
214
Download
0
Embed Size (px)
Citation preview
Multiple Linear Regression
A method for analyzing the effects of several predictor variables concurrently.- Simultaneously- Stepwise
Minimizing the squared deviations from a plane.
y b x b x b xi i 1 1 2 2
We will begin be focusing on Simultaneous Regression.
The regression coefficients for each predictor is estimated while holding the other predictor variables constant. Thus, The slope for a particular predictor variable may change with the presence of different co-predictors or when used as a solitary predictor.
After testing bivariate assumptions, there may remain multivariate outliers.That is, outliers based on a combination of scores. For example being 6 feet tallwill not make one an outlier, nor will being 120 pounds. Being 6 feet tall and being120 lbs will make one an outlier.
Distance: based on residuals, identifying outliers in the criterionLeverage: identifying outliers in the predictors (multivariate)Influence: combines distance and leverage to identify unusually influentialobservations. Cook”s D measures how much change there would be in the SSSlope would occur if that single observation was removed. (Is calculated for each observation).
Tolerance: the degree to which a predictor can be predicted by the other predictors.Singular: occurs when a predictor is perfectly predictable from the other predictors.
Standardized regression Coefficients
This is sometimes related to the question of the relative importance of the predictors.
Remember, slope is sensitive to units of measurement. Thus, larger units produce smaller values than will the same angle but in smaller units of measurement.
For example, if x is measured in seconds, the value of the slope will be smaller than if x were measured in minutes.
Standardized Coefficients (beta) measure the change in the criterion (now measured in standard deviations) that is produced by a one standarddeviation change in the predictor.
Determining which predictor is more important is not merely a matter of comparing the betas, as some textbooks may suggest. There are theoretical and practical mattersto be considered. Additionally, there is the matter of the variances found in the predictors across differing samples, i.e., the standard deviations may change.
Adding predictors may change the regression coefficients and the betas.
R
R 2
is the multiple correlation coefficient. It measures the degree of association (-1.0 to 1.) between the criterion and the predictor variable taken simultaneously.
is the coefficient of multiple determination. It indicates the percentage of thevariance in the criterion variable accounted for by the predictors taken together.
Adding additional predictor variables will never reduce the coefficient of multipledetermination, but it doesn’t necessarily mean that the added predictors are eitherstatistically or theoretically important. (An additional variable may fail to add to the coefficient of multiple determination only if it is uncorrelated to the criterion.
R ad j
2Just as r is adjusted for n, so R 2or R 2 can be adjusted for the
number of predictor variables used in the regression.
Standard Error of the Estimated Coefficient
is a measure of the variability that would be found among the different slopes estimatedfrom other samples drawn from the same population. (n held constant)]•This is analogous to the standard error of the mean and serves an analogous purpose.
One way to look at the standard error of b is to see it as a measure of how sensitive the slope is to a change in a small number of data points from the sample.
A1 A2
B1 B2
C1 C2
In the three panels (A,B,C) the data points are fixedFrom left to right except for the five larger points.
In the cases of A and B, a few changes in datapoints produced large changes in the slopes. This is not the case with C. Why?
The variability in the x variable is greater in the case of C. Furthermore, the correlationis greater in case of C. (this latter means thatthe standard error will be smaller.
Standard Error of the Slope
ss
s Nb
y x
x
.
1
sy y
Ny x.
( )
2
2
with
In multiple regression we may wish to test a hypothesis concerning all of thepredictors or some subset of the predictors, in addition to tests of the individual slopes.
t-tests of individual coefficients, with all other predictors held constantF-tests of whether taken together the predictors are a significant predictor of the criterionF-tests of whether some subset of predictors is significant
y b x b x a 1 1 2 2
Example of strange outcomes.
t-test of b1 may be non-significantt-test of b2 may be non-significantF-test of b1, b2 may be significant
When two predictors are correlated, the standard errors of the coefficients arelarger than they would be in the absence of the other predictor variable.
FR
R
N k
k
2
21
1
(
( ) FN p R
p R
( )
( )
1
1
2
2Or
k and p = # of predictors
df = p(or k), (N-p(or k)-1)
Limitations of test of significance: both of individual predictors and the overall model
If there are small differences in the betas of the various predictors, different patterns of significance may easily arise from another sample. The relative variabilities of the predictors may change.
A significant beta does NOT necessarily mean that the variable is of theoretical orof practical importance. The issue of IMPORTANCE is a difficult one. The relative sizeof the betas is not always the solution. (For example, your ability to manipulate a variablemay be as important an issue in practical terms.)
Difference between two R 2
FR R k k
R N kl er sm a ller l er sm a ller
l er lareger
( ) ( )
( )( )arg arg
arg
2 2
21 1
K = # of variable in the regression
Types of Analysis: Data types
Cross-sectional: cases represent different objects at one point in time
Time-Series: same object and variables are tested over time
- a lagged dependent variable (criterion) (value at previous time) canbe used as an independent variable (predictor)
Continuous versus dummy variables
Dummy variables: categorical, binary, dichotomous (0 and 1).
There may be more than two categories. For example there may be four categories.This would produce three dummy variables. Let us say that there are four types ofpeople: A, B, C, and D. There would be three variables: A (yes/no, 0/1), B(0/1)C(0/1) and all zeros would make the fourth category a yes and is reflected in the intercept.
Interactions: derived variables
Between two continuous variables or between one continuous andone dummy variable.
y a b x b x b x x 1 1 2 2 3 1 2
If x1 is the continuous variable, then b1 tell us its effect on the criterion when x2 = 0.B1 + b3 will tell us the effect when x2 =1. B3 tell us the difference in the two slopes.
For two continuous variable, the additional interaction term will indicate if the effect of x1 at low values of x2 is greater or less than its effect at higher valuesof x2.
Remember, adding additional predictor variables, even interaction terms,can change the betas of all other predictors.
Basic issues
If you omit a relevant variable(s) from your regression, the betas of the included variables will be at best unreliable and at worst invalid.
If you include irrelevant predictor variable, the betas of the other relevant variables remain unbiased but, however, to the extent that this irrelevant variableis correlated with some of the other predictors, it will increase the size of theStandard Errors (reduce power).
If the underlying function between one or more of the predictors and the criterionis something other than linear, then the betas will be biased and unreliable. This is one reason why it is important to look at all bivariate plots prior to the analysis.
Addressing Collinearilty
Ideally, you should collect new data that is free of multiple collinearity. This usually requires an experimental design (creating true independent variables).This is usually not feasible or it would have been done in the first place.
1. Model Respecification: combining correlated variable through various techniques or choosing remove some. (Theoretical & Statistical)
2. Statistical Variable Selectiona. Step-wise procedures: can be deceptive and often fails to maximize b. Examine all subsets: may reveal subsets with similar
but the resulting solution may not fit with either the research question or the theoretical approach.
R 2