Upload
ann-black
View
224
Download
0
Tags:
Embed Size (px)
Citation preview
Model selection
Stepwise regression
Statement of problem
• A common problem is that there is a large set of candidate predictor variables.
• (Note: The examples herein are really not that large.)
• Goal is to choose a small subset from the larger set so that the resulting regression model is simple, yet have good predictive ability.
Example: Cement data
• Response y: heat evolved in calories during hardening of cement on a per gram basis
• Predictor x1: % of tricalcium aluminate
• Predictor x2: % of tricalcium silicate
• Predictor x3: % of tetracalcium alumino ferrite
• Predictor x4: % of dicalcium silicate
Example: Cement data
83.35
105.05
6
16
37.25
59.75
8.75
18.25
83.35
105.0
5
19.5
46.5
6 1637.25
59.75 8.7
518.25 19
.546.5
y
x1
x2
x3
x4
Two basic methods of selecting predictors
• Stepwise regression: Enter and remove predictors, in a stepwise manner, until there is no justifiable reason to enter or remove more.
• Best subsets regression: Select the subset of predictors that do the best at meeting some well-defined objective criterion.
Stepwise regression: the idea
• Start with no predictors in the “stepwise model.”
• At each step, enter or remove a predictor based on partial F-tests (that is, the t-tests).
• Stop when no more predictors can be justifiably entered or removed from the stepwise model.
Stepwise regression: Preliminary steps
1. Specify an Alpha-to-Enter (αE = 0.15) significance level.
2. Specify an Alpha-to-Remove (αR = 0.15) significance level.
Stepwise regression: Step #1
1. Fit each of the one-predictor models, that is, regress y on x1, regress y on x2, … regress y on xp-1.
2. The first predictor put in the stepwise model is the predictor that has the smallest t-test P-value (below αE = 0.15).
3. If no P-value < 0.15, stop.
Stepwise regression:Step #2
1. Suppose x1 was the “best” one predictor.
2. Fit each of the two-predictor models with x1 in the model, that is, regress y on (x1, x2), regress y on (x1, x3), …, and y on (x1, xp-1).
3. The second predictor put in stepwise model is the predictor that has the smallest t-test P-value (below αE = 0.15).
4. If no P-value < 0.15, stop.
Stepwise regression:Step #2 (continued)
1. Suppose x2 was the “best” second predictor.
2. Step back and check P-value for β1 = 0. If the P-value for β1 = 0 has become not significant (above αR = 0.15), remove x1 from the stepwise model.
Stepwise regression:Step #3
1. Suppose both x1 and x2 made it into the two-predictor stepwise model.
2. Fit each of the three-predictor models with x1 and x2 in the model, that is, regress y on (x1, x2, x3), regress y on (x1, x2, x4), …, and regress y on (x1, x2, xp-1).
Stepwise regression:Step #3 (continued)
1. The third predictor put in stepwise model is the predictor that has the smallest t-test P-value (below αE = 0.15).
2. If no P-value < 0.15, stop.
3. Step back and check P-values for β1 = 0 and β2 = 0. If either P-value has become not significant (above αR = 0.15), remove the predictor from the stepwise model.
Stepwise regression:Stopping the procedure
• The procedure is stopped when adding an additional predictor does not yield a t-test P-value below αE = 0.15.
Example: Cement data
83.35
105.05
6
16
37.25
59.75
8.75
18.25
83.35
105.0
5
19.5
46.5
6 1637.25
59.75 8.7
518.25 19
.546.5
y
x1
x2
x3
x4
Predictor Coef SE Coef T PConstant 81.479 4.927 16.54 0.000x1 1.8687 0.5264 3.55 0.005
Predictor Coef SE Coef T PConstant 57.424 8.491 6.76 0.000x2 0.7891 0.1684 4.69 0.001
Predictor Coef SE Coef T PConstant 110.203 7.948 13.87 0.000x3 -1.2558 0.5984 -2.10 0.060
Predictor Coef SE Coef T PConstant 117.568 5.262 22.34 0.000x4 -0.7382 0.1546 -4.77 0.001
Predictor Coef SE Coef T PConstant 103.097 2.124 48.54 0.000x4 -0.61395 0.04864 -12.62 0.000x1 1.4400 0.1384 10.40 0.000
Predictor Coef SE Coef T PConstant 94.16 56.63 1.66 0.127x4 -0.4569 0.6960 -0.66 0.526x2 0.3109 0.7486 0.42 0.687
Predictor Coef SE Coef T PConstant 131.282 3.275 40.09 0.000x4 -0.72460 0.07233 -10.02 0.000x3 -1.1999 0.1890 -6.35 0.000
Predictor Coef SE Coef T PConstant 71.65 14.14 5.07 0.001x4 -0.2365 0.1733 -1.37 0.205x1 1.4519 0.1170 12.41 0.000x2 0.4161 0.1856 2.24 0.052
Predictor Coef SE Coef T PConstant 111.684 4.562 24.48 0.000x4 -0.64280 0.04454 -14.43 0.000x1 1.0519 0.2237 4.70 0.001x3 -0.4100 0.1992 -2.06 0.070
Predictor Coef SE Coef T PConstant 52.577 2.286 23.00 0.000x1 1.4683 0.1213 12.10 0.000x2 0.66225 0.04585 14.44 0.000
Predictor Coef SE Coef T PConstant 71.65 14.14 5.07 0.001x1 1.4519 0.1170 12.41 0.000x2 0.4161 0.1856 2.24 0.052x4 -0.2365 0.1733 -1.37 0.205
Predictor Coef SE Coef T PConstant 48.194 3.913 12.32 0.000x1 1.6959 0.2046 8.29 0.000x2 0.65691 0.04423 14.85 0.000x3 0.2500 0.1847 1.35 0.209
Predictor Coef SE Coef T PConstant 52.577 2.286 23.00 0.000x1 1.4683 0.1213 12.10 0.000x2 0.66225 0.04585 14.44 0.000
Stepwise Regression: y versus x1, x2, x3, x4 Alpha-to-Enter: 0.15 Alpha-to-Remove: 0.15 Response is y on 4 predictors, with N = 13
Step 1 2 3 4Constant 117.57 103.10 71.65 52.58
x4 -0.738 -0.614 -0.237 T-Value -4.77 -12.62 -1.37 P-Value 0.001 0.000 0.205
x1 1.44 1.45 1.47T-Value 10.40 12.41 12.10P-Value 0.000 0.000 0.000
x2 0.416 0.662T-Value 2.24 14.44P-Value 0.052 0.000
S 8.96 2.73 2.31 2.41R-Sq 67.45 97.25 98.23 97.87R-Sq(adj) 64.50 96.70 97.64 97.44C-p 138.7 5.5 3.0 2.7
Caution about stepwise regression!
• Do not jump to the conclusion …– that all the important predictor variables for
predicting y have been identified, or– that all the unimportant predictor variables have
been eliminated.
Caution about stepwise regression!
• Many t-tests for βk = 0 are conducted in a stepwise regression procedure.
• The probability is high … – that we included some unimportant predictors– that we excluded some important predictors
Drawbacks of stepwise regression
• The final model is not guaranteed to be optimal in any specified sense.
• The procedure yields a single final model, although in practice there are often several equally good models.
• It doesn’t take into account a researcher’s knowledge about the predictors.
Example: Modeling PIQ
130.5
91.5
100.728
86.283
73.25
65.75
130.591.
5
170.5
127.5
100.72
886.
283 73.25
65.75
170.5
127.5
PIQ
MRI
Height
Weight
Stepwise Regression: PIQ versus MRI, Height, Weight Alpha-to-Enter: 0.15 Alpha-to-Remove: 0.15 Response is PIQ on 3 predictors, with N = 38
Step 1 2Constant 4.652 111.276
MRI 1.18 2.06T-Value 2.45 3.77P-Value 0.019 0.001
Height -2.73T-Value -2.75P-Value 0.009
S 21.2 19.5R-Sq 14.27 29.49R-Sq(adj) 11.89 25.46C-p 7.3 2.0
The regression equation isPIQ = 111 + 2.06 MRI - 2.73 Height
Predictor Coef SE Coef T PConstant 111.28 55.87 1.99 0.054MRI 2.0606 0.5466 3.77 0.001Height -2.7299 0.9932 -2.75 0.009
S = 19.51 R-Sq = 29.5% R-Sq(adj) = 25.5%
Analysis of Variance
Source DF SS MS F PRegression 2 5572.7 2786.4 7.32 0.002Error 35 13321.8 380.6Total 37 18894.6
Source DF Seq SSMRI 1 2697.1Height 1 2875.6
Example: Modeling BP
120
110
53.25
47.75
97.325
89.375
2.125
1.875
8.275
4.425
72.5
65.5
120110
76.25
30.75
53.25
47.75
97.325
89.375 2.1
251.8
758.2
754.4
25 72.5
65.5
76.25
30.75
BP
Age
Weight
BSA
Duration
Pulse
Stress
Stepwise Regression: BP versus Age, Weight, BSA, Duration, Pulse, Stress Alpha-to-Enter: 0.15 Alpha-to-Remove: 0.15 Response is BP on 6 predictors, with N = 20
Step 1 2 3Constant 2.205 -16.579 -13.667
Weight 1.201 1.033 0.906T-Value 12.92 33.15 18.49P-Value 0.000 0.000 0.000
Age 0.708 0.702T-Value 13.23 15.96P-Value 0.000 0.000
BSA 4.6T-Value 3.04P-Value 0.008
S 1.74 0.533 0.437R-Sq 90.26 99.14 99.45R-Sq(adj) 89.72 99.04 99.35C-p 312.8 15.1 6.4
The regression equation isBP = - 13.7 + 0.702 Age + 0.906 Weight + 4.63 BSA
Predictor Coef SE Coef T PConstant -13.667 2.647 -5.16 0.000Age 0.70162 0.04396 15.96 0.000Weight 0.90582 0.04899 18.49 0.000BSA 4.627 1.521 3.04 0.008S = 0.4370 R-Sq = 99.5% R-Sq(adj) = 99.4%
Analysis of Variance
Source DF SS MS F PRegression 3 556.94 185.65 971.93 0.000Error 16 3.06 0.19Total 19 560.00
Source DF Seq SSAge 1 243.27Weight 1 311.91BSA 1 1.77
Stepwise regression in Minitab
• Stat >> Regression >> Stepwise …
• Specify response and all possible predictors.
• If desired, specify predictors that must be included in every model. (This is where researcher’s knowledge helps!!)
• Select OK. Results appear in session window.