ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Applied linear statistical models: Anoverview
Gunnar Stefansson
1Dept. of MathematicsUniv. Iceland
August 27, 2010
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Outline
The course
Simple linear regressionFittingSome theoryModel evaluation
Multiple linear regressionMatricesSome more theoryApplied stuff
Other things
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Some basics
Course: Applied linear statistical modelsThis lecture: A description of the material to be coveredduring the course and of the framework, where materialis, etc.Course material is at
I Content+quizzes http://tutor-web.net→http://vr3pc109.rhi.hi.is/
I The bookI Other handouts (weekly homeworks, 3-4 projects
(40%), copies from Scheffe etc)I Instructor’s home page: http://www.hi.is/∼gunnar
(data sets,links, this file)I The Univ. Icel. web system (announcements/ email)
Important: Read your e-mail for announcements!Warning: The tutor-web material is under development!
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
The homework
weekly homework, on handouts (possibly within tutorials):
I on-line quizzesI from bookI other exercisesI modify wikipediaI write examples etc for tutor-web
3-4 projects (30%)first handout: tutor-web, “STAT 645 smplreg” – do notprint yetR stuff: tutor-web, “STAT 150 R – do not print yetmath stuff: tutor-web, “MATH 612 ccas
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Outline
The course
Simple linear regressionFittingSome theoryModel evaluation
Multiple linear regressionMatricesSome more theoryApplied stuff
Other things
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Fitting a line to data: STATS 310 smplreg
Simple linear regression: Haven pairs (x1, y1), . . . , (xn, yn) andwant ”best” fitting line.Estimation (OLS):
S(α, β) =∑
i
(yi − (α + βxi))2
Minimize S over α, β to get
a = y − bx
b =
∑i(xi − x)(yi − y)∑
i(xi − x)2
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
1500 2000 2500 3000 3500 4000
400
500
600
700
800
900
1000
Capelin biomass
Cod
gro
wth
Example: Cod growth vs capelin
biomass.
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
A formal statistical model
Fixed numbers, xiRandom variables:Yi ∼ n(α + βxi , σ
2)or: Yi = α + βxi + εiwith εi ∼ n(0, σ2) in-dependent and identicallydistributed (i.i.d.)The data:
yi = α + βxi + ei
So: yi -values are out-comes of the random vari-ables Yi , but xi -values areconstants.
0 1 2 3 4 5
05
1015
20
xy
Usual regression assumptions.
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
The assumptions
The assumptions (which may all fail) are:
I x-values are constants (no error)I linearityI constant varianceI GaussianI Independence
Will test these and modify accordinglyExamples: Fish growth (nonlinear); Bird counts(nonnormal); fuel consumption (heteroscedastic); stockprices (autocorrelated)
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Outline
The course
Simple linear regressionFittingSome theoryModel evaluation
Multiple linear regressionMatricesSome more theoryApplied stuff
Other things
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Some distributionsUnivariate Gaussian density
f (y) =1√2πσ
e−(y−µ)2
2σ2 y ∈ Rn
Product of univariate Gaussian densities, for i.i.d.Gaussians
f (y1, . . . , yn) =∏
i
1√2πσ
e−(yi−µ)2
2σ2
=1
(2π)n/2σn e−P
i(yi−µ)2
2σ2 y1, . . . , yn ∈ R
The general multivariate Gaussian case
f (x) =1
(2π)n/2|Σ|12
e−12 (x−µ)′Σ−1(x−µ) x ∈ Rn
(is a density on Rn if Σ > 0 and µ ∈ Rn)
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Linear combinations of Gaussian randomvariables
Linear combinations (aX + bY , c′Y etc) of Gaussianrandom variables (independent or jointly multivariateGaussian) are also Gaussian.E [aX + bY ] = aµX + bµYV [aX + bY ] = a2σ2
X + b2σ2Y if independent
V [aX + bY ] = a2σ2X + b2σ2
Y + 2abCov(X ,Y ) in generalE [c′Y] = c′µV [c′Y] = c′ΣY cV [AY] = AΣY A′
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Distributions of estimatesThe number b should be viewed as the outcome of therandom variable,
β =
∑i(xi − x)Yi∑i(xi − x)2
(note the rewrite from earlier formula). So β is a linearcombination of Gaussian Y1, . . . ,Yn so β is Gaussian.Will show that E [β] = β, find V [β] = σ2
β= . . . and
β − βσ2β
∼ tn−2
Inference: Hypothesis testing and confidence intervals.Example: Test whether slope is zero (H0 : β = 0) i.e.whether there is some relationship between x and y .Never be content with a mathematical or conceptualmodel: Insist that it is justified by data!
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Outline
The course
Simple linear regressionFittingSome theoryModel evaluation
Multiple linear regressionMatricesSome more theoryApplied stuff
Other things
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Goodness of fit and diagnostics: STATS 310xxxVerifying SLR assumptions...Will derive tests for nonlin-earity (lack-of-fit), normal-ity, homoscedasticity, in-dependence, outliers, etcetc.This is (mainly) based onresiduals ei = yi − yi orvariations thereofSome concepts: Stan-dardized residuals,studentized residuals,deleted residuals etc etc.
●
●●●●●●
●
●●
●●●
●●●●
●●
●●
●
●●
●●●
●●●●●
●●
●●
●
●
●●
●●
●
●●
●
●●●
●
0 10 20 30 40 5020
4060
80
x
y
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0 10 20 30 40 50
−2
02
4
x
resi
d(fm
)
Simplest diagnostics: Plot residuals in all possi-
ble ways
Note: It is never enough to fit a model or use it forpredictions. One must always also verify whether themodel is adequate.
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Outline
The course
Simple linear regressionFittingSome theoryModel evaluation
Multiple linear regressionMatricesSome more theoryApplied stuff
Other things
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
SLR in matrix form
y ∈ Rn = vector of measurements
X =
1 x1...
...1 xn
the “X-matrix”
min∑
(yi − (α+ βxi))2 is equiv-alent to finding
β =
(αβ
)to mininmize ||y− Xβ||2Number notation: y = Xβ + e
Point estimate as a projection
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Multiple regression: STATS 310 mulreg
The model:y = Xβ + e
where X in an n × p matrix.Example: Interaction model: yi = α + βxi + γwi + δxiwi .Defining xi1 = 1, xi2 = xi , xi3 = wi , xi4 = xiwi , thisbecomes a multiple linear regression model.More examples: Estimate single intercept, many slopes;Test whether multiple lines are all parallel; ...Used in all fields of biology (fisheries, genetics, ...),economics, psychology, sociology, ...Need to develop point estimates, methods of validationand testing.
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Matrix solution
The point estimate is
β = (X′X)−1X′Y
which is always unbiased (if the mean of the Y -s iscorrect)
E[β]
= β
and has variance-covariance matrix
V[β]
= σ2(X′X)−1.
(if the variance assumptions are correct)and is multivariate Gaussian (if the Y -values areGaussian).
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Outline
The course
Simple linear regressionFittingSome theoryModel evaluation
Multiple linear regressionMatricesSome more theoryApplied stuff
Other things
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
TestingStatistical tests (of modelreductions) can use pro-jections in Rn where o.n.bases give SSEs and d.f.
y = ζ1u1 + . . . ζquq
+ζq+1uq+1 + . . . ζr ur
+ζr+1ur+1 + . . . ζnun
SSE(F ) = ||y− Xβ||2 =n∑
i=p+1
ζ2i
SSE(F )− SSE(R) = ||Zγ − Xβ||2 =
p∑i=r+1
ζ2i
SSE(R) = ||y− Zγ||2 =n∑
i=r+1
ζ2i
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Estimable functions: STATS 310 xxx
In the one-way layout not all parameter combinations canbe estimated. Those linear combinations of parameters,ψ = c′β which have an unbiased estimate using a linearcombination of y -valules are termed estimable functions.
y︸︷︷︸n×1
∼ n( X︸︷︷︸n×p
β︸︷︷︸p×1
, σ2 I︸︷︷︸n×n
)
ψi = c′iβ; C = (c1, . . . , cq)′; ψ = Cβ
ψ = Ay = Cβ ∼ n(Cβ, σ2AA′)
Theorem: ψ ∼ n(ψ,Σψ
), ||y−Xβ||2
σ2 ∼ χ2n−r and these
two quantities are independent.
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Outline
The course
Simple linear regressionFittingSome theoryModel evaluation
Multiple linear regressionMatricesSome more theoryApplied stuff
Other things
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Model selection
Many x-variables?Need to choose subset for inclusionLook at all subsets?How should quality of fit be measured?Forward and backwards stepwise regression.Example: What drives recruitment to fish stocks? Have aseries of e.g. 50 years of data, but several dozens ofpossible x-values.
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Model validation
Validate mainly using var-ious residualsInvestigate assumptions,but now also investigateinfluential observationsDFFITS etcHat matrix H, particularlythe leverage values hii
H = X(X′X
)−1 X′
projects y to yInvestigate collinearity
●
●
●●●●
●●
●
●
●
0 10 20 30 40 50
−2
02
46
810
●
●
●
●● ●
●
●
●
●
●
●
●
● ●
●
●●
●
●
5 10 15 20 25
−1
01
23
45
6
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●●
●●●
●
●●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●
●●
●
●●●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
0 2 4 6 8 10
02
46
8
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●●
●
●
●
●
●
●
●●
●●●
●
●
●
●●●●
●
●
●
●●
●●●
●
●●●●●●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●
●●●
●
●
●
●●●
●
●
●
●
●●●●
●●●●
●
●●
●
●●●●
●
●
●
●
●●
●●●
●●●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
●●●
●
●●
●
●●
0 50 100 200 300
−2
02
46
8
Examples of problem cases
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Analysis of variance: STATS 310 anova
One-way layout: yij = µ+ αi + eijThis is a particular linear model (multiple regressionmodel), but with special properties! Will developexpressions for solutions, sums of squares etc etc.Example: Breast feeding and IQGroup n IQbar sI 90 92.8 15.2 not breast fedIIa 17 94.8 19.0 failed breast feedingIIb 193 103.7 15.3 breast fed
More examples: Fertilizers, crops; differential geneexpressions; ...Two-way layout etc etc
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Multiple comparisons procedures: ScheffeTheorem: Under the above assumptions and definitions,(
ψ −ψ)′
B−1(ψ −ψ
)/q
||y− Xβ||2/(n − r)∼ Fq,n−r
Pβ
[(ψ −ψ
)′B−1
(ψ −ψ
)≤ qs2Fq,n−r ,1−α
]= 1− α
If Ψ = {ψ = k1ψ1 + . . .+ kqψq}, then
P[ψ −
√qF ∗σψ ≤ ψ ≤ ψ +
√qF ∗σψ ∀ψ ∈ Ψ
]= 1− α
and we are therefore allowed to search among allestimable functions within the set to find significanteffects.Can now do legal data-snooping!
ALSM
GS
The course
Simple linearregressionFitting
Some theory
Model evaluation
Multiple linearregressionMatrices
Some more theory
Applied stuff
Other things
Generalizations and other topics
I The Generalized linear model (GLM) drops theassumption of normality and defines a “link function”
I The Generalized Additive Model (GAM) allowssmoothing function in place of linear combinations.
I Random effects modelsI Nonlinear modelsI Correlated observations