Chapter 1 Linear Regression with One Predictor Variable

Chapter 1

Linear Regression with One Predictor Variable

許湘伶

Applied Linear Regression Models(Kutner, Nachtsheim, Neter, Li)

hsuhl (NUK) LR Chap 1 1 / 52

迴迴迴歸歸歸分分分析析析

Regression analysis is a statistical methodology that utilizes therelation between two or more quantitative variables so that aresponse or outcome variable can be predicted from the other, orothers.

迴歸分析（Regression Analysis）是一種統計學上分析數據的方法

目的在於了解兩個或多個變數間是否相關、相關方向與強度，並建立數學模型以便觀察特定變數來預測研究者感興趣的變數。 (Wiki)


迴迴迴歸歸歸分分分析析析 (cont.)

起源：「迴歸」一詞最早由法蘭西斯·高爾頓（Francis Galton）所使用。

法蘭西斯·高爾頓爵士（1822年2月16日–1911年1月17日）：

查爾斯·達爾文的表弟

是一名英格蘭維多利亞時代的文藝復興人、人類學家、優生學家、熱帶探險家、地理學家、發明家、氣象學家、統計學家、心理學家和遺傳學家。

一生中發表了超過340篇的報告和書籍在1883年率先使用「優生學」（eugenics）一詞。在他於1869年的著作《遺傳的天才》（Hereditary Genius）中，高爾頓主張人類的才能是能夠透過遺傳延續的。



在統計學方面的貢獻：在1877年發表關於種子的研究結果，指出迴歸到平均值（regression toward the mean）現象的存在，這個概念與現代統計學中的「回歸」並不相同，但是卻是回歸一詞的起源。

在此後的研究中，高爾頓第一次使用了相關係數（correlationcoefficient）的概念。他使用字母「r」來表示相關係數。同時他也發表了關於指紋的論文和書籍，被認為對於現代利用指紋進行犯罪搜查方面有很大的貢獻。

他曾對親子間的身高做研究，發現父母的身高雖然會遺傳給子女，但子女的身高卻有逐漸「迴歸到中等（即人的平均值）」(regression to the mean)的現象。



「向平均迴歸」(regression to the mean)現象:非常高的父母所生的子女，往往比父母矮些，而非常矮的雙親所生的孩子，則往往比父母親高。將人的身高從高、矮兩個極端往所有人類的平均值拉。(統計改變了世界)



business: 產品銷售(銷售vs. 廣告支出)

social science

biological sciences

engineering

chemical science

economics

management etc.


Relations between Variable

Functional Relation between Two Variables

functional relation vs. statistical relation

工讀時數 vs. 時薪(120元)

Y = dollar sales of a product

X = a fixed price and number of unitssold



Functional Relation between Two Variables (cont.)

If the selling price is $2 per unit, Y = 2X

(a) price (b) data

Figure : Example of Functional Relation (Y = f (X))




The observations for a statistical relation do not fall directly onthe curve of relationship.

Ex: Employees performance evaluationsY = Year-end evaluations; X = midyear evaluations




Which kind of pattern can be observed in the figure?

What kind of the relation between midyear and year-endevaluations?




the higher the midyearevaluation⇒ the higher tendsto be the year-end evaluation

the relationship is not aperfect one:some of the variation inyear-end evaluation is notaccount for (說明) bymid-year performanceassessments




(a): scatter plot

(b): Plotting a line of relationship to describe the statisticalrelation between X and Y




Figure : Curvilinear Statistical Relationbetween Age and Steroid(膽固醇) Level inHealthy Females Aged 8 to 25.





Regression Models and Their Uses

Basic Concepts

A regression model is a formal means of expressing the two essentialingredients of a statistical relation:

A tendency(趨勢) of the response variable Y to vary with thepredictor variable X in a systematic fashion(系統的方式).

A scattering of points around the curve of statistical relationship.



Basic Concepts (cont.)

A regression model: (Characteristics)

A probability distribution of Y for each level of X

The means of the probability distributions vary in somesystematic fashion with X.

Figure : Pictorial Representation of Regression Model



Construction of Regression Models

Y: the dependent or response variable

X: the independent, explanatory or predictor variable

Construction:

Selection of Predictor Variables: may more than one predictorvariable

Functional Form of Regression Relation: linear, quadraticregression functions . . .

Scope(範圍) of Model: some interval or region of values of thepredictor variables; The shape of the regression function outsidethe range would be in serious doubt.



Extrapolation

(a) extrapolation (b) danger



Uses

Three major purposes:1 description (描繪)

2 control (控制)

3 prediction (預測)



Causality

Regression and Causality (因果關係):

The existence of a statistical relation between Y and X does notimply that Y depends causally (有因果關係地 ) on X.(迴歸並不意味著存在因果關係，即解釋變數是因，反應變數是果)Correlation（關係） 6=Causation（因果）：

有關係不代表有因果關係。已知X、Y之間有相關，可用已知的X來預測Y，但不表示X的改變，定會造成Y有特定的改變。收入 vs. 肥胖 (兩者皆與年紀有關)吸菸 vs. 癌症速度 vs. 車禍次數 (德國高速公路路網的52%無速限（有130km/h的建議速限）)


Simple Linear Regression Model with Distribution of Error TermsUnspecified

Statement of Model

The linear regression function with one predictor variable:

Yi = β0 + β1Xi + εi, i = 1, . . . , n

Yi: the value of response variable in the ith trial

β0, β1: parameters

Xi: a known constant; the value of the predictor variable in the ithtrial

εi: random error term; E{εi} = 0; σ2{εi} = σ2; uncorrelatedσ{εi, εj} = 0 ∀i, j(i 6= j)

simple, linear in the parameters; linear in the predictor variable



Features

1 Yi: the sum of two components:(1) β0 + β1Xi

(2) εi

2 E{εi} = 0:

E{Yi} = E{β0 + β1Xi + εi} = β0 + β1Xi

3 The regression function:

E{Y} = β0 + β1X

(The regression function relates the means of the probabilitydistribution of Y for given X to the level of X.



Features (cont.)

1 Yi in the ith trial exceeds or falls short of the value of theregression function by the error term amount εi

2 σ2{εi} = σ2:σ2{Yi} = σ2

(σ2{β0 + β1Xi + εi} = σ2{εi} = σ2

)3 The error terms are assumed to be uncorrelated, so are the

responses Yi and Yj.



Features (cont.)

Figure : Illustration of Simple Linear Regression Model



Meaning of Regression Parameters

Regression coefficients:

β0(intercept截距), β1(slope斜率)

Figure : Meaning of Parameters of Simple Linear Regression Model



Alternative modification

1 Original regression model:

Yi =β0 + β1Xi + εi, i = 1, . . . , n

2

Yi =β0X0 + β1Xi + εi (X0 ≡ 1)

3

Yi =β0 + β1Xi + εi

=β0 + β1(Xi − X) + β1X + εi

=(β0 + β1X) + β1(Xi − X) + εi

=β∗0 + β1(Xi − X) + εi



Data from Regression Analysis

Unknown the regression parameters β0, β1

Estimate parameters from relevant data

Rely on an analysis of the data for developing a suitableregression model



Figure : Typical Strategy for Regression Analysis


Estimation of Regression Function

Estimate: Method of Least Squares

Example

Xi: the age of the subject

Yi: the number of attempts to accomplish the task beforegiving up



Estimate: Method of Least Squares (cont.)

Observations: (Xi,Yi), i = 1, . . . , n

Deviation (偏差):　Yi − β0 − β1Xi




The least square criterion:

Q =n∑

i=1

(Yi − β0 − β1Xi)2

The property of “Good” estimators?

The least squares estimators b0, b1 minimize the criterion Q forthe given sample observations.How to obtain the estimators b0, b1?

Numerical search procedures until the ones that minimize Q arefound (like Figure 1.9)Analytical procedures



Estimate: Method of Least Squares

問題：

一單變數函數 f (x)

如何找尋臨界點(critical point)?若臨界點為(x0)如何判斷此臨界點之特性 (區域極大、區域極小等)?

一二變數函數 f (x, y)

如何找尋臨界點(critical point)?若臨界點為(x0, y0)如何判斷此臨界點之特性 (區域極大、區域極小等)?

HW(2016.09.19): local minimum之判斷方式




∂Q∂β0

∣∣∣b0,b1

= 0

∂Q∂β1

∣∣∣b0,b1

= 0

⇒b1 =

∑ni=1(Xi − X)(Yi − Y)∑n

i=1(Xi − X)2

b0 = Y − b1X

X, Y: the means of the Xi and Yi, respectivelyHW(2016.09.19): b0, b1 為Q之極小值



Data on Lot Size and Work Hours

Toluca Companymanufactures refrigeration equipment(製冷設備) as well as manyreplacement parts.

In the past, one of the replacement parts has been producedperiodically in lots of varying sizes.

cost improvement program: wished to determine the optimum lotsize for producing this part

One key input: to ascertain the optimum lot size is the relationshipbetween lot size and labour hours required to produce the lot

25 recent production runs were utilized

Table 1.1(next page) contains the data on lot size and work hours.(批貨皆為10的倍數)



Data on Lot Size and Work Hours (cont.)

refrigeration equipmentmanufacturing

Xi: Lot Size (i = 1, . . . , 25)

Yi: Labor work hours

b1 = 3.5702; b0 = 62.37



Data on Lot Size and Work Hours (cont.)

R codes

> toluca = read.table("toluca.txt",header=T)> plot(toluca$Size, toluca$Hrs,xlab="Size",ylab="Hours",pch=16)> toluca$xdif = toluca$Size-mean(toluca$Size)> toluca$ydif = toluca$Hrs-mean(toluca$Hrs)> toluca$crp = toluca$xdif * toluca$ydif> toluca$xsq = toluca$xdifˆ2> # Parameter Estimates> (b1 = sum(toluca$crp)/sum(toluca$xsq))[1] 3.570202> (b0 = mean(toluca$Hrs) - b1 * mean(toluca$Size))[1] 62.36586



Normal equations

∂Q∂β0

∣∣∣b0,b1

= 0

∂Q∂β1

∣∣∣b0,b1

= 0⇒{ ∑

Yi = nb0 + b1∑

Xi∑XiYi = b0

∑Xi + b1

∑X2

i

A test of the second partial derivatives will show that a minimumis obtained with the least squares estimators b0 and b1.

How to verify that b0 & b1 minimize Q?



Property of Least Squares Estimators (Chap 2)

Gauss-Markov theorem

Unbiased:E{b0} = β0; E{b1} = β1

Gauss-Markov theorem

Under the conditions of regression model (1.1), the LSEs b0 & b1

in (1.10) are unbiased and minimum variance among all unbiasedestimators.

將於第二章證明



Property of Least Squares Estimators (Chap 2) (cont.)

More precise:The estimators b0 and b1 are more precise than any otherestimators belonging to the class of unbiased estimators that arelinear functions of the observations Y1, . . . ,Yn.

Linear function of Yi:(Chap. 2)

b1 =∑

kiYi

b0, b1 are linear estimators.



Point Estimation of Mean Response

response: a value of the response variable

Mean response: E{Y} = β0 + β1X

Estimated regression function:

Y = b0 + b1X

(Y: the value of the estimated regression function at X of thepredictor variable)

Y : an unbiased estimator of E{Y}Fitted value Yi:

Yi = b0 + b1Xi, i = 1, . . . , n



Residuals

Residual: ei

ei = Yi − Yi = Yi − (b0 + b1Xi)

is the vertical deviation of Yi from the fitted value Yi on theestimated regression line, and it is known.

Model error term: εi

εi = Yi − E{Y}

the vertical deviation of Yi from the unknown true regression lineand is unknown.



Properties of Fitted Regression Line

The sum of the residuals is zero:n∑

i=1

ei = 0

(Rounded errors may be presented.)

The sum of the squared residuals is a minimum:∑n

i=1 e2i

∵ the criterion Q to be minimized equals∑n

i=1 e2i when b0, b1 are

used for estimating β0, β1

The sum of the observed values Yi equals the sum of the fittedvalues Yi:

n∑i=1

Yi =n∑

i=1

Yi



Properties of Fitted Regression Line (cont.)

The sum of the weighted residuals is zero when the residual in theith trial is weighted by the level of the predictor variable in theithe trial:

n∑i=1

Xiei = 0

The sum of the weighted residuals is zero when the residual in theith trial is weighted by the fitted value of the response variable forthe ith trial:

n∑i=1

Yiei = 0

The regression line always goes through the point (X, Y)



Estimation of σ2

(Method I)σ2{Yi} = σ2: s2 =∑n

i=1(Yi−Y)2

n−1 (n− 1為自由度)

(Method II)The error sum of squares or residual sum of squares:SSE

SSE =n∑

i=1

(Yi − Yi)2 =

n∑i=1

e2i

The residual sum of squares SSE has n− 2 degrees of freedom.(Two degrees of freedom are associated with the estimates b0 andb1 involved in obtaining Yi)

E{SSE} = (n− 2)σ2 (need to be proof)



Estimation of σ2 (cont.)

The error mean square or residual mean square: MSE

MSE =SSE

n− 2=

∑ni=1(Yi − Yi)

2

n− 2=

∑e2

i

n− 2

MSE is an unbiased estimator of σ2:

E{MSE} = σ2

An estimate of σ =√

MSER codes

> # Fitted values, residuals, R-squared> toluca$yhat = b0 + b1*toluca$Size> toluca$residual = toluca$Hrs - toluca$yhat> (mse = sum(toluca$residualˆ2)/(25-2))[1] 2383.716> (hsig = sqrt(mse))[1] 48.82331


Normal Error Regression Model


(Interval estimate & making tests)

The normal error regression model:

Yi = β0 + β1Xi + εi

Yi: the observation responseXi: a known constantβ0, β1: parametersεi, i = 1, . . . , n: independent N(0, σ2)

常態分佈的特性?

The estimators of the parameters β0, β1 and σ2 van be estimatedbe the method of maximum likelihood. (MLE)



Normal Error Regression Model (cont.)

(Illustration) The method of maximum likelihood chooses as themaximum likelihood estimate that value for which the likelihoodvalue is largest.

µ = 230 µ = 259f (Y1) = f1 0.005399 0.026609f (Y2) = f2 0.000087 0.033322f (Y3) = f3 0.000595 0.039894

L(µ) =∏

fi 0.279×10−9 0.0000354

Estimator of µ is the sample mean Y

µ = Y = (250 + 265 + 259)/3 = 258




Two methods for finding MLE:a systematic numerical searchuse of an analytical solution (MLE之步驟?)

Valuef (Y1|σ = 2.5, β0 = 0, β1 = 0.5) = f1 0.021596f (Y2|σ = 2.5, β0 = 0, β1 = 0.5) = f2 0.7175× 10−9

f (Y3|σ = 2.5, β0 = 0, β1 = 0.5) = f3 0.21596L(µ) =

∏fi 0.3346× 10−12




The density of an observation Yi for the normal error regressionmodel:(E{Yi} = β0 + β1Xi;σ

2{Yi} = σ2)

fi =1√2πσ

exp

[−1

2

(Yi − β0 − β1Xi

σ

)2]

The likelihood function for n observations Y1, . . . ,Yn:

L(β0, β1, σ2) =

n∏i=1

fi =n∏

i=1

1√2π

exp

[−1

2

(Yi − β0 − β1Xi

σ

)2]

=1

(2πσ2)n/2 exp

[− 1

2σ2

n∑i=1

(Yi − β0 − β1Xi)2

]hsuhl (NUK) LR Chap 1 49 / 52



The MLE of σ2 is biased.

MSE =n

n− 2σ2

Ex: β0 = b0 = 2.81; β1 = b1 = 0.177



Normal Error Regression Model (cont.)R codes

> # install.packages("stats4")> library(stats4)> x = toluca$Size> y = toluca$Hrs> LogL <- function(b00,b11,sigma) {+ res = y - b00 - x * b11+ ff = dnorm(res,0, sigma)+ -sum(log(ff))+ }> fit <-mle(minuslogl = LogL, start = list(b00 = 62,b11=3.5, sigma=hsig))> summary(fit)Maximum likelihood estimationCall:mle(minuslogl = LogL, start = list(b00 = 62, b11 = 3.5, sigma = hsig))

Coefficients:Estimate Std. Error

b00 62.029377 25.1085865b11 3.574344 0.3328047sigma 46.829487 6.6226616

-2 log L: 263.273> c(lm(toluca$Hrs˜toluca$Size)$coefficients, summary(lm(toluca$Hrs˜toluca$Size))$sigma)(Intercept) toluca$Size

62.365859 3.570202 48.823310

LSE: b0 = 3.5702; b1 = 62.37; σ = 48.82331hsuhl (NUK) LR Chap 1 51 / 52


Normal Error Regression Model (cont.)R codes

> # Simulation Illustration> n=10000> x= runif(n,min(toluca$Size),max(toluca$Size))> y= b0 + b1*x + rnorm(n,0,sqrt(mse))> LogL <- function(b11,b00,sigma) {+ res = y - b00 - x * b11+ ff = dnorm(res, 0, sigma)+ -sum(log(ff))+ }> fit <-mle(minuslogl = LogL, start = list(b00 = 62,b11=3.5, sigma=hsig ))> summary(fit)Maximum likelihood estimationCall:mle(minuslogl = LogL, start = list(b00 = 62, b11 = 3.5, sigma = hsig))

Coefficients:Estimate Std. Error

b11 3.538036 0.01666386b00 63.812279 1.26496190sigma 48.170379 0.34061774

-2 log L: 105873.6> c(lm(y˜x)$coefficients, summary(lm(y˜x))$sigma)(Intercept) x

63.812217 3.538036 48.175037


Documents

Chapter 1 Linear Regression with One Predictor Variable