Engineering Analysis ENG 3420 Fall 2009

Dan C. Marinescu

Office: HEC 439 B

Office hours: Tu-Th 11:00-12:00

22Lecture 22

Lecture 22 Attention: The last homework HW5 and the last project are due on

Tuesday November 24!! Last time:

Linear regression Exponential, power, and saturation non-linear models Linear least squares regression

Today Linear regression versus sample mean. Coefficient of determination Polynomial least squares fit Multiple linear regression General linear squares More on non-linear models Interpolation (Chapter 15)

Polynomial interpolation Newton interpolating polynomials Lagrange interpolating polynomials

Next Time Splines

Quantification of Errors

For a straight line the sum of the squares of the estimate residuals is:

The standard error of the estimate:

sy / x Sr

n 2

Sr ei2

i1

n

yi a0 a1xi 2i1

n

Linear regression versus the sample mean

What is the difference between linear regression and the case when we simply compute the sample mean and draw a line corresponding to the sample mean?

The spread the histogram of the differences between the values predicted by linear regression and the actual sample values.

Regression data showing (a) the spread of data around the mean of the dependent data and (b) the spread of the data around the best fit line:

The reduction in spread represents the improvement due to linear regression.

Coefficient of Determination

The coefficient of determination r2

r2 represents the percentage of the original uncertainty explained by the model.

For a perfect fit, Sr=0 and r2=1. If r2=0, there is no improvement over simply picking the mean. If r2<0, the model is worse than simply picking the mean!

r2 St Sr

St

n

iit yyS

1

2

ExampleV

(m/s)F

(N)

i xi yi a0+a1xi (yi- ȳ)2 (yi-a0-a1xi)2

1 10 25 -39.58 380535 4171

2 20 70 155.12 327041 7245

3 30 380 349.82 68579 911

4 40 550 544.52 8441 30

5 50 610 739.23 1016 16699

6 60 1220 933.93 334229 81837

7 70 830 1128.63 35391 89180

8 80 1450 1323.33 653066 16044

360 5135 1808297 216118

Fest 234.285719.47024v

St yi y 2 1808297

Sr yi a0 a1xi 2 216118

sy 1808297

8 1508.26

sy / x 216118

8 2189.79

r2 1808297 216118

18082970.8805

88.05% of the original uncertaintyhas been explained by the linear model

Polynomial least-fit squares

MATLAB has a built-in function polyfit that fits a least-squares n-th order polynomial to data: p = polyfit(x, y, n)

x: independent data y: dependent data n: order of polynomial to fit p: coefficients of polynomial

f(x)=p1xn+p2xn-1+…+pnx+pn+1

MATLAB’s polyval command can be used to compute a value using the coefficients. y = polyval(p, x)

Fitting an mth order polynomial to n data points

Minimize:

The standard error is:

because the mth order polynomial has (m+1) coefficients. The coefficient of determination r2 is:

n

iit yyS

1

2

Sr ei2

i1

n

yi a0 a1xi a2xi2 am xi

m 2i1

n

sy / x Sr

n m1

r2 St Sr

St

Multiple Linear Regression

Now y is a linear function of two or more independent variables.

The best fit minimize the sum of the squares of the estimate residuals:

For example when:

instead of a line we have a plane

Sr ei2

i1

n

yi a0 a1x1,i a2x2,i amxm,i 2i1

n

y a0 a1x1 a2x2 amxm

22110 xaxaay

General Linear Least Squares Linear, polynomial, and multiple linear regression all belong to the general linear

least-squares model:

where z0, z1, …, zm are a set of m+1 basis functions and e is the error of the fit. The basis functions can be any function data but cannot contain any of the

coefficients a0, a1, etc. The equation can be re-written for each data point as a matrix equation:

where {y} is a vector of n dependent data, {a} is a vector of (m+1) coefficients of the equation, {e} contains the error at each point, and [Z] is:

with zji representing the value of the jth basis function calculated at the ith point.

Z z01 z11 zm1

z02 z12 zm2

z0n z1n zmn

y a0z0 a1z1 a2z2 amzm e

y Z a e

Solving General Linear Least Squares Coefficients

Generally, [Z] is an n x (m+1) matrix. Simple inversion cannot be used to solve for the (m+1) {a}. Instead the sum of the squares of the estimate residuals is minimized:

The outcome of this minimization yields:

Sr ei2

i1

n

yi a jz ji

j0

m

2

i1

n

Z T Z a Z T y

Example

Given the colum vectors x and y, find the coefficients for best fit line y=a0+a1x+a2x2

Z = [ones(size(x) x x.^2]a = (Z’*Z)\(Z’*y)

MATLAB’s left-divide will automatically include the [Z]T terms if the matrix is not square, so

a = Z\ywould work as well

To calculate measures of fit:St = sum((y-mean(y)).^2)Sr = sum((y-Z*a).^2)r2 = 1-Sr/Stsyx = sqrt(Sr/(length(x)-length(a)))

Nonlinear Models

How to deal with nonlinear models (when we cannot fit a straight line) to the sample data Transform the variables and solve for the best fit of the transformed

variables. This works well for exponential, power, saturation models but not all equations can be transformed easily or at all.

Perform nonlinear regression to directly determine the least-squares fit. To perform nonlinear regression:

write a function that returns the sum of the squares of the estimate residuals for a fit and then

use fminsearch function to find the values of the coefficients where a minimum occurs.

The arguments to the function to compute Sr should be the coefficients, the independent variables, and the dependent variables.

Example Given two vectors of n observations ym for the force F and xm for

the velocity v find the coefficients a0 and a1 for the best fit of the equation:

First - write a function called fSSR.m containing the following:

function f = fSSR(a, xm, ym)yp = a(1)*xm.^a(2);f = sum((ym-yp).^2);

Use fminsearch in the command window to obtain the values of a that minimize fSSR:

a = fminsearch(@fSSR, [1, 1], [], v, F)

where [1, 1] is an initial guess for the [a0, a1] vector, [] is a placeholder for the options

F a0va1

Comparison between the transformed of the power equation and the direct method in our example

In the general case the two methods produce different results (the coefficients of the equations are different). The direct method produces the largest r2.

Polynomial Interpolation Problem estimate intermediate values between precise data points.

Related to data fitting but The function uses to interpolate must pass through the data points - this

makes interpolation more restrictive than fitting. Polynomial interpolation an (n-1)th order polynomial is found that

passes through n data points:

How to find the coefficients of the polynomial Use linear algebra to solve a system of n linear equations. Use polyfit and polyval built-in functions. Making sure the order of the fit for n

data points is n-1.

nnnn pxpxpxpxf

12

21

1)(

Matrix formulation of polynomial interpolation: find the coefficients p1, p2 … pn knowing the values of the function f(x1),f(x2)…f(xn)

x1n 1 x1

n 2 x1 1x2

n 1 x2n 2 x2 1

xn 1

n 1 xn 1n 2 xn 1 1

xnn 1 xn

n 2 xn 1

p1

p2

pn 1

pn

f x1 f x2

f xn 1 f xn

Ill conditioned linear problems

A matrix is ill-conditioned if small changes in the coefficients of the solution have drastic effects on the results, which makes iterating the solution to a small residual a tricky operation.

Another type of ill-conditioned matrix is when we have matrix values that vary by several degrees of magnitude. 

Numerical round-off in the system can be challenging for solving a

model having an ill-conditioned matrix.

Problems Vandermonde matrices are very ill-conditioned their solutions are very

sensitive to round-off errors.

Matrices such as that on the left are known as The issue can be minimized by scaling and shifting the data.

x1n 1 x1

n 2 x1 1x2

n 1 x2n 2 x2 1

xn 1

n 1 xn 1n 2 xn 1 1

xnn 1 xn

n 2 xn 1

p1

p2

pn 1

pn

f x1 f x2

f xn 1 f xn

Newton Interpolating Polynomials

The differences between a simple polynomial and Newton’s interpolating polynomial for first and second order interpolations are:

Order Simple Newton1st f1(x)a1 a2x f1(x)b1 b2(x x1)2nd f2 (x)a1 a2xa3x

2 f2 (x)b1 b2(x x1)b3(x x1)(x x2 )

First-order Newton interpolating polynomial

The first-order Newton interpolating polynomial may be obtained from linear interpolation and similar triangles, as shown.

The resulting formula based on known points x1 and x2 and the values of the dependent function at those points is:

f1 x f x1 f x2 f x1 x2 x1

x x1

Second-order Newton interpolating polynomial

The second-order Newton interpolating polynomial introduces some curvature to the line connecting the points, but still goes through the first two points.

The resulting formula based on known points x1, x2, and x3 and the values of the dependent function at those points is:

f2 x f x1 f x2 f x1 x2 x1

x x1

f x3 f x2 x3 x2

f x2 f x1

x2 x1x3 x1

x x1 x x2

Newton interpolating polynomial of degree n-1

In general, an (n-1)th Newton interpolating polynomial has all the terms of the (n-2)th polynomial plus one extra.

The general formula is:

where

and the f[…] represent divided differences.

fn 1 x b1 b2 x x1 bn x x1 x x2 x xn 1

b1 f x1 b2 f x2, x1 b3 f x3, x2, x1

bn f xn, xn 1,, x2 , x1

Divided differences

Divided difference are calculated as follows:

Divided differences are calculated using divided difference of a smaller number of terms:

f xi , x j f xi f x j

xi x j

f xi , x j , xk f xi , x j f x j , xk

xi xk

f xn, xn 1,, x2 , x1 f xn, xn 1,, x2 f xn 1, xn 2,, x1 xn x1

Lagrange interpolating polynomials

Another method that uses shifted value to express an interpolating polynomial is the Lagrange interpolating polynomial.

The differences between a simply polynomial and Lagrange interpolating polynomials for first and second order polynomials is:

where the Li are weighting coefficients that are functions of x.

Order Simple Lagrange1st f1(x)a1 a2x f1(x)L1 f x1 L2 f x2 2nd f2 (x)a1 a2xa3x

2 f2 (x)L1 f x1 L2 f x2 L3 f x3

First-order Lagrange interpolating polynomial The first-order Lagrange

interpolating polynomial may be obtained from a weighted combination of two linear interpolations, as shown.

The resulting formula based on known points x1 and x2 and the values of the dependent function at those points is:

f1(x)L1 f x1 L2 f x2

L1 x x2

x1 x2

,L2 x x1x2 x1

f1(x)x x2

x1 x2

f x1 x x1x2 x1

f x2

Lagrange interpolating polynomial for n points

In general, the Lagrange polynomial interpolation for n points is:

where Li is given by:

fn 1 xi Li x f xi i1

n

Li x x x j

xi x jj1ji

n

Inverse Interpolation

Interpolation general means finding some value f(x) for some x that is between given independent data points.

Sometimes, it will be useful to find the x for which f(x) is a certain value - this is inverse interpolation.

Rather than finding an interpolation of x as a function of f(x), it may be useful to find an equation for f(x) as a function of x using interpolation and then solve the corresponding roots problem: f(x)-fdesired=0 for x.

Extrapolation

Extrapolation is the process of estimating a value of f(x) that lies outside the range of the known base points x1, x2, …, xn.

Extrapolation represents a step into the unknown, and extreme care should be exercised when extrapolating!

Extrapolation Hazards

The following shows the results of extrapolating a seventh-order population data set:

Oscillations Higher-order polynomials can not only lead to round-off errors due to ill-

conditioning, but can also introduce oscillations to an interpolation or fit where they should not be.

In the figures below, the dashed line represents an function, the circles represent samples of the function, and the solid line represents the results of a polynomial interpolation: