Least Squares Problems

From Wikipedia, the free encyclopedia

The method of least squares is a standard approach to the approximate solution of overdetermined systems, i.e., more equations than unknowns.

The most important application is in data fitting

The least-squares method was first described by Carl Friedrich Gauss around 1794

Legendre was the first to publish the method, however.

The Problem:

nmnmAbAx , is ,

residual theis

,min

:such that Find

22

m

n

Axbr

yb-AyAxb

x

The Problem

Range of A

xAx

b

If we have 21 data points we can find a unique polynomial interpolant to these points by solving:

Data-Fitting

20,,0 ),()( ixfxP ii

20

0

20,,0 ),(j

ijii ixfxc

)(

)(

1

1

20

0

20

0

202020

2000

xf

xf

c

c

xx

xx

Without changing the data points we can do better by reducing the degree of the polynomial

In the previous example: Polynomial of degree 8:

Polynomial Least Squares Fitting

)(

)(

1

1

20

0

8

0

82020

800

xf

xf

c

c

xx

xx

Orthogonal Projection and the Normal Equations

Theorem:

ly equivalentor ly equivalent

)range(

min satisfies A vector

given. be and )(Let

22

AxPbbAAxA

Ar

AwbAxbx

bnmA

TT

w

n

mnm

n

Pseudoinverse

mnTT AAAA 1)(

)( 1AAT exists If A has full rank then

Is called the Pseudoinverse, and

bAx Is the least squares solution

Four Algorithms

1. Find the Pseudoinverse

2. Solve the Normal Equation (A full rank):

TT AAAA 1)(

Then calculate bAx

Requires A to have full rank

AAT Is positive definite and we use the Cholesky factorization

Four Algorithms

3- QR Factorization:^^RQA Reduced QR

^^TQQP Orthogonal projector onto range(A)

)(Range ^^

AbQQPby T

hence and such that then yAxx

bQxR

Q

bQQxR

T

T

T

^^

^

^^^^

by multipl-left

Q

Four Algorithms

4- SVD TVUA^^ Reduced SVD

^^TUUP Orthogonal projector onto range(A)

)(Range ^^

AbUUPby T

hence and such that then yAxx

bUxV

U

bUUxVU

TT

T

TT

^^

^

^^^^

by multipl-left