Least Square Approximation and Normal Equation

4th, December 2009Presented by Kwak, Nam-ju

Introduction• Given a data set, we sometimes hope to

find a linear function that represents the data the best.

• We should do our best to minimize the sum of squared error.

Introduction

Normal Equation• Notations

– (vi, wi): an element of the data set– c0+c1v=w: the linear function, supposed to rep-

resent the data set

– Solve the matrix equation to find out the val-ues for c0 and c1.

Normal Equation

• It is expressed again using matrix nota-tions as follows:Ax=d, where each matrix A, x, and d refers to the one of the original expres-sion, in the same order.

Normal Equation• ATAx = ATd is called the normal equation

associated with the least squares prob-lem.

• If ATA has an inverse, then the solution of the normal equation is also a solution of the least squares problem.

Proof• A least squares solution is x such that

rTr=(d-Ax)T(d-Ax) is no larger than(d-Ay)T(d-Ay) for all y’s.

• In other words, we should guarantee that, for all y’s:(d-Ax)T(d-Ax)≤(d-Ay)T(d-Ay).

(d-Ay)T(d-Ay)=((d-Ax)-A(y-x))T((d-Ax)-A(y-x))=((d-Ax)T-(A(y -x))T)((d-Ax)-A(y-x))=(d-Ax)T(d-Ax)-(A(y-x))T(d-Ax) -(d-Ax)TA(y-x)+(A(y-x))T(A(y-x))≥(d-Ax)T(d-Ax)-2((y-x)TAT(d-Ax).

Proofy=x+(y-x)

If this term is 0, the inequality(d-Ax)T(d-Ax)≤(d-Ay)T(d-Ay)Always holds.

Proof• Let’s make ((y-x)TAT(d-Ax)=0.• In general y-x≠0. Therefore AT(d-Ax)=0.

AT(d-Ax)=ATd-ATAx=0ATAx = ATd

• Now, we have the desired result here.• If ATA is invertible, x can be solved.

Example

Further Considerations• With the form of

c0+c1f(v)=walso can be treated using a matrix, which is somewhat transformed from the original one, as follows:

Further Considerations• We can also estimate the curve represent-

ing the data set.– c0+c1v+c2v2=w

References• http://www4.ncsu.edu/eos/users/w/white/

www/white/mamac/Lecture%2011.pdf• http://ceee.rice.edu/Books/LA/leastsq/inde

x.html

Questions and Answers• Any Question?