Least_Squares.pdf

7/27/2019 Least_Squares.pdf

http://slidepdf.com/reader/full/leastsquarespdf 1/9

A Linear System (or system of linear equations) is a collection of one or m

linear equations involving the same set of variables, x1, x2, …, xn.

A system of linear equations:

In matrix-vector form, we have

Where , , and

This is a system of equations and unknowns.

We would like to know when and how many solutions exist for this system

Linear SystemMonday, September 30, 2013

11:59 PM

Linear Systems Page 1



Let , , and

If , there exists a solution.-

The system is called consistent.-

There exists at least one solution if .-

The existence of the solution?

If , there exists only one solution, no ambiguity.-

If , there exists many solutions.-

There exists only one solution if not only , but also

(i.e., columns of A are linearly independent).

-

The uniqueness of the solution?

Remarks: In general

Existence: If is onto (i.e., ) and one-to-one (i.e., there exists one solution.

Uniqueness: The unique solution is given by .

If , matrix is square, and the number of unknowns equals the numbe

equations

○

Existence: if , many choices of lead to , i.e., there are ma

solutions.

Uniqueness: If is one solution (i.e., ) and , then also a solution.

characterizes freedom of input choices.

This system is also called underdetermined.

If , matrix A is called (strictly) fat, and there are more unknowns than

equations

○

Existence: there is no solution.

This system is also called overdetermined.

If , matrix A is called (strictly) skinny, and there are more equations tha

unknowns

○

Existence and Uniqueness of the SolutionTuesday, October 01, 2013

10:07 PM




We just saw that when , the system dose not have any solu

We are looking for a vector for which is the closest to y.

Define the (approximation) error as Find that minimizes

is called the least squares (or approximation) solution of

More precisely,

Least-SquaresTuesday, October 01, 2013

10:34 PM




Let be skinny (i.e., ) and of full rank (i.e.,

Show that the solution to the minimization problem

Min

is

is a linear function of y-

is called the pseudo-inverse of -

is a left inverse of matrix -

If A is square, -

Remarks:

Least-Squares SolutionTuesday, October 01, 2013

11:10 PM




is the closest point in to . It is called the

projection of onto .

is given by-

The projection function is a linear function of y-

is called the projection matrix-

A matrix is a projection matrix if . Why?-

Remarks:

ProjectionTuesday, October 01, 2013

11:31 PM




Show that -

•

•

, where •

, where and •

OrthogonalityWednesday, October 02, 2013

12:24 AM




Suppose is skinny and full rank.

, and

upper triangular and invertible.

Using QR decomposition method, we have

Show that

•

•

Remarks: we have

Least-Squares and Full QR Factorization:

, , and orthogona

upper triangular and invertible

Using full QR decomposition method, we have

Show that

Remarks: we have

–

Least-Squares and QR FactorizationWednesday, October 02, 2013

12:39 AM




–

gives the projection onto –

–

gives the projection onto –




Least-Squares EstimationWednesday, October 02, 2013

1:40 AM

Documents

Least_Squares.pdf