Chapter 5Orthogonality

Outline

Scalar Product in Rn

Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt Orthogonalization Process

Scalar product in Rn

8

2

3

4

1

2

3

:Example

...2211

yx

yx

yxyxyxyx

T

nnT

Def: Let and be vectors in either R2 or R3.

The distance between and is defined to

be the number yx

x

y

x

y

Theorem 5.1.1

If and are two nonzero vectors in either R2 or

R3 and is the angle between them , then

cosTx xy y

x

y

Proof: By the law of cosines,

cos2222

yxyxxy

2 2 21cos

2x y x y y x

3 3 3

22 2

1 1 1

1

2 i i i ii i i

x y y x

x y3

1i i

i

x y

Corollary 5.1.2(Cauchy-Schwarz Inequality)

If and are vectors in either R2 or R3, then

With equality holding if and only if one of the

vectors is or one vector is a multiple of the

other.

Tx y x y

x

y

0

&x y

cosx y

x y

1 1

x y

x y

Note: If is the angle between , then

Thus

Def: The vectors and in R2(or R3)are said to

be orthogonal if .

6

4-

2

3 :B

, 0 :A

Examples2Rxx

x y

0Tx y

Scalar and Vector Projections

Scalar projection of onto :

Vector projection of onto :

1

T

T

T

x y

x y

y

x y

x yp u y y

y y y

x

u

y

z x p ������������� �

p u

cosx

Example: Find the point

on the line that

is closest to the point

(1,4)

Sol: Note that the vector is on the line

Thus the desired point is

xy3

1

3

1w

xy3

1

2.1

0.7

v ww

w w

v

Q

xy3

1

4.1

Example: Find the equation of the plane

passing through and

normal to

Sol:

4,3,2

3,1,2

0

3

1

2

4

3

2

z

y

x

0341322 zyx

Example: Find the distance form

to the plane

Sol: a normal vector to

the plane is

The distance

0,0,2P

022 zyx

2

2

1

3

2 P n

n

��������������

n

P��������������

Application 1: Information Retrieval Revisited Table 1

Frequency of Key words

Modules

Key Words M1 M2 M3 M4 M5 M6 M7 M8

determines 0 6 3 0 1 0 1 1

eignvalues 0 0 0 0 0 5 3 2

linear 5 4 4 5 4 0 3 3

matrices 6 5 3 3 4 4 3 2

numerical 0 0 0 0 3 0 4 3

orthogonality 0 0 0 0 4 6 0 2

spaces 0 0 5 2 3 3 0 1

systems 5 3 3 2 2 2 1 1

transformations 0 0 0 5 3 3 1 0

vector 0 4 4 3 2 2 0 3

Application I: Information Retrieval Revisited

A is the matrix corresponding to Table I, then the columns of the database matrix Q are determined by setting

To do a search for the key words orthogonality, spaces, vector, we form a unit search vector whose entries are all zero except for the three rows(be put in each of the rows) corresponding to the search rows.

x

1 1, ... ,8j j

j

q a ja

1

3

0.000 0.594 0.327 0.000 0.100 0.000 0.147 0.154

0.000 0.000 0.000 0.000 0.000 0.500 0.442 0.309

0.539 0.396 0.436 0.574 0.400 0.000 0.442 0.463

0.647 0.495 0.327 0.344 0.400 0.400 0.442 0.309

0.000 0.000 0.000 0.000 0.300 0.000 0.590 0.4Q

63

0.000 0.000 0.000 0.000 0.400 0.600 0.000 0.309

0.000 0.000 0.546 0.229 0.300 0.300 0.000 0.154

0.539 0.297 0.327 0.229 0.400 0.200 0.147 0.154

0.000 0.000 0.000 0.574 0.100 0.300 0.147 0.000

0.000 0.396 0.436 0.344 0.400 0.200 0.000 0.4

0.000

0.000

0.000

0.000

0.000

0.577

0.577

0.000

0.000

63 0.577

x

i

cos

where is the angle between the unit vectors and .

T Ti i i

i

y Q x y q x

x q

For our example,

0.000, 0.229, 0.567, 0.310, 0.635, 0.577, 0.000, 0.535T

y

Application I: Information Retrieval Revisited

Since is the entry of that is closest to 1,this indicates that the direction of the search vector is closest to the direction of and hence that Module 5 is the one that best matches our search criteria.

y

x

5q

5 0.635y

Application 2: Correlation And Covariance Matrices

Table 2

Math Scores Fall 1996

Scores

Student Assignment Exams Final

S1 198 200 196

S2 160 165 165

S3 158 158 133

S4 150 165 91

S5 175 182 151

S6 134 135 101

S7 152 136 80

Average 161 163 131

37 37 65

1 2 34

3 5 2

11 2 40

14 19 20

27 28 30

9 27 51

X

The column vectors of X represent the deviations from the mean for each of the three sets of scores.

The three sets of translated data specified by the column vectors of X all have mean 0 and all sum to 0.

A cosine value near 1 indicates that the two sets of scores are highly correlated.

Scale to make them unit vectors

Application 2: Correlation And Covariance Matrices

1 2 and x x

1 1 2 21 2

1 1 and u x u x

x x

If we set , thenTC U U

0.74 0.65 0.62

0.02 0.03 0.33

0.06 0.09 0.02

0.22 0.03 0.38

0.28 0.33 0.19

0.54 0.49 0.29

0.18 0.47 0.49

U

1 0.92 0.83

0.92 1 0.83

0.83 0.83 1

C

The matrix C is referred to as a correlation matrix.

The three sets of scores in our example are all positively correlated since the correlation coefficients are all positive.

A negative coefficient would indicate that two data sets were negatively correlated.

A coefficient of 0 would indicate that they were uncorrelated.

Application 2: Correlation And Covariance Matrices

Def: Two subspaces X and Y of are said to be orthogonal if = 0 for every and If X and Y are orthogonal, we write

x X y Y

X Y

5-2 Orthogonal Subspaces

31 2 3

Example:

Let { , } , { }

then

X span e e Y span e

X Y

Tx y

n

Def: Let Y be a subspace of . The set of all vectors in that are orthogonal to every vector in Y will be denoted . Thus = { for every } The set is called the orthogonal complement of Y

Y

| 0n Tx R x y Y y Y

Y

31

2 3

Example:

Let { }

then { , }

X span e

X span e e

nn

Remarks:

1. If X and Y are orthogonal subspaces of , then .

2. If Y is a subspace of , then is also a subspace of .

{0}X Y

Y

2Proof(1). If and , then 0

and hence 0.

Tx X Y X Y x x x

x

1 2

1 2 1 2

Proof(2). If and is a scalar, then for any ,

( ) ( ) 0 0

If and , then

( ) 0

T T

T TT

x Y y Y

x y x y x Y

x x Y

x x y x y x y

1 2

0 0 for each ,

.

Therefore, is a subspace of .n

y Y

x x Y

Y

nn n

Four Fundamental Subspaces

Let nmA

mnAor :x AxmnA

0n nN A x Ax

It will be shown later that N A R A , N A R A

and n N A R A

ARANm

for some m n mR A b b Ax x

0T m T mN A x A x

for some T n T m nR A b b A x x

Theorem 5.2.1(Fundamental Subspace Theorem)

pf: Let and

Also, if

Similarly,

x N A y R A

,: 0 1, ---------(1)A i x i m

1

and :, for some 's --------(2)m

i ii

y A i

(1)

1

:, 0m

ii

x y x A i

ARAN

N A R A

( ,:) 0, 1, 0 ( )A i z i m Az z N A

R A N A

hence, N A R A

:, 0, 1,z R A z A i i m

ARARAN

( ) (Let , then and ) ) ( .( )Tm n TN A R A N A R AA

Example: Let

Clearly,

00

21

02

01AA

1

0spanAN

1

2spanAN

2

1spanAR

0

1spanAR

ARAN

ARAN

Theorem 5.2.2

If S is a subspace of , then

Furthermore, if { } is a basis for S and

{ }is a basis for , then { , }

is a basis for .1,..., rx x

1,...,r nx x

S1,..., rx x

1,...,r nx x

n

n

dim dimS S n

Proof: If The result follows

Suppose . Let

and

{0} nS S

{0}S

1( ) n rrX x x

( ) ( )rank X r rank X ( )R X S

5.2.1

( ) ( )Theorem

S R X N X

3.6.4

dim( ) dim ( )Thm

S N X n r

To show that is a basis for ,

It remains to show their independency.

Let . Then

Similarly,

n

1

0n

i ii

c x

x S

1 1

0n r

i i i ii i

x c x x c x

1

0 0, 1r

i i ii

c x c i r

y S

1 1

0n n

i i i ii i r

y c x y c x

1

0 0, 1,n

i i ii r

c x c i r n

1{ }nx x

Def: If U and V are subspaces of a vector space W

and each can be written uniquely as a

sum , where and ,then we

say that W is a direct sum of U and V, and we

write

w W

u U

v V

W U V

u v

Theorem5.2.3: If is a subspace of ,

then

pf: By Theorem5.2.2,

To show uniqueness,

Suppose

where

nS SSn

SSn

, , & .nx x u v u S v S

1 1 2 2x u v u v

1 2 1 2, & ,u u S v v S

1 2 2 1u u v v S S {0}S S

1 2 2 1&u u v v

Theorem5.2.4: If is a subspace of ,

then

pf: Let

If

S nSS )(

rS )dim(

rSTheorem

)dim(2.2.5

, then 0x S x y y S

x S

SS

SS

Remark: Let . i.e. , Since

and

are bijections .

nmA mnA : ARANn

ArankArank )( ArankAnullityn ArankAnullity

ARARA :

ARARA :

Let nmA nA :

mA :

m

n

A

A

A

A

AN

AN

AN

AN

bijection

bijection

0

0

Cor5.2.5:

Let and . Then

either

(i)

or (ii)

pf:

nmA mb

nx Ax b

my

0 and 0A y y b

(i) ( ) nb R A x Ax b

(ii) ( ) ( ) ( ) 0b R A N A y N A y b

0 & 0my A y y b

( ) ( )m TR A N A )(AR

b

)( AN

for 3m

Example: Let . Find

The basic idea is that the row space and the sol. of

are invariant under row operations.

Sol: (i)

(Why?)

(ii)

(Why?)

(iii) Similarly,

and

(iv) Clearly,

431

110

211

A )(),(),(),( ARANARAN

Ax b

000

110

101

~ r

row

AA

1

1

0

1

0

1

)( spanAR

1 3 2 30 0 & 0rA x x x x x

1

1

1

)( spanAN

000

210

101

~row

A

2

1

0

1

0

1

)( spanAR

1

2

1

)( spanAN

)()(& ARANARAN

Example: Let

(i)

and

(ii) The mapping

and

(iv) What is the matrix representation for ?

23:0

0

3

0

0

2

A

0

1

0

0

0

1

1

0

03 spanspanARAN

2)( AR

( ): ( ) is a bijection

R AA R A R A

0

3

2

02

1

2

1

x

x

x

x

)(:1

)(

ARARAAR

03

12

1

2

1

2

1 y

y

y

y

)( ARA

5-4 Inner Product Spaces

A tool to measure the

orthogonality of two vectors in

general vector space

Def: An inner product on a

vector space is a function

Satisfying the following conditions:

(i) with equality iff

(ii)

(iii)

V

, : ( )V V F orC

, 0x x 0x

, ,x y y x

, , ,x y z x z y z

Example: (i) Let

Then is an inner product of

(ii) Let , Then is an

inner product of (iii) Let and then

is an inner product of

(iv) Let , is a positive function and are distinct real numbers. Then

is an inner product of

, & 0 1, .nix y w i n

1

,n

i i ii

x y w x y

n

ij

n

jij

m

i

baBA

11

,nmBA ,nm ].,[)(,, 0 baCxwgf 0)( xw

b

a

dxxgxfxwgf )()()(,

].,[0 baC

, np g P )(xw

nxx 1

)()()(,1

ii

n

ii xgxPxwgp

nP

Def: Let be an inner product of a

vector space and .

we say

The length or norm of is given

by

,

V ,u v V

, 0u v u v

v

,v v v

Theorem5.4.1: (The Pythagorean Law)

pf:

2 2 2u v u v u v

2 2u v

2,u v u v u v

, , , ,u u u v v u v v

u

v u v

Example 1: Consider with inner product

(i)

(ii)

(iii)

(iv) (Pythagorean Law)

or

]1,1[0 C

1

1)()(, dxxgxfgf

xxdxx 101,11

1

212111,11

1 dx

32

3

2,

1

1 xxdxxxx

3

8

3

2211

222 xx

3

8)1(1,11

21

1

2 dxxxxx

Example 2: Consider with inner product

It can be shown that

(i)

(ii)

(iii)

Thus is an orthonormal

set.

0[ , ]C

dxxgxfgf )()(

1,

0sin,cos mxnx

cos ,cos

sin ,sin

mn

mn

mx nx

mx nx

1,cos ,sin

2nx nx n N

1 1, 1

2 2

Remark

Remark: The inner product in example 2 plays a key

role in Fourier analysis application involving trigo-

nometric approximation of functions.

2 2cos sin cos sin 2x x x x

Example 3: Let

and let

Then is not orthogonal to

, ,m nA B

4

0

1

3

3

1

,

3

2

1

3

1

1

BA

ABA 6, B

6,5 FF

BA

1 1 1

, ( ) ( )

,

m n mT T

ij ij iii j i

F

A B a b trace AB AB

A A A

Def: Let be two vectors in an

inner product space . Then

the scalar projection of onto is

defined as

The vector projection of onto is

& 0u v

V

u

v

,1,

u vu v

vv

,1

,

u vp v v

v v v

u

v

Lemma: Let be the vector projection

of onto . Then

for some

pf:

0 &v p

u v

( )

( )

i u p p

i u p u kv

2 2

( ) , , ,

, , 0

, ,

( ) .

i p u p p u p p

u v u v

v v v v

p u p

ii trivial

u

v p

u p

k

Theorem5.4.2: (Cauchy-Schwarz Inequality)

Let be two vectors in an

inner product space . Then

Moreover, equality holds are linear dependent.

pf: If

If

Equality holds

i.e., equality holds iff are linear dependent.

&u v

V,u v u v

&u v

0, then , 0

0, then

v u v u v

v

2

2 2 2,

,

Pythagorean Theoremu vp u u p

v v

2 2 2 2 2 22,u v u v v u p u v

,0, or

,

u vv u p v

v v

&u v

u

v p

u p

Note: From Cauchy-Schwarz Inequality for .

This, we can define as the angle between the two nonzero vectors

,1 1 if 0 and 0

,! 0, cos .

u vu v

u v

u v

u v

& .u v

F

Def: Let be a vector space a function

is said to be a norm if it satisfies

V

: {0}V v v

( ) 0 with equality 0

( ) , scalar .

( ) (triangle inequality)

i v v

ii v v

iii v w v w

Remark: Such a vector space is called a normed linear space.

Theorem5.4.3: If is an inner product

space, then

defines a norm on

pf: trivial

Def: The distance between is defined

as.u v

V

.V

&u v

, v v v v V

Example: Let , thennx

11

1

1

1

2

21

( ) is a norm.

( ) is a norm.

( ) is a norm for any 1.

in particular =2,

, is the euclidean norm.

max

n

ii

ii n

n PP

iPi

n

ii

i x x

ii x x

iii x x p

p

x x x x

Example: Let

Thus,

However,

(Why?)

1 2

1 4&

2 2x x

1 2, 0x x

2 2 2

2 1 22 2 25 20 25x x x x

2 2

1 2

2

1 2

4 16

20 16

x x

x x

Remark: In the case of a norm that is not derived from an inner product, the Pythagorean Law will not hold.

Example: Let ,

then

4

5

3

x

1

2

12

5 2

5

x

x

x

Example: Let

Then

2 1B x x

1B2B B

1 1 1

5-3 Least Squares Problems

Least squares problems

A typical example: Given

Find the best line to fit the data . or

or find such that is minimum Geometrical meaning :

, 1,i

i

xi n

y

0 1y c c x 1 1

2 20

1

1

1

1 n n

x y

x ycsolve

c

x y

Ac y

0 1,c cAc y

0 1y c c x

( , )n nx y

1 1( , )x y

Least squares problems:

Given

then the equation

may not have solutions

The objective of least square problem is

trying to find such that

is minimum value

i.e., find satisfying

& ,m n mA b

Ax b

. ., ( ) ( )i e b Col A R A

x

b Ax

x

minnx

b A x b Ax

( )R A

b

Ax

Preview of the results:

It will be shown that

If columns of are linear independent .

( )! ( ), min

y R Ap R A b p y b

( ) ( )

0

0

b p R A N A

A b p

A b Ax

A Ax A b

1x A A A b

A

b

( )R Ap

Theorem5.3.1: Let be a subspace of , then

(i) for all

(ii)

pf:

(i)

where

If

(ii) follows directly from (i) by noting that

S m, ! , mb p S b y b p

\{ }y S p

miny S

p b y b b p S

m S S

b p z

&p S z S

\{ }y S p

2

2 2 2

0

Pythogorean Theorem

Sz S

b y b p p y b p p y

b p z S

p

b

S

unique expression

Question: How to find which solves

Ans.:

From previous Theorem , we know that

x

min ?nx

A x b b Ax

( ) ( )

( ) 0

0

b p R A N A

A b p

A b A Ax

( )R A

p Ax

b

Definition ： is called normal equatio .ntheA Ax A b

Remark: In general, it is possible to have more than one solution to the normal equation. If is a solution, then the general solution is of the form

ˆ where ( )x h h N A

x̂

Theorem5.3.2: Let and

Then the normal equation

has an unique solution .

and is the unique least squares solution to

pf: To show that is nonsingular

nmA .)( nArank A Ax A b

1x A A A b

x

.Ax b

AA

Let 0 ( ) ( ) {0}

0 0 ( ( ) )

T TA Ax Ax N A R A

Ax x rank A n

1 is the unique solution.x A A A b

Note: The projection vector

is the element of that

is closet to in the least squares

sense .

Thus, The matrix is called the

projection matrix (that project any vector of

to )

1p Ax A A A A b

)(AR

b

1P A A A A

)(ARm

b

)(AR

p

Suppose a spring obeys the Hook’s law

and a series of data are taken (with measurement

error) as

How to determine ?

sol: Note that is inconsistent

The normal equation is

so,

KxF

11

8

7

5

4

3

x

F

8

5

3

11

7

4

811

57

34

Kor

K

K

K

8

5

3

1174

11

7

4

1174 K

K

186 135 0.726K K

Application 2: Spring Constants

Example 2: Given the data

Find the best least squares fit by a linear function.

sol: Let the desired linear function be The problem becomes to find the least squares solution of

is the unique solution.

Thus, the best linear least square fit is

5

6

4

3

1

0

y

x

xccy 10

0

1

1 0 1

1 3 4

1 6 5c

yA

c

c

10

1

4

32

3

cA A A y

c

xy3

2

3

4

∵ rank(A)=2

Example3: Find the best quadratic least squares fit to the data

sol:

Let the desired quadratic function be

The problem becomes to find the least square

solution of

is the unique solution.

Thus, the best quadratic least square fit is

0

1

1

2

2.75

0.25

0.25

c

c A A A y

c

4

3

4

2

2

1

3

0

y

x

2210 xcxccy

2

1

0

9

4

1

0

3

2

1

0

1

1

1

1

4

4

2

3

c

c

c

225.025.075.2 xxy

∵ rank(A)=3

5-5 Orthonormal Sets

Orthonormal Set

Simplify the least squares solution

(avoid computing inverse) Numerical computational stability

Def: is said to be an orthogonal set in

an inner product space if

Moreover, if , then is said

to be orthonormal.

1 nv v

V

, 0i jv v for all i j

,i j ijv v 1 nv v

Example 2:

is an

orthogonal set but not orthonormal.

However ,

is orthonormal.

1 2 3

1 2 4

1 , 1 , 5

1 3 1

v v v

1 1 2 2 3 3

1 1 1, ,

3 14 42u v u v u v

Theorem5.5.1: Let be an orthogonal

set of nonzero vectors in an inner product

space . Then they are linear independent.

pf: Suppose that

1 nv v

V

1

0n

i ii

c v

1 1

1

0 , , ,

0, 1,

is linearly independent.

n n

j i i i j i j j ji i

j

n

v c v c v v c v v

c j n

v v

Example:

is an

orthonormal set of with inner

product .

Note: Now you know the meaning what one

says that .

1, cos , sin

2nx nx n N

,0 C

dxxgxfgf )()(

1,

xx sincos

Theorem5.5.2: Let be an orthonormal

basis for an inner product space .

If , then .

pf:

1 nu u

V

1

n

i ii

v c u

,i ic u v

1 1

1

, , ,

n n

i i j j j i jj j

n

j ij ij

u v u c u c u u

c c

Cor: Let be an orthonormal basis for an inner product space .

If and , then .

pf:

V

1 nu u

1

n

i ii

u a u

1

n

i ii

v b u

1

,n

i ii

u v a b

1 1

5.5.2

1

, , ,

n n

i i i ii i

nTheorem

i ii

u v a u v a u v

a b

Cor: (Parseval’s Formula)

If is an orthonormal basis for an

inner product space and , then

pf: By Corollary 5.5.3,

1 nu u

V1

n

i ii

v c u

2 2

1

n

ii

v c

2 2

1

,n

ii

v v v c

Example 4:

and form

an orthonormal basis for .

If , then

and

1

1

21

2

u

2

1

21

2

u

21 2

2

xx

x

1 2 1 21 2

5.5.21 2 1 2

1 2

, , ,2 2

2 2

Theorem

x x x xx u x u

x x x xx u u

2 22 2 21 2 1 2

1 22 2

x x x xx x x

Example 5: Determine without computing

antiderivatives .

sol:

xdx

4sin

24 2 2 2

2

0

24 2

2 2

sin sin ,sin sin

1 cos 2 1 1 1sin cos 2

2 22 2

1and ,cos 2 is an orthonormal set of ,

2

sin sin

1 1 3

2 42

xdx x x x

xx x

x C

xdx x

Def: is said to be an orthogonal matrix if the column vectors of form an

orthonormal set in .

Example 6:

The rotational matrix

and the elementary reflection matrix

are orthogonal matrix .

n nQ

nQ

cossin

sincos

cossin

sincos

Properties of orthogonal matrices:

If is orthogonal, thenn nQ

1

2 2

( ) The column vectors of form an orthonormal

basis for .

( )

( )

( ) , , preserve inner product

( ) preserve norm

( ) preserve angle

n

i Q

ii Q Q I QQ

iii Q Q

iv Qx Qy x y

v Qx x

vi

Theorem 5.5.6: If the columns of form an orthonormal set in , then and the least squares solution to is

This avoid computing matrix inverse .

m

m nA IAA

Ax b

1x A A A b A b

Theorem 5.5.7 & 5.5.8:

Let be a subspace of an inner product

space and let . Let be

an orthonormal basis for .

If , where ,

then

1 2, , , nx x x

S

x V

(i)

(ii) - - in .

p x S

y x p x y p S

VS

1

n

i ii

p c x

, for each i ic x x i

The vector is said to be t projectionhe of onto .p x S

Cor5.5.9:

Let be a subspace of and

If be an orthonormal basis for

and then

the projection of onto is .

pf:

1 ku u

S

S

1 ,kU u u

m

Sb

p

p UU b

1 1 2 2

1 1

2 2

From Thm.5.5.8, ... ,

where

Therefore, .

k k

T

TT

Tk k

p c u c u c u Uc

c u b

c u bc U b

c u b

p UU b

.mb

Note: Let columns of be an

orthonormal set

1 kU u u

1

1

1

k

k

k

i ii

u

p UU b u u b

u

u b u

(i) The projection of onto ( ) is the

sum of the projection of onto each .

(ii) The matrix is called the projection

matrix onto .

i

T

b R U

b u

UU

S

Example 7: Let

Find the vector in that is closet to

Sol: 5,3,4 .w

, ,0 , .S x y x y p

S

1 2

1 0

Clearly, , is a basis for . Let 0 1 ,

0 0

Thus,

1 0 0 5 5

0 1 0 3 3

0 0 0 4 0

e e S U

p UU w

1 1

2 21 1

: , what is ?2 2

0 0

THW U UU

Approximation of functions

Example 8: Find the best least squares approximation to

on by a linear function .

Sol:

xe 1,0

2

2( )

12 2

0

( . ., Find g( ) 0,1 ( ) min ( ) ,

where , .)

x x

p x Pi e x P e g x e p x

f f f f dx

2

1

0

1 2

(i) Clearly, 1, 0,1 ,but 1, is not orthonormal.

(ii) seek a function of the form , ( ) 1

1 1 1, ( ) 0

2 21 1

2 12

1 1, 12( ) for

2

span x P x

x a x a

x a x a dx a a

x

u u x

2m an orthonormal set of 0,1 .P

Sol:

1

1 1 0

1

2 2 20

1 1 2 2

(iii) c , 1

c , 3 3

Thus, the projection

( )

1 ( 1) 1 3(3 )( 12( ))

2

x x

x x

u e e e

u e u e dx e

p x c u c u

e e x

(4 10) 6(3 )

is the best linear least square approximation to on 0,1 .x

e e x

e

Approximation of trigonometric polynomialsFACT: forms an orthonormal set in with respect to the inner product

Problem: Given a continuous 2π-periodic function , find a trigonometric polynomial of degree n

which is a best least squares approximation to .

1,cos ,sin

2nx nx n N

,0 C

dxxgxfgf )()(

1,

)(xf

0

1

( ) cos sin2

n

n k kk

at x a kx b kx

)(xf

Sol: It suffices to find the projection of onto

the subspace

The best approximation of

has coefficients

)(xf

1,cos ,sin 1, ,

2span kx kx k n

( )nt x

0

1 1, ( )

2 2

1,cos ( )cos

1,sin ( )sin

k

k

a f f x dx

a f kx f x kxdx

b f kx f x kxdx

Example: Consider with inner product of

(i) Check that is orthonormal

(ii) Let

,0 C

dxxgxfgf )()(

2

1,

0, 1, ,ikxe k n

1( )

21

( )2

nikx

n kk n

ikxk

k k

t c e

c f x e dx

a ib

Similarly, k kc c

(iii)

(iv)

cos sin

ikx ikxk k

k k

c e c e

a kx b kx

0

1

cos sin2

nikx

n kk n

n

k kk

t c e

aa kx b kx

5-6 Gram-Schmidt Orthogonalization Process

Cram-Schmidt Orthogonalization ProcessQuestion: Given an ordinary basis ,

how to transform them into an orthonormal

basis ?

1 2, ,..., nx x x

1 2, ,..., nu u u

Given ,Clearly

Clearly,

Similarly,

Clearly, We have the next result

1 nx x

1 11

1u x

x 1 1{ } { }span u span x

1 2 1 1 2 2 12 1

1, , ( )p x u u u x p

x p

1 2 1 2 1 2& { , } { , }u u span x x span u u

2 3 1 1 3 2 2

3 3 23 2

, ,

1( )

p x u u x u u

and u x px p

3 1 3 2 1 2 3 1 2 3, & { , , } { , , }u u u u span x x x span u u u

1u

1p

2x

Theorem5.6.1: (The Gram-Schmidt process)

H. (i) Let be a basis for an inner

product space .

(ii)

C. is an orthonormal basis. 1 nu u

1 nx x

V

1 11

1 11

11

1,

1, 1, , 1

,

K K KK K

K

K K j jj

u xx

u x p K nx p

where p x u u

Example: Find an orthonormal basis for with

inner product given by

, where

Sol: Starting with a basis

3P

),()(,3

1i

ii xgxPgP

.1&0,1 321 xxx

2,,1 xx

1 2 1

1 2

11 1 1 11 1

Let , ,..., be the projection vectors defines in Thm. 5.6.1, and

let , ,..., be the orthonormal basis of ( ) derived from the

Gram-Schmidt process.

Define

n

n

kk

p p p

q q q R A

r a a r q

r

1 for 2,...,

and for 1,..., 1 by the Gram-Schmidt process.

k k

Tik i k

a p k n

r q a i k

Theorem5.6.2: (QR Factorization)

If A is an m×n matrix of rank n, then A

can be factored into a product QR, where Q

is an m×n matrix with orthonormal columns

and R is an n×n matrix that is upper triangular

and invertible.

Proof. of QR-Factorization

1 2 1

1 2 1

11 1

Let , ,..., be the projection vectors defined in Thm.5.6.1,

and let , ,..., be the orthonormal basis of ( ) derived from

the Gram-Schmidt process.

Define

n

n

kk k k

p p p

q q q R A

r a

r a p

1

1 11 1

2 12 1 22 2

1 1

and for 1,... -1 for 2,...,

By the Gram-Schmidt process,

Tik i k

n n

r q a i kk n

a r q

a r q r q

a r q

... nn nr q

Proof. of QR-Factorization (cont.)

1 2

11 12 1

22 2

If we set ( , ,..., ) and define to be the upper triangular matrix

0 ,

0 0

then the th column of the product wi

n

n

n

nn

Q q q q R

r r r

r rR

r

j QR

1 1 2 2

1 2

ll be

... for 1,... .

Therefore,

( , ,..., )

j j j jj j j

n

Qr r q r q r q a j n

QR a a a A

Theorem5.6.3:

If A is an m×n matrix of rank n, then the

solution to the least squares problem

is given by , where Q and R are the

matrices obtained from Thm.5.6.2. The solution

may be obtained by using back substitution to solve .

Ax b

1x̂ R Q b

x̂

ˆRx Q b

Proof. of Thm.5.6.3

ˆLet be the solution to the leaset squares problem

ˆ

ˆ

ˆ ( ) ( ) ( )

ˆ ( )

TAT T

T T

T T

I

x

Ax b

A Ax A b

QR QRx QR b QR Factorization

R Q Q R

1

ˆ ( is invertible)

ˆ ˆ or

T T

T T T

T

x R Q b

R Rx R Q b R

Rx Q b x R Q b

Example 3: Solve

By direct calculation,

1

2

3

1 2 1 1

2 0 1 1

2 4 2 1

4 0 0 2bA

x

x

x

R

Q

QRA

200

140

125

1

2

2

4

2

4

1

2

4

2

2

1

5

1

1

1

2

Q b

The solution can be obtained from

5 2 1 1

0 4 1 1

0 0 2 2