1

Chapter 8 – Symmetric Matricesand Quadratic Forms

Outline8.1 Symmetric Matrices8.2 Quardratic Forms8.3 Singular Values

2

8.1 Symmetric Matrices• (Spectral theorem) A matrix A is orthogonally diagonalizable

(i.e., there is an orthogonal S such that S-1AS=STAS is diagonal) if and only if A is symmetric (i.e., AT=A).

• Consider a symmetric matrix A. If and are eigenvectors of A with distinct eigenvalues λ1 and λ2, then ; that is, is orthogonal to .

• A symmetric n×n matrix A has n real eigenvalues if they are counted with their algebraic multiplicities.

1v

2v

021 vv

2v

1v

3

Example 1• If A is orthogonally diagonalizable, what is the

relationship between AT and A?• (sol)

– We have S-1AS=D or A=SDS-1=SDST, for an orthogonal S and a diagonal D. Then AT=(SDST)T=SDTST=SDST=A.

– We find that A is symmetric: AT=A.

4

Example 2• For the symmetric matrix find an orthogonal S such

that S-1AS is diagonal.• (sol)

– We will first find an eigenbasis. The eigenvalues of A are 3 and 8, with corresponding eigenvectors respectively.

7224

A

,21

and 1

2

5

Example 2 (II)– Note that the two eigenspaces, E3 and E8, are perpendicular. Therefore,

we can find an orthonormal eigenbasis simply by dividing the given eigenvectors by their lengths:

– If we define the orthogonal matrix

– then S-1AS will be diagonal, namely,

.21

51 ,

12

51

21

vv

,2112

51

||

||

21

vvS

.80031

ASS

6

Example 3• For the symmetric matrix find an orthogonal S such

that S-1AS is diagonal.• (sol)

– The eigenvalues are 0 and 3, with

– Note that the two eigenspaces are indeed perpendicular to one another.– We can construct an orthonormal eigenbasis for A by picking an

orthonormal basis of each eigenspace.

,111111111

A

.111

span and 101

,011

span 30

EE

7

Example 3 (II)– In Figure 3, the vectors form an orthonormal basis of E0, and

is a unit vector in E3. Then, is an orthonormal eigenbasis for A. We can let to diagonalize A orthogonally.

– If we apply the Gram–Schmidt process to the vectors– spanning E0, we find

– The computations are left as an exercise. For E3, we get

– Therefore, the orthogonal matrix

– diagonalizes the matrix A:

21,vv 3v

321 ,, vvv

321 vvvS

101

,011

.211

61 and

011

21

21

vv

111

31

3v

3/16/203/16/12/13/16/12/1

|||

|||

321 vvvS

300000000

1ASS

8

Orthogonal Diagonalization of a Symmetric Matrix A

• Find the eigenvalues of A, and find a basis of each eigenspace.• Using the Gram–Schmidt process, find an orthonormal basis of

each eigenspace.• Form an orthonormal eigenbasis for A by

combining the vectors you found in part (b), and let

S is orthogonal, and S-1AS will be diagonal.

nvvv

,,, 21

.|||

|||

21

nvvvS

9

Example 4• Consider an invertible symmetric 2×2 matrix A. Show that the

linear transformation T(x)=Ax maps the unit circle into an ellipse, and find the lengths of the semimajor and the semiminor axes of this ellipse in terms of the eigenvalues of A. Compare this with Exercise 2.2.50.

• (sol)– The spectral theorem tells us that there is an orthonormal eigenbasis

for T , with associated real eigenvalues λ1 and λ2. Suppose that |λ1| ≥ |λ2|. These eigenvalues will be nonzero, since A is invertible. The unit circle in R2 consists of all vectors of the form

21,vv

.)sin()cos( 21 vtvtv

10

Example 4 (II)– The image of the unit circle consists of the vectors

– an ellipse whose semimajor axis has the length , while the length of the semiminor axis is .

– In the example illustrated in Figure 4, the eigenvalue λ1 is positive, and λ2 is negative.

,)sin()cos()()sin()()cos()(

2211

21

vtvtvTtvTtvT

11v

111 v

222 v

11

8.2 Quardratic Forms• A function q(x1, x2, . . . , xn) from Rn to R is called a quadratic

form if it is a linear combination of functions of the form xixj (where i and j may be equal). A quadratic form can be written as

for a symmetric n × n matrix A.• Consider a quadratic form from Rn to R. Let B be an

orthonormal eigenbasis for A, with associated eigenvalues λ1, . . . , λn. Then

where the ci are the coordinates of with respect to B.

,)( xAxxAxxq T

xAxxq

)(

,)( 2222

211 nncccxq

x

12

Example 1

• Consider the function q(x1, x2) = 8x12-4x1x2+5x2

2 from R2 to R.Determine whether q(0,0)=0 is the global maximum, the global minimum, or neither.Recall that q(0,0) is called the global (or absolute) minimum if q(0,0)≤q(x1, x2) for all real numbers x1, x2; the global maximum is defined analogously.

• (sol)– There are a number of ways to do this problem, some of which

you may have seen in a previous course. Here we present an approach based on matrix techniques. We will first develop some theory, and then do the example.

13

Example 1 (II)– Note that we can write

– More succinctly, we can write

or– The matrix A is symmetric by construction. We find

with associated eigenvalues λ1 = 9 and λ2 = 4.– If we write , we can express the value of the function as

follows:

(Recall that since is an orthonormal basis of R2.)

21

21

2

12221

21

2

1

5228

548xx

xxxx

xxxxxx

q

5228

where,)( AxAxxq

.)( xAxxq T

,21

51 ,

12

51

21

vv

2211 vcvcx

.49)()()( 22

21

222

2112221112211 ccccvcvcvcvcxAxxq

,1 and ,0 ,1 222111 vvvvvv

21,vv

14

Example 1 (III)– The formula shows that for all nonzero ,

because at least one of the terms is positive.– Thus q(0,0)= 0 is the global minimum of the function. The preceding

work shows that the c1–c2 coordinate system defined by an orthonormal eigenbasis for A is “well adjusted” to the function q. The formulais easier to work with than the original formulabecause no term involves c1c2:

22

21 49)( ccxq

0)( xq x

22

21 4 and 9 cc

22

21 49 cc

,548 2221

21 xxxx

22

21

2221

2121 49548),( ccxxxxxxq

15

Example 2• Consider the quadratic form

Find a symmetric matrix A such that for all in R3.• (sol)

– We let

– Therefore,

.642379),,( 32312123

22

21321 xxxxxxxxxxxxq

xAxxq )( x

. if ), oft coefficien( ), oft coefficien( 212 jixxaaxa jijiijiii

332371

219

16

Positive Definite Quadratic Forms• Consider a quadratic form , where A is a symmetric

n×n matrix.We say that A is positive definite if is positive for all nonzero x in Rn, and we call A positive semidefinite if , for all x in Rn. Negative definite and negative semidefinite symmetric matrices are defined analogously. Finally, we call A indefinite if q takes positive as well as negative values.

• A symmetric matrix A is positive definite if (and only if) all of its eigenvalues are positive. The matrix A is positive semidefinite if (and only if) all of its eigenvalues are positive or zero.

• Consider a symmetric n×n matrix A. For m=1,...,n, let A(m) be the m×m matrix obtained by omitting all rows and columns of A past the mth. These matrices A(m) are called the principal submatrices of A. The matrix A is positive definite if (and only if) det(A(m))>0, for all m=1,...,n.

xAxxq )(

)(xq

0)( xq

17

Example 3• Consider an m×n matrix A. Show that the function is a

quadratic form, find its matrix, and determine its definiteness.• (sol)

– We can writeThis shows that q is a quadratic form, with matrix ATA. This quadratic form is positive semidefinite, because for all vectors in Rn.

– Note that if and only if x is in the kernel of A. Therefore, the quadratic form is positive definite if and only if

2)( xAxq

).()()()()()( xAAxxAAxxAxAxAxAxq TTTT

0)( 2 xAxq x

0)( xq

}.0{)ker(

A

18

Example 4• Sketch the curve• (sol)

– We found that we can write this equation aswhere c1, c2 are the coordinates of with respect to the orthonormal eigenbasis for

We sketch this ellipse in Figure 4.– The c1- and the c2-axes are called the principal

axes of the quadratic form– Note that these are the eigenspaces of the

matrix of the quadratic form.

.1548 2221

21 xxxx

,149 22

21 cc

x

,21

51 ,

12

51

21

vv .

5228

A

.548),( 2221

2121 xxxxxxq

.5228

A

19

Principal Axes• Consider a quadratic form , where A is a symmetric

n×n matrix with n distinct eigenvalues. Then the eigenspaces of A are called the principal axes of q. (Note that these eigenspaces will be one-dimensional.)

• Consider the curve C in R2 defined by

Let λ1 and λ2 be the eigenvalues of the matrix of q.both λ1 and λ2 are positive, then C is an ellipse. If there is a positive and a negative eigenvalue, then C is a hyperbola.

xAxxq

)(

.1),( 2221

2121 cxxbxaxxxq

cb

ba2/

2/

20

8.3 Singular Values• Show that if is a linear transformation from R2 to R2,

then there are two orthogonal unit vectors in R2 such that vectors are orthogonal as well (although not necessarily unit vectors).

• (sol)– Following the hint, we first note that matrix ATA is symmetric, since

(ATA)T=AT(AT)T=ATA. The spectral theorem tells us that there is an orthonormal eigenbasis for ATA, with associated eigenvalues λ1, λ2. We can verify that vectors and are orthogonal, as claimed:

xAxL )(

21 and vv

)L( and )( 21 vvL

21,vv

11)( vAvL

22 )( vAvL

.0)()()()()( 212221212121 vvvvvAAvvAvAvAvA TTTT

21

Example 2• Consider the linear transformation where

– Find an orthonormal basis of R2 such that vectors are orthogonal.

– Show that the image of the unit circle under transformation L is an ellipse. Find the lengths of the two semiaxes of this ellipse, in terms of the eigenvalues of matrix ATA.

• (sol)– We will find an orthonormal eigenbasis for matrix ATA:

– The characteristic polynomial of ATA isλ2-125λ+2500=(λ-100)(λ-25),so that the eigenvalues of ATA are λ1=100 and λ2=25.

xAxL )(

6726

A

21,vv )( and )( 21 vLvL

40303085

6726

6276

AAT

.21

span1530

3060ker and

12

span60303015

ker 25100

EE

22

Example 2 (II)– To find an orthonormal basis, we need to multiply these vectors by the

reciprocals of their lengths:

– The unit circle consists of the vectors of the form , and the image of the unit circle consists of the vectors . This image is the ellipse whose semimajor and semiminor axes are . What are the lengths of these axes?Likewise,Thus,

– We can compute the lengths of vectors directly, of course, but the way we did it before is more informative. For example,

so that

21

51,

12

51

21 vv

21 )sin()cos( vtvtx

)()sin()()cos()( 21 vLtvLtxL

)( and )( 21 vLvL

111111111112

1 )()()()( )( vvvvvAAvvAvAvL TTT

22

2 )( vL

.525 )( and 10100 )( 2211 vLvL

)( and )( 21 vLvL

,20

105

11

26726

51)( 11

vAvL

.1020

105

1)( 1

vL

23

Example 2 (III)

24

Singular Values• The singular values of an m×n matrix A are the square roots of the

eigenvalues of the symmetric n× n matrix ATA, listed with their algebraic multiplicities. It is customary to denote the singular values by σ1, σ2,..., σn, and to list them in σ1≥σ2≥···≥σn.

• Let be an invertible linear transformation from R2 to R2. The image of the unit circle under L is an ellipse E. The lengths of the semimajor and the semiminor axes of E are the singular values σ1 and σ2 of A, respectively.

• Let be a linear transformation from Rn to Rm. Then there is an orthonormal basis of Rn such that– Vectors are orthogonal, and– The lengths of vectors are the singular values σ1,

σ2, . . . , σn of matrix A.– To construct , find an orthonormal eigenbasis for matrix ATA.

Make sure that the corresponding eigenvalues λ1, λ2, . . . , λn appear in descending order: λ1≥λ2≥···≥λn.

xAxL

)(

xAxL

)(nvvv

,,, 21

)(,),(),( 21 nvLvLvL

)(,),(),( 21 nvLvLvL

nvvv

,,, 21

25

Example 3• Consider the linear transformation

– Find the singular values of A.– Find orthonormal vectors in R3 such that

are orthogonal.– Sketch and describe the image of the unit sphere under the transformation

L.

• (sol)–

– The eigenvalues are λ1=3, λ2=1, λ3=0. The singular values of A are

.011110

where)(

AxAxL

321 ,, vvv )( and ),(),( 321 vLvLvL

.110121011

011110

011110

AAT

.0 ,1 ,3 332211

26

Example 3 (II)– Find an orthonormal eigenbasis v1, v2, v3 for ATA:

– We compute and check orthogonality:

– We can also check that the length of is σi :

11

1

31,

101

21,

121

61

11

1span)ker(,

101

span,121

span

321

013

vvv

AEEE

321 ,, vAvAvA

.00

,11

21,

33

61

321

vAvAvA

ivA

.0,1,3 332211 vAvAvA

27

Example 3 (III)– The unit sphere in R3 consists of all vectors of the form

– The image of the unit sphere consists of the vectors.1 where 2

322

21332211 cccvcvcvcx

.1 where)()()( 22

212211 ccvLcvLcxL

28

Singular-Value Decomposition (SVD)• If A is an m×n matrix of rank r , then the singular values σ1,..., σr

are nonzero, while σr+1,..., σn are zero.• Any m×n matrix A can be written as A=UΣVT , where U is an

orthogonal m×m matrix; V is an orthogonal n×n matrix; and Σ is an m×n matrix whose first r diagonal entries are the nonzero singular values σ1,... , σr of A, and all other entries are zero (where r = rank(A)). Alternatively, this singular value decomposition can be written as

where the and the are the columns of U and V, respectively.

,111Trrr

T vuvuA

iu

iv

29

Singular-Value Decomposition (SVD) (II)

30

Example 4• Find an SVD for• (sol)

– We find

– The columns of U are defined by

– Finally,

– You can check that A=UΣVT .

.6726

A

2112

51 that so ,

21

51 and

12

51

21 Vvv

21 and uu

.1221

51 thereforeand

,12

511,

21

511

22

211

1

U

vAuvAu

.50010

00

2

1

31

Example 5• Find an SVD for• (sol)

– We find that

– Check that A = UΣVT.

.011110

A

.010003 and

,2/12/12/12/1,

3/12/16/13/106/2

3/12/16/1

UV