Second Edition
DeKalb, Illinois
Copyright© 2010 by the Society for Industrial and Applied Mathematics
10987654321
/'.II rights reserved. Pnnted in the United States of America. No part of this book may be reproduced, stored, or transmitted in any manner without the written permission of the publisher. For informatron, write to the Society for lndustnal and Applied Mathematics, 3600 Market Street, 6th Floor, Philadelphia, PA 19104-2688 USA.
Trademarked names may be used 1n this book without the inclusion of a trademark symbol. These names are used in an editorial context only; no infringement of trademark is intended.
MATLI\B is a registered trademark of The Math Works, Inc. For MATLAB product information, please contact The Math Works, Inc .. 3 Apple Hill Drive, Natick, MA 01760-2098 USA, 508-647-7000, Fax: 508-647-7001, info@mathworks.com, www.mathworks.com.
Library of Congress Cataloging-in-Publication Data Datta, Biswa Nath.
Numerical linear algebra and applications I Biswa Nath Datta.-- 2nd ed. p. em.
Includes bibliographical references and index. ISBN 978-0-898716-85-6 1. Algebras, Linear. 2. Numerical analysis. I. Title. QA 184.2.038 2009 512' 5--dc22
2009025104
Karabi
tor her love ot the subject and her endless encouragement
Preface O.l 02 0.3 0.4
0.5
Contents
Special Features . . . . . .... Additional Features and Topics for Second Edition Intended Audience . . . . . . . Some Guidelines for Using tl1is Book ...... . 0.4.1 A First Course in Numerical Linear Algebra (Advanced
Undergraduate/One Semester) ......... . 0.4.2 A Second Course in Numerical Linear Algebra (Beginning
0.4.3 Gruduate Course) . . . . . . . . A One-Semester Course in Numerical Linear Algebra for Engineers
Acknowledgments . . . . . .
:X"Yii
1 Linear Algebra Problems, Tbeir Importance, and Computational Difficulties l
2
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . l l.2 Fundamental Linear Algebra Problems and Their Importance . . . . . l 1.3 Computational Difficulties Using Theoretical Linear Algebra Techniques 3
A Review of Some Required Concepts from Core Linear Algebra 2.1 Introduction . . . . . . . . . . , . . . . . 2.2
2.3
2.4
2.5
Orthogonality, Subspace, and Basis . Vectors . 2.2.1 Tv1atrices 2.3.1 2.3.2 2.3.3
Range and Null Spaces Rank of a Matrix ... The Inverse of a Matrix
2.3.4 Similar Matrices .. Some Special Matrices 2.4.1 Diagonal and Triangular lv1atriccs 2.4 .2 Unitary and Orthogonal Matrices . 2.4.3 Symmetric and Hermitian Matricc.s . 2.4.4 Hesscnberg Matrices (Almost Triangular) Vector and Matrix Nurms 2.5.1 Vector Norms 2.5.2 Matrix Nonns .
vii
7 7 7 8 y
12 12 l3 14 14 14 15 Hi 16 17 17 18
viii
3
4
Contents
2.5.3 Norms and Inverses ...... , .... . 2.5.4 Norm Invariant Properties of Orthogonal and Unitary
Matrices ..... . 2.6 Singular Vaiue Decomposition 2.7 Review and Summary .....
2.7 .I Special Matrices . 2.7 .2 Rank, Determinant, Inverse. and Eigenvalues 2.7.3 Vector and Matrix Norms
2.8 Suggestions for Further Reading Exercises on Chapter 2 . , . . . . . . .
Floating Point Numbers and Errors in Computntions 3.1 Flouting Point Number Systems . . 3.2 Rounding Errors 33· Laws of Floating Point Arithmetic 3.4 Addition of 11 Flouting Point Numbers 3.5 Multiplication of n Floating Point Numbers 3.6 Inner Product Computation . . . . . 3.7 Error Bounds for Floating Point Matrix Operations . 3.8 Round-Off Errors Due to Cancellation and Recursive Computations . 3.9 Review and Summary ... 3.! 0 Suggestions for Further Reading Exercises on Chapter 3 . . . . . . . . .
Stability of Algorithms and Conditioning of Problems 4.1 Introduction ......... .
4.1.! Computing the Norm of a Vector ... 4.1.2 Computing the Inner Product of Two Vectors 4.1.3 Solution of an Upper Triangular System 4.1.4 Solution of a Lower Triangular System .
4.2 Efficiency of an Algorithm 4.3 Definition and Concept of Stability 4.4 Conditioning of the Problem and
Perturbation Analysis 4.5 Conditioning of the Problem, Stability of the Algorithm, and Accuracy
of the Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Perturbation Analysis of the Linear System Problem . . ....
4.6.1 EtTcct of Perturbation on the Right-Hand Side Vector b 4.6.2 Effect of Perturbation in the Matrix A 4.6.3 Effect of Perturbations in Both Matrix A and Vector b .
4. 7 Some Properties of the Condition Number of a Matrix . . 4.7.1 Some Well-Known HI-Conditioned Matrices .. 4.7 .2 How Large Must the Condition Number Be for
Ill-Conditioning? . . . . . . . . . . . . . . . 4.7.3 The Condition Number and Nearness to Singularity 4.7.4 Examples of Ill-Conditioned Eigenvalue Problems
4.8 Some Guide1ines for Designing Stable Algorithms .....
22
22 23 24 24 24 24 24 25
29 29 31 33 36 37 38 39 40 44 45 45
49 49 49 49 51 51 52 52
57
69 69 70 72
Contents ix
4.9 Review and Summary , , , ... 4.9. I Conditioning of the Problem 4.9.2 Stability of an Algorithm , 4,9,3 Effects of Conditioning and S!ability on the Accuracy
of the Solution , 4.10 Suggestions for Further Reading Exercises on Chapter 4 . . . . . .
72 72 73
74 74 74
6
5. I A Computational Template in Numerical Linear Algebra , 8 I 5.2 LU Factorization Using Gaussian Elimination . 82
5.2.1 Creating Zeros in a Vector or ~latrix Using Elementary Matrix , . . . , . . . , , . . . . . . . 83
5.2.2 Triangularization Using Gaussian Eliminntion . . . 84 5.2.3 Permutation Matrices and Their Properties . . . 96 5.2.4 Gaussian Elimination with Partial Pivoting (GEPP) 97 5.2.5 Gaussian Elimination with Complete Pivoting (GECP) I 04 5.2.6 Summary of Gaussian Elimination and LU Factoriwtions. 106
5.3 Stability of Gaussian Elimination . . . . . . . . 106 5.4 Summary and Table of Comparisons . l09
5.4.1 Elementary Lower Triangular Matrix . 110 5.4.2 LU Factorization . . . . . . . . . 110 5.4.3 Stability of Gaussian Elimination . 110 5.4.4 Table of Comparisons. 110
5.5 Suggestions for Further Reading 110 Exercises on Chapter 5 . . . . . Ill
Numerical Solutions of Linear Systems 6.1 Introduction . , .. , . , ... 6.2 6.3
6A
6.5
Basic Results on Existence and Uniqueness . . . . . Some Applications Giving Rise to Linear Systems Problems . 6.3. I An Electric Circuit Problem , 6.3.2 Analysis of a Processing Plant Consisting of Interconnected
Reactors . . . . . . . . . . . . . . . , . . . . . . . . . . 6.3.3 Linear Systems Arising from Ordinary Differential Equa­
tions: A Case Study on a Spring-Mass Problem ..... , 6.3.4 Linear Systems Arising from Partial Di!Tcrcntial Equations:
A Case Study on Temperature Distribution . , . 6.3.5 Approximation of a Function by a Polynomial:
Hilbert System , , , .. , , . , , .... , . . , ... LU Factorization Methods . . . . . . . . . . . . . 6.4. I Solution of the System Ax = b Using LU Factorization . 6.4.2 Solution of Ax = b Using Factorization: M A = U 6.4.3 Solution of Ax = b without Explicit Factorization . 6.4.4 Solving a Linear System with Multiple Right-Hand Sides Scaling
117 117 118 ll9 119
120
122
123
X
7
6.6
6.7
6.8
Contents
Concluding Remarks on the Use of Gaussian Elimination for Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . 136 Inverses and Detcnninant , . . . . . . . . . . . . . . . . . . 136 6.7.1 Avoiding Explicit Computation of the Inverses . . . . 137 6.7 .2 The Shcnnan-Morrison Forrnula for Matrix Inverse . 137 6.7.3 Computing the Inverse of an Arbitrary Nonsingu1ar Matrix 138 6.7.4 Computing the Determinant of a Matrix . . . . . . . 139 Effect of the Condition Number on Accuracy of the Computed Solution . . . . . . . . . . . . . . . 139 6.8.1 Conditioning and Pivoting . . . . 140 6.8.2 Conditioning and Scaling . . 141 Computing and Estimating the Condition Number J 41 Componentwise Perturbations and the Errors . . . 143 Iterative Refinement , . . . . . . . . . . . . 144 sp·ec-tai·Systems:-·Positivc Definite. Diagonatly Dominant, .. Hessenherg, and Tridiagonal . . . . . . . . . . . . . . . . 146 6. 12. I Special Linear Systems Arising from Finite Difference
Methods . . . . . . . . . . . . . . . . . . . . . 147 6. i 2.2 Special Linear Systems Arising from Finite Element
Methods . . . . . . . . . . . . . . . . 150 6. 12.3 Symmetric Positive Definite Systems. 6.12.4 Hesscnberg System . . . . . . 6.12.5 Diagonally Dominant Systems 6.12.6 Tridiagonal Systems . . . . . .
6.13 Review and Summary ......... .
!53 !58 160 161 167 167 168 168 169 169 169 170
6.13.1 Numerical Methods for Arbitrary Linear System Problems 6. 13.2 Special Systems , . . . . . . . . . . . . . . . . . 6.13.3 Inverse and Detcnninant .. , ......... . 6.13.4 The Condition Number and Accuracy of Solution 6.13.5 Iterative Refinement.
6.14 Suggestions for Further Reading Exercises on Chapter 6 . , . . , . . . .
QR Factorization, Singular Value Decomposition, and Projections 7.1 Introduction .. , ........ . 7.2 Householder's Matrices and QR Factorization . . ..
7.2.1 Definition and Basic Properties .... 7.2.2 Householder's Method for QR Factorization
7.3 Complex QR Factorization . . . . , . 7.4 Givens Matrices and QR Factorization , ...
7.4.1 Definition and Basic Properties . 7.4.2 Givens Method for QR Factorization 7.4.3 QR Factorization of a Hessenberg Matrix Using Givens
MaLrices . . . . , . . 7.5 Classical and Modified Gram-Schmidt Algorithms for
QR Factorizations . . . . . . . . . . . . . 7.6 Solution of Ax =bUsing QR Factorization .....
181 181 183 183 188 194 194 194 199
.201
Projections Using QR Factorization . . . , , ... 7.7. I Orthogonal Projections and Orthonmmal Bases 7. 7.2 Projection of a Vector omo the
7.7.3 Range and the Null Space of a Matrix , , . Orthonormal Bases und Orthogonal Projections onto the Range and Null Space Using QR Factorization
Singular Value Decomposition and Its Properties 7.8. I Singular Values and Singular Vectors . , .. , 7.8.2 Computation of the SVD (MATLAB Commandj 7.8.3 The SVD ol' a Complex Matrix ..... 7 .8.4 Geometric Interpretation of the Singular Values and Singular
'209 . 209
7.8.9
Reduced SVD , . , . . . . . . . . . . . 214 Sensitivity of the Singular Values ..... , . 2 !5 Norms, Condition Number. and Rank via the SVD . 216 The Distance to Singularity, Runk-Deficiency, and Numeri- cal Rank via the SVD. , ........ . Numerical Rank . .
. 217
. 219 7.8.1 0 Orthonormal Bases and Projections from the SVD . . 220
7.9 Some Practical Applications of the SVD ...... , ... , , 221 7JO Geometric Mean and Generalized Triangular Decompositions . 226 7.11 Rcvic\V and Summary . . . . . . . . . 227
7. I 1.1 QR Factorization . . . . . . 227 7. 11.2 The SVD, GMD, and GTD , 227 7 .11.3 Projections . . 228
7. !2 Suggestions for Further Reading . 228 Exercises on Chapter 7 . . . . . . . . . . 228
Least-Squares Solutions to Linear Systems 8.1 Introduction . . . . . . . . . . . . .
237 . 237
8.2 Geometric Interpretation of the Least-Squares Problem . 238 8.3 Existence and Uniqueness . . . . . . . 239
8.3.1 Existence and Uniqueness Theorem . . . . . . 240 8.3.2 Normal Equations. Projections. and Least-Squares Solutions 24 I
8.4 Some Applications of the Least-Squares Problem. . . 242 8.4. I Polynomial-Filling to Experimental Data . . 242 8.4.2 Predicting Future Sales . . , 245
8.5 Pseudoinverse and the Least-Squares Problem . . . . 245 8.6 Sensitivity of the Least-Squares Problem . . . . . . . 246 8.7 Computational Methods for Overdetermined Problems: Normal Equa-
tions, QR, and SVD Methods . 252 8.7. I The Normal Equations Method . 252 8,7.2 QR Factorization Method .. , , 254 8.7.3 The SVD Method . . . . 259 8.7.4 Solving the Linear System Using the SVD . 263
8.8 Underdctermined Linear Systems . . . 263 8.8.1 The QR Approach for the Minimum-Norm Solution . . 264
xii
8.8.2 The SVD Approach for the Minimum-Nom1 Solution Least-Squares Iterative Refinement . Review and Summary . . . . . . . 8.10.1 Existence and Uniqueness 8.10.2 Overdetermined Problems 8.1 0.3 The Underdetennined Problem 8.10.4 Perturbation Analysis ..... 8.1 0.5 llerati ve Refinement . . . . . . 8.10.6 Comparison of Least-Squares Methods.
8.11 Suggestions for Further Reading Exercises on Chapter 8 . , . , . . . .
Contents
. 265
. 266
. 269
. 269
. 269
. 270
. 270
. 270
. 270
. 271
. 271
9 Numerical Matrix Eigenvalue Problems 281 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 281 9.2· ··· ·· EigenvalueProblemsArising in Practical Applications . ·'" . . 282
9 .2.1 Stability Problems for Differential and Difference Equations . . . . . . . . . . . . . . . . . . . . . 282
9.2.2 Phenomenon of Resonance . . . . . . . . . . . . . 285 9.2.3 Buckling Problem (a Boundary Value Problem) . . 286 9.2.4 Simulating Transient Current for an Electric Circuit . 288 9 .2.5 An Example of the Eigenvalue Problem Arising in Statistics:
Principal Component Analysis . 290 9.3 LocaJizalion of Eigenvalues . . . . . . . . . 292
9.3.1 The GerSgorin Disk Theorems . . . . . 293 9.3.2 Eigenvalue Bounds and Matrix Norms . 295
9.4 Computing Selected Eigenvalues and Eigenvectors. . 295 9.4.1 The Power Method, the Inverse Iteration. and the Rayleigh
Quotient Iteration . . . . . . . . . . ... , . . 296 9.5 Similarity Transformations and Eigenvalue Computations . . . . . . 304
9.5.1 Diagonalization of a Matrix . . . . . . . . . . . . . 305 9.5.2 Numerical Instability of Nonorthogona1 Diagonalization . 306 9.5.3 Reduction to Hcssenberg Form via Orthogonal Similarity . 307 9.5.4 Uniqueness of Hcssenberg Reduction .... , . 312 9.5.5 Eigenvalue Computations Using the Characteristic
Polynomial . . . . . . . . 313 9.6 Eigenvalue Sensitivity . . . . . . . . . . . . . . . . 315
9.6.1 The Bauer-Fike Theorem . . . . . . . . . . 315 9.6.2 Sensitivity of the Individual Eigenvalues. . 317
9.7 Eigenvector Sensitivity . . . . 319 9.8 The Real Schur Fonn and QR Iterations . 320
9.8.1 The Basic QR Iteration . . . . . 322 9.8.2 The Hessenberg QR Iteration . . 324 9.8.3 Convergence of the QR Iterutions and the Shift of Origin . 325 9.8.4 The Double-Shift QR Iteration . . 326 9.R.5 Implicit QR Iteration . . . 327 9.8.6 Obtaining the Real Schur Form A . 331 9.8.7 The Real Schur Fom1 and Invariant Subspaccs . . 334
Contents xiii
9.9 Computing the Eigenvectors. . . . . . . 335 9.9.1 The Hessenberg-Inversc Iteration . . 335
9.10 Review and Summary . 336 9.10.1 Applications of the Eigenvalues and Eigenvecwrs . 336 9.10.2 Localization of Eigenvalues . . . . . . . . . 336 9.10.3 The Power Method and the Inverse Iteration . 337 9.10.4 The Rayleigh Quotient Iteration . . . . . . . 337 9.10.5 Sensitivity of Eigenvalues and Eigenvectors . 337 9.10.6 Eigenvalue Computation via the Characteristic Polynomial
and the Jordan Canonical Form . 338 9.10.7 Hesscnberg Transformation. . 338 9. 10.8 The QR Iteration Algorithm . . 338 9.10.9 Ordering the Eigenvalues . 339 9.10. IO Computing the Eigenvectors . 339
9.1 I Suggestions Jor Flllther Reading . 339 Exercises on Chapter 9 . . . . . . . 341
Numerical Symmetric Eigenvalue Problem and Singular Value Decomposition 351
. 351
. 352 10.1 Introduction ..... 10.2 Computational Methods for the Symmetric Eigenvalue Problem .
!0.2.1 Some Special Properties of the Symmetric Eigenvalue Problem . . . . . . . . . . . . . . . ....
10.2.2 The Bisection Method for the Symmetric Tridiagonal . 352
Matrix . . . . . . . . . . . . . . . . 354 10.2.3 The Symmetric QR Iteration Method . . 357 I 0.2.4 The Divide-and-Conquer Method . . . . 359 10.2.5 The Jacobi Method . . . . . 363 10.2.6 Comparison of the Symmetric Eigenvalue Methods . 364
10.3 The Singular Value Decomposition and Its Computation . . . . 365 10.3.1 The Relationship between the Singular Values and the
Eigenvalues . . . . . . . . . . . . , .. , . 366 10.3.2 Sensitivity of the Singular Values. . . . . . . . 367 10.3.3 Computing the Variance-Covariance Matrix with SVD . 367 10.3.4 Computing the Pscudoinverse with SVD . . 368 10.3.5 Computing the SVD . 369 10.3.6 The Golub-Knhan-Reinsch Algorithm . 370 10.3.7 The Chan SVD Algorithm . 378
10.4 Generalized SVD . . . . . . . 379 10.5 Review and Summary . 380
10.5.1 The Symmetric Eigenvalue Computation . . 380 !0.5.2 The SVD . . . 380
10.6 Suggestions for Further Reading . 381 Exercises on Chapter I 0 . . . . . . . . . 381
Generalized and Quadratic Eigenvalue Problems ll.l Introduction . . . . . . .
387 . 387
Eigenvalue-Eigenvector Properties of Equivalent Pencils Generalized Schur and Real Schur Decompositions The QZ Algorithm . . . . . . . . 11.4.1 Stage I: Reduction to Hcsscnberg Triangular Form . 11.4.2 Stage II: Reduction to the Generalized Real Schur Fonn . Computations of Generalized Eigenvectors . , . , , . The Symmetric Positive Definite Generalized Eigenvalue Problem 11.6. 1 Eigenvalues and Eigenvectors of Symmetric Definite
Pencil , ........... · · · · · · · · 11.6.2 Conditioning of the Eigenvalues of the Symmetric Definile
. 389
. 389
. 390
. 391
. 393
. 398
. 399
. 399
Pencil . . . . . . . . . . . 400 11.6.3 11.6.4
The QZ Method…