Numerical Analysis for Applied Science

Pure and Applied Mathematics: A Wiley Series of Texts, Monographs, and Tracts

Founded by RICHARD COURANTEditors Emeriti: MYRON B. ALLEN III, DAVID A. COX, PETER HILTON, HARRY HOCHSTADT, ERWIN KREYSZIG, PETER LAX, JOHN TOLAND

A complete list of the titles in this series appears at the end of this book.

Numerical Analysis for Applied Science

Second Edition

Myron B. Allen IIIUniversity of WyomingLaramie, USA

Eli L. Isaacson† University of Wyoming Laramie, USA

This edition first published 2019

© 2019 John Wiley & Sons, Inc.

Edition HistoryJohn Wiley & Sons, Inc. (1e, 1998).

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.

The right of Myron B. Allen III and Eli L. Isaacson to be identified as the authors of this work has been asserted in accordance with law.

Registered OfficeJohn Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA

Editorial Office111 River Street, Hoboken, NJ 07030, USA

For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.

Wiley also publishes its books in a variety of electronic formats and by print‐on‐demand. Some content that appears in standard print versions of this book may not be available in other formats.

Limit of Liability/Disclaimer of WarrantyWhile the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Library of Congress Cataloging‐in‐Publication Data

Names: Allen, Myron B., 1954- author. | Isaacson, Eli L., author.Title: Numerical analysis for applied science / Myron B. Allen III

(University of Wyoming, Laramie, USA), Eli L. Isaacson (University of Wyoming, Laramie, USA).

Description: Second edition. | Hoboken, NJ : Wiley, [2019] | Series: Pure and applied mathematics | Includes index. |

Identifiers: LCCN 2018046975 (print) | LCCN 2018055540 (ebook) | ISBN 9781119245667 (Adobe PDF) | ISBN 9781119245650 (ePub) | ISBN 9781119245469 (hardcover)

Subjects: LCSH: Numerical analysis.Classification: LCC QA297 (ebook) | LCC QA297 .A53 2019 (print) | DDC

518–dc23LC record available at https://lccn.loc.gov/2018046975

Cover design: WileyCover image: Courtesy of Myron B. Allen III

Set in 10/12pt WarnockPro by SPi Global, Chennai, India

Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

PREFACE

Once in a while you get shown the lightIn the strangest of places if you look at it right.

Robert Hunter

Preface to the First Edition

We intend this book to serve as a first graduate-level text for applied mathematicians,scientists, and engineers. We hope that these students have had some exposure tonumerics, but the book is self-contained enough to accommodate students with nonumerical background. Students should know a computer programming language,though.

In writing the text, we have tried to adhere to three principles:

1. The book should cover a significant range of numerical methods now used inapplications, especially in scientific computation involving differential equa-tions.

v

vi PREFACE

2. The book should be appropriate for mathematics students interested in the the-ory behind the methods.

3. The book should also appeal to students who care less for rigorous theory thanfor the heuristics and practical aspects of the methods.

The first principle is a matter of taste. Our omissions may appall some readers;they include polynomial root finders, linear and nonlinear programming, digital fil-tering, and most topics in statistics. On the other hand, we have included topics thatreceive short shrift in many other texts at this level. Examples include:

• Multidimensional interpolation, including interpolation on triangles.

• Quasi-Newton methods in several variables.

• A brief introduction to multigrid methods.

• Conjugate-gradient methods, including error estimates.

• Rigorous treatment of the QR method for eigenvalues.

• An introduction to adaptive methods for numerical integration and ordinary dif-ferential equations.

• A thorough treatment of multistep schemes for ordinary differential equations(odes).

• Consistency, stability, and convergence of finite-difference schemes for partialdifferential equations (pdes).

• An introduction to finite-element methods, including basic convergence argu-ments and methods for time-dependent problems.

All of these topics are prominent in scientific applications.

The second and third principles conflict. Our strategy for addressing this con-flict is threefold. First, most sections of the book have a “pyramid” structure. Webegin with the motivation and construction of the methods, then discuss practicalconsiderations associated with their implementation, then present rigorous mathe-matical details. Thus, students in a “methods” course can concentrate on motivation,construction, and practical considerations, perhaps grazing from the mathematicaldetails according to the instructor’s tastes. Students in an “analysis” course shoulddelve into the mathematical details as well as the practical considerations.

Second, we have included Chapter 1, “Some Useful Tools,” which reviews essen-tial notions from undergraduate analysis and linear algebra. Mathematics students

PREFACE vii

should regard this chapter as a review; engineering and applied science students mayprofit by reading it thoroughly.

Third, at the end of each chapter are both theoretical and computational exercises.Engineers and applied scientists will probably concentrate on the computational ex-ercises. Mathematicians should work a variety of both theoretical and computationalproblems. Numerical analysis without computation is a sterile enterprise.

The book’s format allows instructors to use it in either of two modes. For a “meth-ods” course, one can cover a significant set of topics in a single semester by cover-ing the motivation, construction, and practical considerations. At the University ofWyoming, we teach such a course for graduate engineers and geophysicists. For an“analysis” course, one can construct a two- or three-semester sequence that involvesproofs, computer exercises, and projects requiring written papers. At Wyoming, weoffer a two-semester course along these lines for students in applied mathematics.

Most instructors will want to skip topics. The following remarks may help avoidinfelicitous gaps:

• We typically start our courses with Chapter 2. Sections 2.2 and 2.3 (on polyno-mial interpolation) and 2.7 (on least squares) seem essential.

• Even if one has an aversion to direct methods for linear systems, it is worth-while to discuss Sections 3.1 and 3.3. Also, the introduction to matrix normsand condition numbers in Sections 1.4 and 3.6 is central to much of numericalanalysis.

• While Sections 4.1–4.4 contain the traditional core material on nonlinear equa-tions, our experience suggests that engineering students profit from some cov-erage of the multidimensional methods discussed in Sections 4.6 and 4.7.

• Even in a proof-oriented course, one might reasonably leave some of the theoryin Sections 5.3 and 5.4 for independent reading. Section 5.6, The Conjugate-Gradient Method, is independent of earlier sections in that chapter.

• Taste permitting, one can skip Chapter 6, Eigenvalue Problems, completely.

• One should cover Section 7.1 and at least some of Section 7.2, Newton–CotesFormulas, in preparation for Chapter 8. Engineers use Gauss quadrature sooften, and the basic theory is so elegant, that we seldom skip Section 7.4.

• We rarely cover Chapter 8 (on odes) completely. Still, in preparation for Chap-ter 9, one should cover at least the most basic material – through Euler meth-ods – from Sections 8.1 and 8.2.

viii PREFACE

• While many first courses in numerics omit the treatment of pdes, at least somecoverage of Chapter 9 seems crucial for virtually all of the students who takeour courses.

• Chapter 10, on finite-element methods, emphasizes analysis at the expense ofcoding, since the latter seems to lie at the heart of most semester-length en-gineering courses on the subject. It is hard to get this far in a one-semester“methods” course.

We owe tremendous gratitude to many people, including former teachers andmany remarkable colleagues too numerous to list. We thank the students and col-leagues who graciously endured our drafts and uncovered an embarrassing numberof errors. Especially helpful were the efforts of Marian Anghel, Damian Betebenner,Bryan Bornholdt, Derek Mitchum, Patrick O’Leary, Eun-Jae Park, Gamini Wickra-mage, and the amazingly keen-eyed Li Wu. (Errors undoubtedly remain; they are ourfault.) The first author wishes to thank the College of Engineering and Mathematicsat the University of Vermont, at which he wrote early drafts during a sabbatical year.Finally, we thank our wives, Adele Aldrich and Lynne Ipina, to whom we dedicatethe book. Their patience greatly exceeds that required to watch a book being written.

PREFACE ix

Midnight on a carousel ride,Reaching for the gold ring down inside.Never could reachIt just slips away when I try.

Robert Hunter

Preface to the Second Edition

In producing this second edition of Numerical Analysis for Applied Science, I pur-sued two goals. First, I incorporated many suggestions and corrections made bypeople who have used the book since it first appeared in print. I owe my sincerestthanks to colleagues who have shared these improvements over the years. ProfessorScott Fulton of Clarkson University and Professor Aleksey Telyakovskiy of the Uni-versity of Nevada at Reno deserve special thanks for their extraordinary generosityin this respect.

Second, I have incorporated new topics or expanded treatments of existing topics,to reflect some of the evolving applications of numerical analysis during the past twodecades. Among the new contents are the following:

• A description of the symmetric successive overrelaxation method in Chapter 5,to facilitate an expanded discussion of preconditioners later in the chapter.

• A separate section in Chapter 5 on multigrid methods for solving linear systems,including more detail than the first edition’s brief discussion.

• A revised section in Chapter 5 on the conjugate-gradient method, including amore detailed discussion of preconditioners.

• A short discussion in Chapter 5 of the method of steepest descent.

• More details on the power and qr methods for computing eigenvalues in Chap-ter 6.

• An introduction in Chapter 6 to the singular value decomposition and its appli-cation to principal components analysis.

• Revised and expanded discussion in Chapter 9 of the approximation of ellipticpdes by finite-difference methods, including the treatment of irregular bound-aries in two dimensions.

• RevisedL2 approximation error estimates for the finite-element method in Chap-ter 10.

x PREFACE

• An additional section in Chapter 10 on the condition number of the stiffnessmatrix, a major motivation for many advances in numerical linear algebra overthe past three decades.

• Seven new pseudocodes, bringing the total to 32.

• More than twice as many problems as appeared in the first edition.

Also, I moved a section on eigenvalues and matrix norms to Chapter 1 and a sectionon the condition number to Section 3.2, to make it easier to skip most of Chapter 3in favor of iterative methods for linear systems. However, I recommend not skippingSections 3.1 or 3.2. Finally, I removed a short section on Broyden’s method, whichappeared in Chapter 4 of the first edition. I hope these changes make the book moreuseful to the next generation of numerical analysts and modelers.

I owe many thanks to Professor David Isaacson and to staff members at JohnWiley & Sons for helping to settle some of the details associated with the contractfor this edition. Ezhilan Vikraman and Kathleen Pagliaro were especially helpfulwith these matters.

I wish I were writing “we” instead of “I.” My coauthor, Professor Eli Isaacson,passed away in May, 2017. Eli was a gifted mathematician, a superb colleague, andan insightful teacher. I learned a great deal about Mathematics from him, and heand I shared many delightful conversations about how people learn Mathematics andhow we should teach the subject. It was a privilege to know him.

MYRON B. ALLEN IIILaramie, Wyoming

August 2018

CONTENTS

Preface v

1 Some Useful Tools 1

1.1 Introduction 11.2 Bounded Sets 4

1.2.1 The Least Upper Bound Principle 41.2.2 Bounded Sets in Rn 5

1.3 Normed Vector Spaces 81.3.1 Vector Spaces 81.3.2 Matrices as Linear Operators 101.3.3 Norms 121.3.4 Inner Products 151.3.5 Norm Equivalence 17

1.4 Eigenvalues and Matrix Norms 191.4.1 Eigenvalues and Eigenvectors 191.4.2 Matrix Norms 21

1.5 Results from Calculus 26

xi

xii CONTENTS

1.5.1 Seven Theorems 261.5.2 The Taylor Theorem 28

1.6 Problems 33

2 Approximation of Functions 37

2.1 Introduction 372.2 Polynomial Interpolation 38

2.2.1 Motivation and Construction 382.2.2 Practical Considerations 422.2.3 Mathematical Details 432.2.4 Further Remarks 46

2.3 Piecewise Polynomial Interpolation 482.3.1 Motivation and Construction 482.3.2 Practical Considerations 502.3.3 Mathematical Details 542.3.4 Further Remarks 55

2.4 Hermite Interpolation 552.4.1 Motivation and Construction 552.4.2 Practical Considerations 592.4.3 Mathematical Details 60

2.5 Interpolation in Two Dimensions 632.5.1 Constructing Tensor-product Interpolants 642.5.2 Error Estimates for Tensor-product Methods 682.5.3 Interpolation on Triangles: Background 702.5.4 Construction of Planar Interpolants on Triangles 722.5.5 Error Estimates for Interpolation on Triangles 74

2.6 Splines 782.6.1 Motivation and Construction 782.6.2 Practical Considerations 842.6.3 Mathematical Details 852.6.4 Further Remarks 94

2.7 Least-squares Methods 952.7.1 Motivation and Construction 962.7.2 Practical Considerations 1002.7.3 Mathematical Details 1012.7.4 Further Remarks 103

2.8 Trigonometric Interpolation 1042.8.1 Motivation and Construction 105

CONTENTS xiii

2.8.2 Practical Considerations: Fast Fourier Transform 1092.8.3 Mathematical Details 1162.8.4 Further Remarks 118

2.9 Problems 118

3 Direct Methods for Linear Systems 125

3.1 Introduction 1253.2 The Condition Number of a Linear System 1273.3 Gauss Elimination 131

3.3.1 Motivation and Construction 1313.3.2 Practical Considerations 1333.3.3 Mathematical Details 139

3.4 Variants of Gauss Elimination 1483.4.1 Motivation 1483.4.2 The Doolittle and Crout Methods 1483.4.3 Cholesky Decomposition 152

3.5 Band Matrices 1553.5.1 Motivation and Construction 1553.5.2 Practical Considerations 1613.5.3 Mathematical Details 1633.5.4 Further Remarks 166

3.6 Iterative Improvement 1673.7 Problems 169

4 Solution of Nonlinear Equations 175

4.1 Introduction 1754.2 Bisection 179

4.2.1 Motivation and Construction 1794.2.2 Practical Considerations 181

4.3 Successive Substitution in One Variable 1834.3.1 Motivation and Construction 1834.3.2 Practical Considerations 1844.3.3 Mathematical Details 190

4.4 Newton’s Method in One Variable 1924.4.1 Motivation and Construction 1924.4.2 Practical Considerations 1944.4.3 Mathematical Details 199

4.5 The Secant Method 203

xiv CONTENTS

4.5.1 Motivation and Construction 2034.5.2 Practical Considerations 2054.5.3 Mathematical Details 206

4.6 Successive Substitution: Several Variables 2114.6.1 Motivation and Construction 2114.6.2 Convergence Criteria 2134.6.3 An Application to Differential Equations 217

4.7 Newton’s Method: Several Variables 2194.7.1 Motivation and Construction 2194.7.2 Practical Considerations 2214.7.3 Mathematical Details: Newton’s Method 2244.7.4 Mathematical Details: Finite-difference Newton Methods229

4.8 Problems 233

5 Iterative Methods for Linear Systems 239

5.1 Introduction 2395.2 Conceptual Foundations 2435.3 Matrix-Splitting Techniques 248

5.3.1 Motivation and Construction: Jacobi and Gauss–SeidelMethods 248

5.3.2 Practical Considerations 2545.3.3 Mathematical Details 258

5.4 Successive Overrelaxation 2665.4.1 Motivation 2665.4.2 Practical Considerations 2665.4.3 Mathematical Details 2725.4.4 Further Remarks: The Power Method and Symmetric

SOR 2795.5 Multigrid Methods 280

5.5.1 Motivation: Error Reduction Versus Smoothing 2805.5.2 A Two-Grid Algorithm 2845.5.3 V-cycles and the Full Multigrid Algorithm 289

5.6 The Conjugate-Gradient Method 2935.6.1 Motivation and Construction 2935.6.2 Practical Considerations 2985.6.3 Mathematical Details 3035.6.4 Further Remarks: Krylov Methods and Steepest Descent 309

5.7 Problems 311

CONTENTS xv

6 Eigenvalue Problems 317

6.1 More About Eigenvalues 3186.2 Power Methods 323

6.2.1 Motivation and Construction 3236.2.2 Practical Considerations 325

6.3 The QR Decomposition 3286.3.1 Geometry and Algebra of the QR Decomposition 3286.3.2 Application to Least-Squares Problems 3336.3.3 Further Remarks 335

6.4 The QR Algorithm for Eigenvalues 3386.4.1 Motivation and Construction 3386.4.2 Practical Considerations 3416.4.3 Mathematical Details 3476.4.4 Further Remarks 350

6.5 Singular Value Decomposition 3526.5.1 Theory of the Singular Value Decomposition 3526.5.2 Computing Singular Value Decompositions 3546.5.3 Application to Principal Component Analysis 354

6.6 Problems 358

7 Numerical Integration 363

7.1 Introduction 3637.2 Newton–Cotes Formulas 364

7.2.1 Motivation and Construction 3647.2.2 Practical Considerations: Composite Formulas 3677.2.3 Mathematical Details 3697.2.4 Further Remarks 373

7.3 Romberg and Adaptive Quadrature 3737.3.1 Romberg Quadrature 3747.3.2 Adaptive Quadrature 379

7.4 Gauss Quadrature 3857.4.1 Motivation and Construction 3857.4.2 Practical Considerations 3887.4.3 Mathematical Details 390

7.5 Problems 399

8 Ordinary Differential Equations 403

8.1 Introduction 403

xvi CONTENTS

8.2 One-Step Methods 4068.2.1 Motivation and Construction 4068.2.2 Practical Considerations 4098.2.3 Mathematical Details 4108.2.4 Further Remarks: The Runge–Kutta–Fehlberg Algorithm416

8.3 Multistep Methods: Consistency and Stability 4208.3.1 Motivation 4208.3.2 Adams–Bashforth and Adams–Moulton Methods 4228.3.3 Consistency of Multistep Methods 4238.3.4 Stability of Multistep Methods 4268.3.5 Predictor-Corrector Methods 4308.3.6 Mathematical Details: The Root Condition 431

8.4 Multistep Methods: Convergence 4388.4.1 Convergence Implies Stability and Consistency 4398.4.2 Consistency and Stability Imply Convergence 441

8.5 Problems 448

9 Difference Methods for PDEs 453

9.1 Introduction 4539.1.1 Classification 4549.1.2 Characteristic Curves and Characteristic Equations 4559.1.3 Grid Functions and Difference Operators 460

9.2 The Poisson Equation 4629.2.1 The Five-Point Method 4639.2.2 Consistency and Convergence 4669.2.3 Accommodating Variable Coefficients 4719.2.4 Accommodating Other Boundary Conditions 4729.2.5 Accommodating Nonrectangular Domains 473

9.3 The Advection Equation 4759.3.1 The Courant–Friedrichs–Lewy Condition 4769.3.2 Stability of Approximations to Time-Dependent

Problems 4809.3.3 Sufficient Conditions for Convergence 4859.3.4 Further Remarks 488

9.4 Other Time-Dependent Equations 4899.4.1 The Heat Equation 4899.4.2 The Advection-Diffusion Equation 4989.4.3 The Wave Equation 503

CONTENTS xvii

9.5 Problems 505

10 Introduction to Finite Elements 511

10.1 Introduction and Background 51110.1.1 A Model Boundary-Value Problem 51210.1.2 Variational Formulation 513

10.2 A Steady-State Problem 51710.2.1 Construction of a Finite-Element Approximation 51710.2.2 A Basic Error Estimate 52010.2.3 Optimal-Order Error Estimates 52510.2.4 Other Boundary Conditions 52810.2.5 Condition Number of the Finite-Element Matrix 532

10.3 A Transient Problem 53710.3.1 A Semidiscrete Formulation 53710.3.2 A Fully Discrete Method 53910.3.3 Convergence of the Fully Discrete Method 540

10.4 Problems 547

A Divided Differences 549

B Local Minima 553

C Chebyshev Polynomials 555

References 559

Index 563

CHAPTER 1

SOME USEFUL TOOLS

1.1 Introduction

One aim of this book is to make a significant body of mathematics accessible topeople in various disciplines, including engineering, geophysics, computer science,the physical sciences, and applied mathematics. People who have had substantialmathematical training enjoy a head start in this enterprise, since they are more likelyto be familiar with ideas that, too often, receive little emphasis outside departmentsof mathematics. The purpose of this preliminary chapter is to level the playing fieldby reviewing mathematical notations and concepts used throughout the book. Weassume that the reader is familiar with concepts from elementary calculus, such aslimits, continuity, differentiation, and integration. In three sections (2.8, 7.3, and 9.3)we refer to concepts associated with Fourier series.

Virtually every entity in mathematics is a set. If x is an element of the set S, wewrite x ∈ S and say that x belongs to S. If every element of a set R also belongs tothe set S, we say that R is a subset of S and write R ⊂ S. Using this concept, we

Numerical Analysis for Applied Science, Second Edition . Myron B. Allen III and Eli L. Isaacsonc© 2019 John Wiley & Sons, Inc. Published 2019 by John Wiley & Sons, Inc.

1

2 SOME USEFUL TOOLS

say that R = S provided R ⊂ S and S ⊂ R. There are several ways to specify theelements of a set. One way is simply to list them:

R ={

2, 4, 6}, S =

{2, 4, 6, 8, 10, . . .

}.

Another is to give a rule for selecting elements from a previously defined set. Forexample,

R ={x ∈ S

∣∣ x 6 6}

denotes the set of all elements of S that are less than or equal to 6. If the statementx ∈ S fails for all x, then S is the empty set, denoted as ∅.

The notation x = y should be familiar enough, but two related notions are worthmentioning. By x ← y, we mean “assign the value held by the variable y to thevariable x.” Distinguishing between x = y and x ← y can seem pedantic untilone recalls such apparent nonsense as “k = k + 1” that occur in Fortran and otherprogramming languages. Also, we use x := y to indicate that x is defined to havethe value y.

If R and S are sets, then R ∪ S is their union, which is the set containing allelements ofR and all elements of S. The intersectionR∩S is the set of all elementsthat belong to both R and S. If Si is a set for each i belonging to some index set I ,then ⋃

i∈ISi,

⋂i∈I

Si

denote, respectively, the set containing all elements that belong to at least one of thesets Si and the set containing just those elements that belong to every Si. The setdifference R\S = {x ∈ R |x 6∈ S} is the set of all elements of R that do not belongto S. If S1, S2, . . . , Sn are sets, then their Cartesian product S1×S2× · · · ×Sn isthe set of all ordered n-tuples (x1, x2, . . . , xn), where each xi ∈ Si. Two such n-tuples (x1, x2, . . . , xn) and (y1, y2, . . . , yn) are equal precisely when x1 = y1, x2 =

y2, . . . , xn = yn.

Among the most commonly occurring sets in this book are R, the set of all realnumbers; C, the set of all complex numbers x + iy, where x, y ∈ R and i2 = −1,and

Rn := R× R× · · · × R︸ ︷︷ ︸n times

,

INTRODUCTION 3

the set of all n-tuples x = (x1, x2, . . . , xn) of real numbers. We often write thesen-tuples as column vectors:

x =

x1

x2

...

xn

.

R itself has several important types of subsets, including open intervals,

(a, b) :={x ∈ R

∣∣ a < x < b}

;

closed intervals,[a, b] :=

{x ∈ R

∣∣ a 6 x 6 b}

;

and the half-open intervals

[a, b) :={x ∈ R

∣∣ a 6 x < b}, (a, b] :=

{x ∈ R

∣∣ a < x 6 b}.

To extend this notation, we sometimes use the symbol∞ in a slightly abusive fash-ion:

(a,∞) :={x ∈ R

∣∣ a < x},

(−∞, b] :={x ∈ R

∣∣ x 6 b},

(−∞,∞) := R,

and so forth.

In specifying functions, we write f : R→ S. This graceful notation indicates thatf(x) is defined for every element x belonging to R, the domain of f , and that eachsuch value f(x) belongs to the set S, called the codomain of f . The codomain off contains as a subset the set f(R) of all images f(x) of points x belonging to thedomain R. We call f(R) the range of f .

The notation f : x 7→ y indicates that f(x) = y, the domain and codomain of fbeing understood from context. Sometimes we write x 7→ y when the function itselfas well as its domain and codomain are understood from context.

Throughout this book we assume that readers are familiar with the basics of calcu-lus and linear algebra. However, it may be useful to review a few notions from thesesubjects. We devote the rest of this chapter to a summary of facts about bounded setsand normed vector spaces and some frequently used results from calculus.

4 SOME USEFUL TOOLS

1.2 Bounded Sets

In numerical analysis, sets of real numbers arise in many contexts. Examples includesequences of approximate values for some quantity, ranges of values for the errors insuch approximations, and so forth. It is often important to estimate where these setslie on the real number line – for example, to guarantee that the possible values for anumerical error lie in a small region around the origin. We say that a set S ⊂ R isbounded above if there exists a number B ∈ R such that x 6 B for every x ∈ S. Inthis case, B is an upper bound for S. Similarly, S is bounded below if, for someb ∈ R, b 6 x for every x ∈ S. In this case, b is a lower bound for S. A boundedset is one that is bounded both above and below. A set S is bounded if and only ifthere exists a number M ∈ R such that |x| 6M for every x ∈ S.

By extension, if f : S → R is a function whose range f(S) is bounded above,bounded below, or bounded, then we say that f is bounded above, bounded below,or bounded, respectively.

1.2.1 The Least Upper Bound Principle

Most upper and lower bounds give imprecise information. For example, 17 is anupper bound for the set S = (0, 2), but, as Figure 1.1 illustrates, the upper bound 2 issharper. We call B0 a least upper bound or supremum for S ⊂ R if B0 is an upperbound for S andB0 6 B wheneverB is an upper bound for S. In this case, we writeB0 = supS. Similar reasoning applies to lower bounds: −109 is a lower bound for(0, 2), but so is the more informative number 0. We call b0 a greatest lower boundor infimum for S ⊂ R if b0 is a lower bound for S and b0 > b whenever b is alsoa lower bound for S. We write b0 = inf S. The notations inf and sup have obviousextensions. For example, if S2 :=

{(x, y) ∈ R2 : x2 + y2 = 1

}denotes the unit

circle in R2 and f : S2 → R is a real-valued function defined on S2, then

supS2

f := sup(x,y)∈S2

f(x, y) := sup{f(x, y) ∈ R

∣∣ x2 + y2 = 1}. (1.1)

Shortly we discuss conditions under which this quantity exists.

(2 170)

Figure 1.1 The set (0, 2) ⊂ R and two of its upper bounds.

Not every set has a supremum or an infimum. For example, the set

Z ={. . . ,−2,−1, 0, 1, 2, . . .

}

BOUNDED SETS 5

of all integers has neither a supremum nor an infimum. The set

N ={

1, 2, 3, . . .}

of natural numbers has infimum 1 but no supremum. One should take care todistinguish between supS and inf S and the notions of maximum and minimum.By a maximum of a set S ⊂ R, we mean an element M ∈ S for which x 6 M

whenever x ∈ S, and we write M = maxS. Thus sup(0, 2) = 2 = sup[0, 2] =

max[0, 2], but max(0, 2) does not exist. Similarly, an element m ∈ S is a minimumof S if m 6 x for every x ∈ S. Thus, inf(0, 2) = 0 = inf[0, 2] = min[0, 2], whilemin(0, 2) does not exist. These examples illustrate the fact that sup and inf are moregeneral notions than max and min: supS = maxS when supS ∈ S, but supS

may exist even when maxS does not. A corresponding statement holds for inf andmin.

The following principle, which one can take as a defining characteristic of R,confirms the fundamental importance of sup and inf:

Least-upper-bound principle. If a nonempty subset of R is bounded above,then it has a least upper bound.

Spivak [46, Chapter 8] gives an accessible introduction to this principle. Similarly,every nonempty subset of R that is bounded below has a greatest lower bound. Forexample,

inf{

12 ,

13 ,

14 , . . .

}= 0, sup(−∞, 0) = 0, sup

{2, 4, 6

}= 6.

The set{

2, 4, 6, 8, 10, . . .}

, however, is not bounded above, and it has no least upperbound. The least-upper-bound principle ensures that supS2

f , defined in Eq. (1.1),exists whenever the set of real numbers{

f(x, y) ∈ R∣∣ (x, y) ∈ S2

}is bounded above. However, without knowing more about f , we cannot guaranteethe existence of a point (x, y) ∈ S2 where f attains the value supS2

f .

1.2.2 Bounded Sets in Rn

Which subsets of Rn are bounded? Here we generally have no linear order analogousto the relation 6 on which to base a definition of boundedness. Instead, we rely onthe idea of distance, which is familiar from geometry.

6 SOME USEFUL TOOLS

Definition. The Euclidean length of x = (x1, x2, . . . , xn) ∈ Rn is

‖x‖2 :=

√√√√ n∑j=1

x2j .

The Euclidean distance between two points x,y ∈ Rn is the Euclidean length oftheir difference, ‖y − x‖2.

Given a point x ∈ Rn and a positive real number r, we call the set of all pointsin Rn whose Euclidean distance from x is less than r the ball of radius r about x.We denote this set as Br(x). Figure 1.2 depicts such a set in R2. A set S ⊂ Rn isbounded if it is a subset of Br(0) for some r > 0. Observe that, if x ∈ R = R1,then Br(x) = (x− r, x+ r). One easily checks that a subset of R is bounded in thissense if and only if it is bounded above and below.

x

r

Figure 1.2 The ball Br(x) of radius r about the point x ∈ R2.

Other structural aspects of Rn also prove useful. Let S ⊂ Rn. A point x ∈ S isan interior point of S if there is some ball Br(x) such that Br(x) ⊂ S. In Figure1.3, the point a is an interior point of S, but b and c are not. A point x ∈ Rn (notnecessarily belonging to S) is a limit point of S if every ball Br(x) contains at leastone element of S distinct from x. In Figure 1.4, a and b are limit points of S, butc is not. If every element of S is an interior point, then we call S an open set. If Scontains all of its limit points, then we say that S is a closed set. The definitions areby no means mutually exclusive: ∅ and Rn are both open and closed.

Finally, a subset of Rn that is both closed and bounded is compact.1 Thus thefollowing subsets of R2 are compact:

[0, 1]× [0, 1],{

(0, 0), (0, π), (1,−π)}, S2 =

{x ∈ R2

∣∣ ‖x‖2 = 1},

while the sets

(0, 1)× (0, 1), B1(0),{

(0, 0), (1, 1), (2, 2), . . .}

1This characterization of compactness is not the most general one, but it suffices for Rn. For the moregeneral definition, see [40, Chapter 2].

BOUNDED SETS 7

S

a

b

c

Figure 1.3 A set S ⊂ R2, showing an interior point a and two points b, c that are notinterior points.

S

a

b

c

Figure 1.4 A set S ⊂ R2, along with two limit points a and b and a point c that is not alimit point of S.

are not. Compact sets in Rn have several interesting properties, one of which isespecially useful in numerical analysis.

Theorem 1.2.1 (maximum and minimum values on compact sets) IfS ⊂ Rn is nonempty and compact and f : S → R is a continuous function, thenthere are points a,b ∈ S for which f(a) and f(b) are the minimum and maximum,respectively, of the set f(S).

For a proof, see [40, Chapter 4].

This theorem partially settles an issue raised earlier: If f is a continuous, real-valued function defined on the unit circle S2, then there is at least one point (x, y) ∈S2 where f takes the value supS2

f defined in Eq. (1.1). By considering the function−f , one can also show that f takes the value infS2 f at some point in S2. Both ofthese statements hold just as well in Rn, where S2 :=

{x ∈ Rn | ‖x‖2 = 1

}. We

use this generalization in the next section.

8 SOME USEFUL TOOLS

1.3 Normed Vector Spaces

1.3.1 Vector Spaces

Vector spaces are ubiquitous.

Definition. A set V is a vector space over R if there are two operations, addition(+) and scalar multiplication, that obey the following rules for any x, y, z ∈ V anda, b ∈ R:

1. x+y ∈ V and ax ∈ V; in other words, V is closed algebraically under additionand scalar multiplication.

2. x+ y = y + x.

3. x+ (y + z) = (x+ y) + z.

4. There is a unique vector 0 ∈ V such that x+ 0 = x for all x ∈ V.

5. For any x ∈ V, there is a unique vector −x ∈ V such that −x+ x = 0.

6. 1x = x.

7. a(bx) = (ab)x.

8. a(x+ y) = ax+ ay.

9. (a+ b)x = ax+ bx.

We refer to R as the field of scalars. The elements of V are vectors. A set U is asubspace of V if every element of U belongs to V and U is a vector space under theoperations that it inherits from V. Analogous definitions hold for vector spaces overthe field C of complex numbers.

We denote the scalar multiple ax by juxtaposing the scalar a and the vector x. Inmost cases of interest in this book, the algebraic properties of addition and scalarmultiplication are obvious from the definitions of the two operations, and the mainissue is whether V is closed algebraically under these two operations.

Among the common examples of vector spaces are the finite-dimensional Euclid-ean spaces Rn, with their familiar rules of addition and scalar multiplication:

x + y =

x1

...

xn

+

y1

...

yn

:=

x1 + y1

...

xn + yn

;

NORMED VECTOR SPACES 9

ax = a

x1

...

xn

:=

ax1

...

axn

.In this vector space, the zero vector is 0, the array that has 0 as each of its n entries.The real line R is perhaps the simplest Euclidean space.

Various sets of functions constitute another important class of vector spaces. Forexample, if S ⊂ R is an interval, then Ck(S) signifies the vector space of all func-tions f : S → R for which f and its derivatives f ′, f ′′, . . . , f (k) through order kare continuous. By extension of this notation, C∞(S) denotes the vector space offunctions that have continuous derivatives of all orders on S. On all of these spaceswe define addition and scalar multiplication pointwise:

(f + g)(x) := f(x) + g(x); (af)(x) := a f(x).

Here, the vector 0 is the function that assigns the number 0 to all arguments x. Aslightly more general function space is L2(S). Although the rigorous definition ofthis space involves some technicalities, for our purposes it suffices to think of L2(S)

as the set of all functions f : S → R for which∫Sf2(x) dx exists and is finite.

Readers who are curious about the technicalities may consult [40, Chapter 11].

A third class of vector spaces consists of the sets Rm×n of real m × n matrices.Our notational convention is to a use sans-serif capital letter, such as A, to signify thematrix whose entry in row i, column j is the number denoted by the correspondinglowercase symbol ai,j . If C and D are two such matrices, then

C + D =

c1,1 · · · c1,n

......

cm,1 · · · cm,n

+

d1,1 · · · d1,n

......

dm,1 · · · dm,n

:=

c1,1 + d1,1 · · · c1,n + d1,n

......

cm,1 + dm,1 · · · cm,n + dm,n

,

aC :=

ac1,1 · · · ac1,n

......

acm,1 · · · acm,n

.The additive identity in Rm×n is the m× n matrix 0 all of whose entries are 0.

Finally, the set {0} is trivially a vector space.

10 SOME USEFUL TOOLS

One can use addition and scalar multiplication to construct subspaces.

Definition. If V is a real vector space, a linear combination of the vectorsx1, x2, . . . , xn ∈ V is a vector of the form c1x1 + c2x2 + · · · + cnxn, wherec1, c2, . . . , cn ∈ R. If S ⊂ V, the span of S, denoted span(S), is the set of alllinear combinations of vectors belonging to S. If U = span(S), then S spans U.

Problem 1.2 asks for proof that span(S) is a subspace of V whenever S ⊂ V.

Definition. If V is a vector space, then a set S ⊂ V is linearly independent if novector x ∈ S belongs to span (S\{x}), that is, no vector in S is a linear combinationof the other vectors in S. Otherwise, S is linearly dependent.

One can regard a linearly independent set as containing minimal information neededto determine its span.

Definition. A subset S of a vector space V is a basis for V if S is linearly inde-pendent and span(S) = V.

A basic theorem of linear algebra asserts that, whenever two finite sets S1 and S2 arebases for a vector space V, S1 and S2 have the same number of elements (see Ref.[48, Chapter 2]) We call this number the dimension of V. For example, Rn has thestandard basis {e1, e2, . . . , en}, where

e1 :=

1

0...

0

, e2 :=

0

1...

0

, . . . , en :=

0

0...

1

.

If V has a basis containing finitely many vectors, then we say that V is finite-dimensional. If not, then V is infinite-dimensional.

1.3.2 Matrices as Linear Operators

Given matrices A ∈ Rm×n and B ∈ Rn×p, one can compute their matrix product

AB =

a1,1 · · · a1,n

......

am,1 · · · am,n

b1,1 · · · b1,p

......

bn,1 · · · bn,p

=

c1,1 · · · c1,p

......

cm,1 · · · cm,p

,