Heinz H. Bauschke

Patrick L. Combettes

Convex Analysis and

Monotone Operator Theory

in Hilbert Spaces

Springer

Foreword

This self-contained book offers a modern unifying presentation of three ba-sic areas of nonlinear analysis, namely convex analysis, monotone operatortheory, and the fixed point theory of nonexpansive mappings.

This turns out to be a judicious choice. Showing the rich connectionsand interplay between these topics gives a strong coherence to the book.Moreover, these particular topics are at the core of modern optimization andits applications.

Choosing to work in Hilbert spaces offers a wide range of applications,while keeping the mathematics accessible to a large audience. Each topic isdeveloped in a self-contained fashion, and the presentation often draws onrecent advances.

The organization of the book makes it accessible to a large audience. Eachchapter is illustrated by several exercises, which makes the monograph anexcellent textbook. In addition, it offers deep insights into algorithmic aspectsof optimization, especially splitting algorithms, which are important in theoryand applications.

Let us point out the high quality of the writing and presentation. The au-thors combine an uncompromising demand for rigorous mathematical state-ments and a deep concern for applications, which makes this book remarkablyaccomplished.

Montpellier (France), October 2010 Hedy Attouch

vii

Preface

Three important areas of nonlinear analysis emerged in the early 1960s: con-vex analysis, monotone operator theory, and the theory of nonexpansive map-pings. Over the past four decades, these areas have reached a high level ofmaturity, and an increasing number of connections have been identified be-tween them. At the same time, they have found applications in a wide ar-ray of disciplines, including mechanics, economics, partial differential equa-tions, information theory, approximation theory, signal and image process-ing, game theory, optimal transport theory, probability and statistics, andmachine learning.

The purpose of this book is to present a largely self-contained accountof the main results of convex analysis, monotone operator theory, and thetheory of nonexpansive operators in the context of Hilbert spaces. Authori-tative monographs are already available on each of these topics individually.A novelty of this book, and indeed, its central theme, is the tight interplayamong the key notions of convexity, monotonicity, and nonexpansiveness. Weaim at making the presentation accessible to a broad audience, and to reachout in particular to the applied sciences and engineering communities, wherethese tools have become indispensable. We chose to cast our exposition in theHilbert space setting. This allows us to cover many applications of interestto practitioners in infinite-dimensional spaces and yet to avoid the technicaldifficulties pertaining to general Banach space theory that would exclude alarge portion of our intended audience. We have also made an attempt todraw on recent developments and modern tools to simplify the proofs of keyresults, exploiting for instance heavily the concept of a Fitzpatrick functionin our exposition of monotone operators, the notion of Fejer monotonicity tounify the convergence proofs of several algorithms, and that of a proximityoperator throughout the second half of the book.

The book in organized in 29 chapters. Chapters 1 and 2 provide back-ground material. Chapters 3 to 7 cover set convexity and nonexpansive op-erators. Various aspects of the theory of convex functions are discussed inChapters 8 to 19. Chapters 20 to 25 are dedicated to monotone operator the-

ix

x Preface

ory. In addition to these basic building blocks, we also address certain themesfrom different angles in several places. Thus, optimization theory is discussedin Chapters 11, 19, 26, and 27. Best approximation problems are discussedin Chapters 3, 19, 27, 28, and 29. Algorithms are also present in variousparts of the book: fixed point and convex feasibility algorithms in Chap-ter 5, proximal-point algorithms in Chapter 23, monotone operator splittingalgorithms in Chapter 25, optimization algorithms in Chapter 27, and bestapproximation algorithms in Chapters 27 and 29. More than 400 exercisesare distributed throughout the book, at the end of each chapter.

Preliminary drafts of this book have been used in courses in our institu-tions and we have benefited from the input of postdoctoral fellows and manystudents. To all of them, many thanks. In particular, HHB thanks LiangjinYao for his helpful comments. We are grateful to Hedy Attouch, Jon Borwein,Stephen Simons, Jon Vanderwerff, Shawn Wang, and Isao Yamada for helpfuldiscussions and pertinent comments. PLC also thanks Oscar Wesler. Finally,we thank the Natural Sciences and Engineering Research Council of Canada,the Canada Research Chair Program, and France’s Agence Nationale de laRecherche for their support.

Kelowna (Canada) Heinz H. BauschkeParis (France) Patrick L. CombettesOctober 2010

Contents

1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Sets in Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.4 Nets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 The Extended Real Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.6 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.7 Topological Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.8 Two–Point Compactification of the Real Line . . . . . . . . . . . . . . 91.9 Continuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.10 Lower Semicontinuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.11 Sequential Topological Notions . . . . . . . . . . . . . . . . . . . . . . . . . . . 151.12 Metric Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2 Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.1 Notation and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.2 Basic Identities and Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . 292.3 Linear Operators and Functionals . . . . . . . . . . . . . . . . . . . . . . . . 312.4 Strong and Weak Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332.5 Weak Convergence of Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . 362.6 Differentiability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3 Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.1 Definition and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.2 Best Approximation Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . 443.3 Topological Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523.4 Separation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

xi

xii Contents

4 Convexity and Nonexpansiveness . . . . . . . . . . . . . . . . . . . . . . . . . 594.1 Nonexpansive Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.2 Projectors onto Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614.3 Fixed Points of Nonexpansive Operators . . . . . . . . . . . . . . . . . . . 624.4 Averaged Nonexpansive Operators . . . . . . . . . . . . . . . . . . . . . . . . 674.5 Common Fixed Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

5 Fejer Monotonicity and Fixed Point Iterations . . . . . . . . . . . . 755.1 Fejer Monotone Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 755.2 Krasnosel’skiı–Mann Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 785.3 Iterating Compositions of Averaged Operators . . . . . . . . . . . . . 82Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

6 Convex Cones and Generalized Interiors . . . . . . . . . . . . . . . . . . 876.1 Convex Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876.2 Generalized Interiors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 906.3 Polar and Dual Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 966.4 Tangent and Normal Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1006.5 Recession and Barrier Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

7 Support Functions and Polar Sets . . . . . . . . . . . . . . . . . . . . . . . . 1077.1 Support Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1077.2 Support Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1097.3 Polar Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

8 Convex Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1138.1 Definition and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1138.2 Convexity–Preserving Operations . . . . . . . . . . . . . . . . . . . . . . . . . 1168.3 Topological Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

9 Lower Semicontinuous Convex Functions . . . . . . . . . . . . . . . . . 1299.1 Lower Semicontinuous Convex Functions . . . . . . . . . . . . . . . . . . 1299.2 Proper Lower Semicontinuous Convex Functions . . . . . . . . . . . . 1329.3 Affine Minorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1339.4 Construction of Functions in Γ0(H) . . . . . . . . . . . . . . . . . . . . . . . 136Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

10 Convex Functions: Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14310.1 Between Linearity and Convexity . . . . . . . . . . . . . . . . . . . . . . . . . 14310.2 Uniform and Strong Convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . 14410.3 Quasiconvexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

Contents xiii

11 Convex Variational Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15511.1 Infima and Suprema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15511.2 Minimizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15611.3 Uniqueness of Minimizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15711.4 Existence of Minimizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15711.5 Minimizing Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

12 Infimal Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16712.1 Definition and Basic Facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16712.2 Infimal Convolution of Convex Functions . . . . . . . . . . . . . . . . . . 17012.3 Pasch–Hausdorff Envelope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17212.4 Moreau Envelope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17312.5 Infimal Postcomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

13 Conjugation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18113.1 Definition and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18113.2 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18413.3 The Fenchel–Moreau Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 190Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

14 Further Conjugation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19714.1 Moreau’s Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19714.2 Proximal Average . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19914.3 Positively Homogeneous Functions . . . . . . . . . . . . . . . . . . . . . . . . 20114.4 Coercivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20214.5 The Conjugate of the Difference . . . . . . . . . . . . . . . . . . . . . . . . . . 204Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

15 Fenchel–Rockafellar Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20715.1 The Attouch–Brezis Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20715.2 Fenchel Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21115.3 Fenchel–Rockafellar Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21315.4 A Conjugation Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21715.5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

16 Subdifferentiability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22316.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22316.2 Convex Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22716.3 Lower Semicontinuous Convex Functions . . . . . . . . . . . . . . . . . . 22916.4 Subdifferential Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

xiv Contents

17 Differentiability of Convex Functions . . . . . . . . . . . . . . . . . . . . . 24117.1 Directional Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24117.2 Characterizations of Convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . 24417.3 Characterizations of Strict Convexity . . . . . . . . . . . . . . . . . . . . . 24617.4 Directional Derivatives and Subgradients . . . . . . . . . . . . . . . . . . 24717.5 Gateaux and Frechet Differentiability . . . . . . . . . . . . . . . . . . . . . 25117.6 Differentiability and Continuity . . . . . . . . . . . . . . . . . . . . . . . . . . 257Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

18 Further Differentiability Results . . . . . . . . . . . . . . . . . . . . . . . . . . 26118.1 The Ekeland–Lebourg Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 26118.2 The Subdifferential of a Maximum . . . . . . . . . . . . . . . . . . . . . . . . 26418.3 Differentiability of Infimal Convolutions . . . . . . . . . . . . . . . . . . . 26618.4 Differentiability and Strict Convexity . . . . . . . . . . . . . . . . . . . . . 26718.5 Stronger Notions of Differentiability . . . . . . . . . . . . . . . . . . . . . . 26818.6 Differentiability of the Distance to a Set . . . . . . . . . . . . . . . . . . . 271Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273

19 Duality in Convex Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 27519.1 Primal Solutions via Dual Solutions . . . . . . . . . . . . . . . . . . . . . . . 27519.2 Parametric Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27919.3 Minimization under Equality Constraints . . . . . . . . . . . . . . . . . . 28319.4 Minimization under Inequality Constraints . . . . . . . . . . . . . . . . . 285Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291

20 Monotone Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29320.1 Monotone Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29320.2 Maximally Monotone Operators . . . . . . . . . . . . . . . . . . . . . . . . . . 29720.3 Bivariate Functions and Maximal Monotonicity . . . . . . . . . . . . 30220.4 The Fitzpatrick Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308

21 Finer Properties of Monotone Operators . . . . . . . . . . . . . . . . . 31121.1 Minty’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31121.2 The Debrunner–Flor Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 31521.3 Domain and Range . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31621.4 Local Boundedness and Surjectivity . . . . . . . . . . . . . . . . . . . . . . . 31821.5 Kenderov’s Theorem and Frechet Differentiability . . . . . . . . . . 320Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321

22 Stronger Notions of Monotonicity . . . . . . . . . . . . . . . . . . . . . . . . 32322.1 Para, Strict, Uniform, and Strong Monotonicity . . . . . . . . . . . . 32322.2 Cyclic Monotonicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32622.3 Rockafellar’s Cyclic Monotonicity Theorem . . . . . . . . . . . . . . . . 32722.4 Monotone Operators on R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

Contents xv

23 Resolvents of Monotone Operators . . . . . . . . . . . . . . . . . . . . . . . 33323.1 Definition and Basic Identities . . . . . . . . . . . . . . . . . . . . . . . . . . . 33323.2 Monotonicity and Firm Nonexpansiveness . . . . . . . . . . . . . . . . . 33523.3 Resolvent Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33723.4 Zeros of Monotone Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34423.5 Asymptotic Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349

24 Sums of Monotone Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35124.1 Maximal Monotonicity of a Sum. . . . . . . . . . . . . . . . . . . . . . . . . . 35124.2 3∗ Monotone Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35424.3 The Brezis–Haraux Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35724.4 Parallel Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361

25 Zeros of Sums of Monotone Operators . . . . . . . . . . . . . . . . . . . . 36325.1 Characterizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36325.2 Douglas–Rachford Splitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36625.3 Forward–Backward Splitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37025.4 Tseng’s Splitting Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37225.5 Variational Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378

26 Fermat’s Rule in Convex Optimization . . . . . . . . . . . . . . . . . . . 38126.1 General Characterizations of Minimizers . . . . . . . . . . . . . . . . . . . 38126.2 Abstract Constrained Minimization Problems . . . . . . . . . . . . . . 38326.3 Affine Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38626.4 Cone Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38726.5 Convex Inequality Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . 38926.6 Regularization of Minimization Problems . . . . . . . . . . . . . . . . . . 393Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395

27 Proximal Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39927.1 The Proximal-Point Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 39927.2 Douglas–Rachford Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40027.3 Forward–Backward Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40527.4 Tseng’s Splitting Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40727.5 A Primal–Dual Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411

28 Projection Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41528.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41528.2 Projections onto Affine Subspaces . . . . . . . . . . . . . . . . . . . . . . . . 41728.3 Projections onto Special Polyhedra . . . . . . . . . . . . . . . . . . . . . . . 41928.4 Projections Involving Convex Cones . . . . . . . . . . . . . . . . . . . . . . 42528.5 Projections onto Epigraphs and Lower Level Sets . . . . . . . . . . . 427

xvi Contents

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429

29 Best Approximation Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 43129.1 Dykstra’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43129.2 Haugazeau’s Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440

Bibliographical Pointers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441

Symbols and Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449

Chapter 5

Fejer Monotonicity and Fixed PointIterations

A sequence is Fejer monotone with respect to a set C if each point in thesequence is not strictly farther from any point in C than its predecessor. Suchsequences possess very attractive properties that greatly simplify the analy-sis of their asymptotic behavior. In this chapter, we provide the basic theoryfor Fejer monotone sequences and apply it to obtain in a systematic fash-ion convergence results for various classical iterations involving nonexpansiveoperators.

5.1 Fejer Monotone Sequences

The following notion is central in the study of various iterative methods, inparticular in connection with the construction of fixed points of nonexpansiveoperators.

Definition 5.1 Let C be a nonempty subset of H and let (xn)n∈N be asequence in H. Then (xn)n∈N is Fejer monotone with respect to C if

(∀x ∈ C)(∀n ∈ N) ‖xn+1 − x‖ ≤ ‖xn − x‖. (5.1)

Example 5.2 Let (xn)n∈N be a bounded sequence in R that is increasing(respectively decreasing). Then (xn)n∈N is Fejer monotone with respect to[sup{xn}n∈N,+∞[ (respectively ]−∞, inf{xn}n∈N]).

Example 5.3 Let D be a nonempty subset of H, let T : D → D be aquasinonexpansive—in particular, nonexpansive—operator such that FixT 6=∅, and let x0 ∈ D. Set (∀n ∈ N) xn+1 = Txn. Then (xn)n∈N is Fejer mono-tone with respect to FixT .

We start with some basic properties.

75

76 5 Fejer Monotonicity and Fixed Point Iterations

Proposition 5.4 Let (xn)n∈N be a sequence in H and let C be a nonemptysubset of H. Suppose that (xn)n∈N is Fejer monotone with respect to C. Thenthe following hold:

(i) (xn)n∈N is bounded.(ii) For every x ∈ C, (‖xn − x‖)n∈N converges.(iii) (dC(xn))n∈N is decreasing and converges.

Proof. (i): Let x ∈ C. Then (5.1) implies that (xn)n∈N lies in B(x; ‖x0 − x‖).(ii): Clear from (5.1).(iii): Taking the infimum in (5.1) over x ∈ C yields (∀n ∈ N) dC(xn+1) ≤

dC(xn). ⊓⊔

The next result concerns weak convergence.

Theorem 5.5 Let (xn)n∈N be a sequence in H and let C be a nonempty sub-set of H. Suppose that (xn)n∈N is Fejer monotone with respect to C and thatevery weak sequential cluster point of (xn)n∈N belongs to C. Then (xn)n∈N

converges weakly to a point in C.

Proof. The result follows from Proposition 5.4(ii) and Lemma 2.39. ⊓⊔

Example 5.6 Suppose that H is infinite-dimensional and let (xn)n∈N be anorthonormal sequence in H. Then (xn)n∈N is Fejer monotone with respect to{0}. As seen in Example 2.25, xn ⇀ 0 but xn 6→ 0.

While a Fejer monotone sequence with respect to a closed convex set Cmay not converge strongly, its “shadow” on C always does.

Proposition 5.7 Let (xn)n∈N be a sequence in H and let C be a nonemptyclosed convex subset of H. Suppose that (xn)n∈N is Fejer monotone with re-spect to C. Then the shadow sequence (PCxn)n∈N converges strongly to apoint in C.

Proof. It follows from (5.1) and (3.6) that, for every m and n in N,

‖PCxn − PCxn+m‖2 = ‖PCxn − xn+m‖2 + ‖xn+m − PCxn+m‖2

+ 2 〈PCxn − xn+m | xn+m − PCxn+m〉

≤ ‖PCxn − xn‖2 + d2C(xn+m)

+ 2 〈PCxn − PCxn+m | xn+m − PCxn+m〉

+ 2 〈PCxn+m − xn+m | xn+m − PCxn+m〉

≤ d2C(xn)− d2C(xn+m). (5.2)

Consequently, since (dC(xn))n∈N was seen in Proposition 5.4(iii) to converge,(PCxn)n∈N is a Cauchy sequence in the complete set C. ⊓⊔

Corollary 5.8 Let (xn)n∈N be a sequence in H, let C be a nonempty closedconvex subset of H, and let x ∈ C. Suppose that (xn)n∈N is Fejer monotonewith respect to C and that xn ⇀x. Then PCxn → x.

5.1 Fejer Monotone Sequences 77

Proof. By Proposition 5.7, (PCxn)n∈N converges strongly to some point y ∈C. Hence, since x − PCxn → x − y and xn − PCxn ⇀ x − y, it followsfrom Theorem 3.14 and Lemma 2.41(iii) that 0 ≥ 〈x− PCxn | xn − PCxn〉 →‖x− y‖2. Thus, x = y. ⊓⊔

For sequences that are Fejer monotone with respect to closed affine sub-spaces, Proposition 5.7 can be strengthened.

Proposition 5.9 Let (xn)n∈N be a sequence in H and let C be a closed affinesubspace of H. Suppose that (xn)n∈N is Fejer monotone with respect to C.Then the following hold:

(i) (∀n ∈ N) PCxn = PCx0.(ii) Suppose that every weak sequential cluster point of (xn)n∈N belongs to

C. Then xn ⇀PCx0.

Proof. (i): Fix n ∈ N, α ∈ R, and set yα = αPCx0 + (1 − α)PCxn. Since Cis an affine subspace, yα ∈ C, and it therefore follows from Corollary 3.20(i)and (5.1) that

α2‖PCxn − PCx0‖2 = ‖PCxn − yα‖

2

≤ ‖xn − PCxn‖2 + ‖PCxn − yα‖

2

= ‖xn − yα‖2

≤ ‖x0 − yα‖2

= ‖x0 − PCx0‖2 + ‖PCx0 − yα‖

2

= d2C(x0) + (1− α)2‖PCxn − PCx0‖2. (5.3)

Consequently, (2α− 1)‖PCxn − PCx0‖2 ≤ d2C(x0) and, letting α → +∞, we

conclude that PCxn = PCx0.(ii): Combine Theorem 5.5, Corollary 5.8, and (i). ⊓⊔

We now turn our attention to strong convergence properties.

Proposition 5.10 Let (xn)n∈N be a sequence in H and let C be a subset ofH such that intC 6= ∅. Suppose that (xn)n∈N is Fejer monotone with respectto C. Then (xn)n∈N converges strongly to a point in H.

Proof. Take x ∈ intC and ρ ∈ R++ such that B(x; ρ) ⊂ C. Define a sequence(zn)n∈N in B(x; ρ) by

(∀n ∈ N) zn =

x, if xn+1 = xn;

x− ρxn+1 − xn

‖xn+1 − xn‖, otherwise.

(5.4)

Then (5.1) yields (∀n ∈ N) ‖xn+1 − zn‖2 ≤ ‖xn − zn‖

2 and, after expanding,we obtain

78 5 Fejer Monotonicity and Fixed Point Iterations

(∀n ∈ N) ‖xn+1 − x‖2 ≤ ‖xn − x‖2 − 2ρ‖xn+1 − xn‖. (5.5)

Thus,∑

n∈N‖xn+1−xn‖ ≤ ‖x0−x‖2/(2ρ) and (xn)n∈N is therefore a Cauchy

sequence. ⊓⊔

Theorem 5.11 Let (xn)n∈N be a sequence in H and let C be a nonemptyclosed convex subset of H. Suppose that (xn)n∈N is Fejer monotone with re-spect to C. Then the following are equivalent:

(i) (xn)n∈N converges strongly to a point in C.(ii) (xn)n∈N possesses a strong sequential cluster point in C.(iii) lim dC(xn) = 0.

Proof. (i)⇒(ii): Clear.(ii)⇒(iii): Suppose that xkn

→ x ∈ C. Then dC(xkn) ≤ ‖xkn

− x‖ → 0.(iii)⇒(i): Proposition 5.4(iii) implies that dC(xn) → 0. Hence, xn −

PCxn → 0 and (i) follows from Proposition 5.7. ⊓⊔

We conclude this section with a linear convergence result.

Theorem 5.12 Let (xn)n∈N be a sequence in H and let C be a nonemptyclosed convex subset of H. Suppose that (xn)n∈N is Fejer monotone with re-spect to C and that for some κ ∈ [0, 1[,

(∀n ∈ N) dC(xn+1) ≤ κdC(xn). (5.6)

Then (xn)n∈N converges linearly to a point x ∈ C; more precisely,

(∀n ∈ N) ‖xn − x‖ ≤ 2κndC(x0). (5.7)

Proof. Theorem 5.11 and (5.6) imply that (xn)n∈N converges strongly to somepoint x ∈ C. On the other hand, (5.1) yields

(∀n ∈ N)(∀m ∈ N) ‖xn − xn+m‖ ≤ ‖xn − PCxn‖+ ‖xn+m − PCxn‖

≤ 2dC(xn). (5.8)

Letting m → +∞ in (5.8), we conclude that ‖xn − x‖ ≤ 2dC(xn). ⊓⊔

5.2 Krasnosel’skiı–Mann Iteration

Given a nonexpansive operator T , the sequence generated by the Banach–Picard iteration xn+1 = Txn of (1.66) may fail to produce a fixed point ofT . A simple illustration of this situation is T = −Id and x0 6= 0. In this case,however, it is clear that the asymptotic regularity property xn − Txn → 0does not hold. As we shall now see, this property is critical.

5.2 Krasnosel’skiı–Mann Iteration 79

Theorem 5.13 Let D be a nonempty closed convex subset of H, let T : D →D be a nonexpansive operator such that FixT 6= ∅, and let x0 ∈ D. Set

(∀n ∈ N) xn+1 = Txn (5.9)

and suppose that xn − Txn → 0. Then the following hold:

(i) (xn)n∈N converges weakly to a point in FixT .(ii) Suppose that D = −D and that T is odd: (∀x ∈ D) T (−x) = −Tx.

Then (xn)n∈N converges strongly to a point in FixT .

Proof. From Example 5.3, (xn)n∈N is Fejer monotone with respect to FixT .(i): Let x be a weak sequential cluster point of (xn)n∈N, say xkn

⇀ x.Since Txkn

− xkn→ 0, Corollary 4.18 asserts that x ∈ FixT . Appealing to

Theorem 5.5, the assertion is proved.(ii): Since D = −D is convex, 0 ∈ D and, since T is odd, 0 ∈ FixT .

Therefore, by Fejer monotonicity, (∀n ∈ N) ‖xn+1‖ ≤ ‖xn‖. Thus, thereexists ℓ ∈ R+ such that ‖xn‖ ↓ ℓ. Now let m ∈ N. Then, for every n ∈ N,

‖xn+1+m + xn+1‖ = ‖Txn+m − T (−xn)‖ ≤ ‖xn+m + xn‖, (5.10)

and, by the parallelogram identity,

‖xn+m + xn‖2 = 2

(

‖xn+m‖2 + ‖xm‖2)

− ‖xn+m − xn‖2. (5.11)

However, since Txn − xn → 0, we have limn ‖xn+m − xn‖ = 0. Therefore,since ‖xn‖ ↓ ℓ, (5.10) and (5.11) yield ‖xn+m+xn‖ ↓ 2ℓ as n → +∞. In turn,we derive from (5.11) that ‖xn+m − xn‖

2 ≤ 2(‖xn+m‖2 + ‖xm‖2)− 4ℓ2 → 0as m,n → +∞. Thus, (xn)n∈N is a Cauchy sequence and xn → x for somex ∈ D. Since xn+1 → x and xn+1 = Txn → Tx, we have x ∈ FixT . ⊓⊔

We now turn our attention to an alternative iterative method, known asthe Krasnosel’skiı–Mann algorithm.

Theorem 5.14 (Krasnosel’skiı–Mann algorithm) Let D be a nonemptyclosed convex subset of H, let T : D → D be a nonexpansive operator such thatFixT 6= ∅, let (λn)n∈N be a sequence in [0, 1] such that

∑

n∈Nλn(1 − λn) =

+∞, and let x0 ∈ D. Set

(∀n ∈ N) xn+1 = xn + λn

(

Txn − xn

)

. (5.12)

Then the following hold:

(i) (xn)n∈N is Fejer monotone with respect to FixT .(ii) (Txn − xn)n∈N converges strongly to 0.(iii) (xn)n∈N converges weakly to a point in FixT .

Proof. Since x0 ∈ D and D is convex, (5.12) produces a well-defined sequencein D.

80 5 Fejer Monotonicity and Fixed Point Iterations

(i): It follows from Corollary 2.14 and the nonexpansiveness of T that, forevery y ∈ FixT and n ∈ N,

‖xn+1 − y‖2 = ‖(1− λn)(xn − y) + λn(Txn − y)‖2

= (1− λn)‖xn − y‖2 + λn‖Txn − Ty‖2

− λn(1− λn)‖Txn − xn‖2

≤ ‖xn − y‖2 − λn(1 − λn)‖Txn − xn‖2. (5.13)

Hence, (xn)n∈N is Fejer monotone with respect to FixT .(ii): We derive from (5.13) that

∑

n∈Nλn(1−λn)‖Txn−xn‖

2 ≤ ‖x0−y‖2.Since

∑

n∈Nλn(1 − λn) = +∞, we have lim ‖Txn − xn‖ = 0. However, for

every n ∈ N,

‖Txn+1 − xn+1‖ = ‖Txn+1 − Txn + (1− λn)(Txn − xn)‖

≤ ‖xn+1 − xn‖+ (1− λn)‖Txn − xn‖

= ‖Txn − xn‖. (5.14)

Consequently, (‖Txn − xn‖)n∈N converges and we must have Txn − xn → 0.(iii): Let x be a weak sequential cluster point of (xn)n∈N, say xkn

⇀ x.Then it follows from Corollary 4.18 that x ∈ FixT . In view of Theorem 5.5,the proof is complete. ⊓⊔

Proposition 5.15 Let α ∈ ]0, 1[, let T : H → H be an α-averaged oper-ator such that FixT 6= ∅, let (λn)n∈N be a sequence in [0, 1/α] such that∑

n∈Nλn(1− αλn) = +∞, and let x0 ∈ H. Set

(∀n ∈ N) xn+1 = xn + λn

(

Txn − xn

)

. (5.15)

Then the following hold:

(i) (xn)n∈N is Fejer monotone with respect to FixT .(ii) (Txn − xn)n∈N converges strongly to 0.(iii) (xn)n∈N converges weakly to a point in FixT .

Proof. Set R = (1− 1/α)Id+ (1/α)T and (∀n ∈ N) µn = αλn. Then FixR =FixT and R is nonexpansive by Proposition 4.25. In addition, we rewrite(5.15) as (∀n ∈ N) xn+1 = xn + µn

(

Rxn − xn

)

. Since (µn)n∈N lies in [0, 1]and

∑

n∈Nµn(1− µn) = +∞, the results follow from Theorem 5.14. ⊓⊔

Corollary 5.16 Let T : H → H be a firmly nonexpansive operator such thatFixT 6= ∅, let (λn)n∈N be a sequence in [0, 2] such that

∑

n∈Nλn(2 − λn) =

+∞, and let x0 ∈ H. Set (∀n ∈ N) xn+1 = xn + λn

(

Txn − xn

)

. Then thefollowing hold:

(i) (xn)n∈N is Fejer monotone with respect to FixT .(ii) (Txn − xn)n∈N converges strongly to 0.

5.2 Krasnosel’skiı–Mann Iteration 81

(iii) (xn)n∈N converges weakly to a point in FixT .

Proof. In view of Remark 4.24(iii), apply Proposition 5.15 with α = 1/2. ⊓⊔

Example 5.17 Let T : H → H be a firmly nonexpansive operator such thatFixT 6= ∅, let x0 ∈ H, and set (∀n ∈ N) xn+1 = Txn. Then (xn)n∈N

converges weakly to a point in FixT .

The following type of iterative method involves a mix of compositions andconvex combinations of nonexpansive operators.

Corollary 5.18 Let (Ti)i∈I be a finite family of nonexpansive operatorsfrom H to H such that

⋂

i∈I FixTi 6= ∅, and let (αi)i∈I be real num-bers in ]0, 1[ such that, for every i ∈ I, Ti is αi-averaged. Let p be astrictly positive integer, for every k ∈ {1, . . . , p}, let mk be a strictlypositive integer and ωk be a strictly positive real number, and supposethat i :

{

(k, l)∣

∣ k ∈ {1, . . . , p}, l ∈ {1, . . . ,mk}}

→ I is surjective and that∑p

k=1 ωk = 1. For every k ∈ {1, . . . , p}, set Ik = {i(k, 1), . . . , i(k,mk)}, andset

α = max1≤k≤p

ρk, where (∀k ∈ {1, . . . , p}) ρk =mk

mk − 1 +1

maxi∈Ik

αi

, (5.16)

and let (λn)n∈N be a sequence in [0, 1/α] such that∑

n∈Nλn(1−αλn) = +∞.

Furthermore, let x0 ∈ H and set

(∀n ∈ N) xn+1 = xn + λn

(

p∑

k=1

ωkTi(k,1) · · ·Ti(k,mk)xn − xn

)

. (5.17)

Then (xn)n∈N converges weakly to a point in⋂

i∈I FixTi.

Proof. Set T =∑p

k=1 ωkRk, where (∀k ∈ {1, . . . , p}) Rk = Ti(k,1) · · ·Ti(k,mk).Then (5.17) reduces to (5.15) and, in view of Proposition 5.15, it sufficesto show that T is α-averaged and that FixT =

⋂

i∈I FixTi. For every k ∈{1, . . . , p}, it follows from Proposition 4.32 and (5.16) that Rk is ρk-averagedand, from Corollary 4.37 that FixRk =

⋂

i∈IkFixTi. In turn, we derive from

Proposition 4.30 and (5.16) that T is α-averaged and, from Proposition 4.34,that FixT =

⋂pk=1 FixRk =

⋂pk=1

⋂

i∈IkFixTi =

⋂

i∈I FixTi. ⊓⊔

Remark 5.19 It follows from Remark 4.24(iii) that Corollary 5.18 is appli-cable to firmly nonexpansive operators and, a fortiori, to projection operatorsby Proposition 4.8.

Corollary 5.18 provides an algorithm to solve a convex feasibility problem,i.e., to find a point in the intersection of a family of closed convex sets. Hereare two more examples.

82 5 Fejer Monotonicity and Fixed Point Iterations

Example 5.20 (string-averaged relaxed projections) Let (Ci)i∈I bea finite family of closed convex sets such that C =

⋂

i∈I Ci 6= ∅. Forevery i ∈ I, let βi ∈ ]0, 2[ and set Ti = (1 − βi)Id + βiPCi

. Let p bea strictly positive integer; for every k ∈ {1, . . . , p}, let mk be a strictlypositive integer and ωk be a strictly positive real number, and supposethat i :

{

(k, l)∣

∣ k ∈ {1, . . . , p}, l ∈ {1, . . . ,mk}}

→ I is surjective and that∑p

k=1 ωk = 1. Furthermore, let x0 ∈ H and set

(∀n ∈ N) xn+1 =

p∑

k=1

ωkTi(k,1) · · ·Ti(k,mk)xn. (5.18)

Then (xn)n∈N converges weakly to a point in C.

Proof. For every i ∈ I, set αi = βi/2 ∈ ]0, 1[. Since, for every i ∈ I, Proposi-tion 4.8 asserts that PCi

is firmly nonexpansive, Corollary 4.29 implies thatTi is αi-averaged. Borrowing notation from Corollary 5.18, we note that forevery k ∈ {1, . . . , p}, maxi∈Ik αi ∈ ]0, 1[, which implies that ρk ∈ ]0, 1[ andthus that α ∈ ]0, 1[. Altogether, the result follows from Corollary 5.18 withλn ≡ 1. ⊓⊔

Example 5.21 (parallel projection algorithm) Let (Ci)i∈I be a finitefamily of closed convex subsets of H such that C =

⋂

i∈I Ci 6= ∅, let (λn)n∈N

be a sequence in [0, 2] such that∑

n∈Nλn(2 − λn) = +∞, let (ωi)i∈I be

strictly positive real numbers such that∑

i∈I ωi = 1, and let x0 ∈ H. Set

(∀n ∈ N) xn+1 = xn + λn

(

∑

i∈I

ωiPixn − xn

)

, (5.19)

where, for every i ∈ I, Pi denotes the projector onto Ci. Then (xn)n∈N

converges weakly to a point in C.

Proof. This is an application of Corollary 5.16(iii) with T =∑

i∈I ωiPi. In-deed, since the operators (Pi)i∈I are firmly nonexpansive by Proposition 4.8,their convex combination T is also firmly nonexpansive by Example 4.31.Moreover, Proposition 4.34 asserts that FixT =

⋂

i∈I FixPi =⋂

i∈I Ci = C.Alternatively, apply Corollary 5.18. ⊓⊔

5.3 Iterating Compositions of Averaged Operators

Our first result concerns the asymptotic behavior of iterates of a compositionof averaged nonexpansive operators with possibly no common fixed point.

Theorem 5.22 Let D be a nonempty weakly sequentially closed (e.g., closedand convex) subset of H, let m be a strictly positive integer, set I =

5.3 Iterating Compositions of Averaged Operators 83

{1, . . . ,m}, let (Ti)i∈I be a family of nonexpansive operators from D to Dsuch that Fix(T1 · · ·Tm) 6= ∅, and let (αi)i∈I be real numbers in ]0, 1[ suchthat, for every i ∈ I, Ti is αi-averaged. Let x0 ∈ D and set

(∀n ∈ N) xn+1 = T1 · · ·Tmxn. (5.20)

Then xn − T1 · · ·Tmxn → 0, and there exist points y1 ∈ FixT1 · · ·Tm, y2 ∈FixT2 · · ·TmT1, . . . , ym ∈ FixTmT1 · · ·Tm−1 such that

xn ⇀ y1 = T1y2, (5.21)

Tmxn ⇀ ym = Tmy1, (5.22)

Tm−1Tmxn ⇀ ym−1 = Tm−1ym, (5.23)

...

T3 · · ·Tmxn ⇀ y3 = T3y4, (5.24)

T2 · · ·Tmxn ⇀ y2 = T2y3. (5.25)

Proof. Set T = T1 · · ·Tm and (∀i ∈ I) βi = (1− αi)/αi. Now take y ∈ FixT .The equivalence (i)⇔(iii) in Proposition 4.25 yields

‖xn+1 − y‖2 = ‖Txn − Ty‖2

≤ ‖T2 · · ·Tmxn − T2 · · ·Tmy‖2

− β1‖(Id− T1)T2 · · ·Tmxn − (Id− T1)T2 · · ·Tmy‖2

≤ ‖xn − y‖2 − βm‖(Id− Tm)xn − (Id− Tm)y‖2

− βm−1‖(Id− Tm−1)Tmxn − (Id− Tm−1)Tmy‖2 − · · ·

− β2‖(Id− T2)T3 · · ·Tmxn − (Id− T2)T3 · · ·Tmy‖2

− β1‖(Id− T1)T2 · · ·Tmxn − (T2 · · ·Tmy − y)‖2. (5.26)

Therefore, (xn)n∈N is Fejer monotone with respect to FixT and

(Id− Tm)xn − (Id− Tm)y → 0, (5.27)

(Id− Tm−1)Tmxn − (Id− Tm−1)Tmy → 0, (5.28)

...

(Id− T2)T3 · · ·Tmxn − (Id− T2)T3 · · ·Tmy → 0, (5.29)

(Id− T1)T2 · · ·Tmxn − (T2 · · ·Tmy − y) → 0. (5.30)

Upon adding (5.27)–(5.30), we obtain xn − Txn → 0. Hence, since T isnonexpansive as a composition of nonexpansive operators, it follows fromTheorem 5.13(i) that (xn)n∈N converges weakly to some point y1 ∈ FixT ,which provides (5.21). On the other hand, (5.27) yields Tmxn − xn →Tmy1 − y1. So altogether Tmxn ⇀ Tmy1 = ym, and we obtain (5.22). Inturn, since (5.28) asserts that Tm−1Tmxn − Tmxn → Tm−1ym − ym, we ob-

84 5 Fejer Monotonicity and Fixed Point Iterations

tain Tm−1Tmxn ⇀ Tm−1ym = ym−1, hence (5.23). Continuing this process,we arrive at (5.25). ⊓⊔

As noted in Remark 5.19, results on averaged nonexpansive operators ap-ply in particular to firmly nonexpansive operators and projectors onto convexsets. Thus, by specializing Theorem 5.22 to convex projectors, we obtain theiterative method described in the next corollary, which is known as the POCS(Projections Onto Convex Sets) algorithm in the signal recovery literature.

Corollary 5.23 (POCS algorithm) Let m be a strictly positive integer,set I = {1, . . . ,m}, let (Ci)i∈I be a family of nonempty closed convex subsetsof H, let (Pi)i∈I denote their respective projectors, and let x0 ∈ H. Supposethat Fix(P1 · · ·Pm) 6= ∅ and set

(∀n ∈ N) xn+1 = P1 · · ·Pmxn. (5.31)

Then there exists (y1, . . . , ym) ∈ C1 × · · · × Cm such that xn ⇀ y1 = P1y2,Pmxn⇀ym = Pmy1, Pm−1Pmxn⇀ym−1 = Pm−1ym, . . ., P3 · · ·Pmxn⇀y3 =P3y4, and P2 · · ·Pmxn ⇀ y2 = P2y3.

Proof. This follows from Proposition 4.8 and Theorem 5.22. ⊓⊔

Remark 5.24 In Corollary 5.23, suppose that, for some j ∈ I, Cj isbounded. Then Fix(P1 · · ·Pm) 6= ∅. Indeed, consider the circular compositionof the m projectors given by T = Pj · · ·PmP1 · · ·Pj−1. Then Proposition 4.8asserts that T is a nonexpansive operator that maps the nonempty boundedclosed convex set Cj to itself. Hence, it follows from Theorem 4.19 that thereexists a point x ∈ Cj such that Tx = x.

The next corollary describes a periodic projection method to solve a convexfeasibility problem.

Corollary 5.25 Let m be a strictly positive integer, set I = {1, . . . ,m}, let(Ci)i∈I be a family of closed convex subsets of H such that C =

⋂

i∈I Ci 6= ∅,let (Pi)i∈I denote their respective projectors, and let x0 ∈ H. Set (∀n ∈ N)xn+1 = P1 · · ·Pmxn. Then (xn)n∈N converges weakly to a point in C.

Proof. Using Corollary 5.23, Proposition 4.8, and Corollary 4.37, we obtainxn ⇀ y1 ∈ Fix(P1 · · ·Pm) =

⋂

i∈I FixPi = C. Alternatively, this is a specialcase of Example 5.20. ⊓⊔

Remark 5.26 If, in Corollary 5.25, all the sets are closed affine subspaces,so is C and we derive from Proposition 5.9(i) that xn⇀PCx0. Corollary 5.28is classical, and it states that the convergence is actually strong in this case.In striking contrast, the example constructed in [146] provides a closed hy-perplane and a closed convex cone in ℓ2(N) for which alternating projectionsconverge weakly but not strongly.

Exercises 85

The next result will help us obtain a sharper form of Corollary 5.25 forclosed affine subspaces.

Proposition 5.27 Let T ∈ B(H) be nonexpansive and let x0 ∈ H. Set V =FixT and (∀n ∈ N) xn+1 = Txn. Then xn → PV x0 ⇔ xn − xn+1 → 0.

Proof. If xn → PV x0, then xn − xn+1 → PV x0 − PV x0 = 0. Conversely,suppose that xn − xn+1 → 0. We derive from Theorem 5.13(ii) that thereexists v ∈ V such that xn → v. In turn, Proposition 5.9(i) yields v = PV x0.

⊓⊔

Corollary 5.28 (von Neumann–Halperin) Let m be a strictly positiveinteger, set I = {1, . . . ,m}, let (Ci)i∈I be a family of closed affine subspaces ofH such that C =

⋂

i∈I Ci 6= ∅, let (Pi)i∈I denote their respective projectors,let x0 ∈ H, and set

(∀n ∈ N) xn+1 = P1 · · ·Pmxn. (5.32)

Then xn → PCx0.

Proof. Set T = P1 · · ·Pm. Then T is nonexpansive, and FixT = C by Corol-lary 4.37.

We first assume that each set Ci is a linear subspace. Then T is odd,and Theorem 5.22 implies that xn − Txn → 0. Thus, by Proposition 5.27,xn → PCx0.

We now turn our attention to the general affine case. Since C 6= ∅, thereexists y ∈ C such that for every i ∈ I, Ci = y+Vi, i.e., Vi is the closed linearsubspace parallel to Ci, and C = y+V , where V =

⋂

i∈I Vi. Proposition 3.17implies that, for every x ∈ H, PCx = Py+V x = y + PV (x − y) and (∀i ∈ I)Pix = Py+Vi

x = y + PVi(x− y). Using these identities repeatedly, we obtain

(∀n ∈ N) xn+1 − y = (PV1· · ·PVm

)(xn − y). (5.33)

Invoking the already verified linear case, we get xn − y → PV (x0 − y) andconclude that xn → y + PV (x0 − y) = PCx0. ⊓⊔

Exercises

Exercise 5.1 Find a nonexpansive operator T : H → H that is not firmlynonexpansive and such that, for every x0 ∈ H, the sequence (T nx0)n∈N con-verges weakly but not strongly to a fixed point of T .

Exercise 5.2 Construct a non-Cauchy sequence (xn)n∈N in R that is asymp-totically regular, i.e., xn − xn+1 → 0.

86 5 Fejer Monotonicity and Fixed Point Iterations

Exercise 5.3 Find an alternative proof of Theorem 5.5 based on Corol-lary 5.8 in the case when C is closed and convex.

Exercise 5.4 Let C be a nonempty subset of H and let (xn)n∈N be a se-quence in H that is Fejer monotone with respect to C. Show that (xn)n∈N isFejer monotone with respect to convC.

Exercise 5.5 Let T : H → H be a nonexpansive operator such that FixT 6=∅, and let (xn)n∈N be a sequence in H such that

(i) for every x ∈ FixT , (‖xn − x‖)n∈N converges;(ii) xn − Txn → 0.

Show that (xn)n∈N converges weakly to a point in FixT .

Exercise 5.6 Find a nonexpansive operator T : H → H that is not firmlynonexpansive and such that, for every x0 ∈ H, the sequence (T nx0)n∈N con-verges weakly but not strongly to a fixed point of T .

Exercise 5.7 Let m be a strictly positive integer, set I = {1, . . . ,m}, let(Ci)i∈I be a family of closed convex subsets of H such that C =

⋂

i∈I Ci 6= ∅,and let (Pi)i∈I be their respective projectors. Derive parts (ii) and (iii) from(i) and Theorem 5.5, and also from Corollary 5.18.

(i) Let i ∈ I, let x ∈ Ci, and let y ∈ H. Show that ‖Piy−x‖2 ≤ ‖y−x‖2−‖Piy − y‖2.

(ii) Set x0 ∈ H and

(∀n ∈ N) xn+1 =1

m

(

P1xn + P1P2xn + · · ·+ P1 · · ·Pmxn

)

. (5.34)

(a) Let x ∈ C and n ∈ N. Show that ‖xn+1 − x‖2 ≤ ‖xn − x‖2 −(1/m)

∑

i∈I ‖Pixn − x‖2.(b) Let x be a weak sequential cluster point of (xn)n∈N. Show that

x ∈ C.(c) Show that (xn)n∈N converges weakly to a point in C.

(iii) Set x0 ∈ H and

(∀n ∈ N) xn+1 =1

m− 1

(

P1P2xn+P2P3xn+· · ·+Pm−1Pmxn

)

. (5.35)

(a) Let x ∈ C and n ∈ N. Show that ‖xn+1 − x‖2 ≤ ‖xn − x‖2 −∑m−1

i=1 (‖Pi+1xn − xn‖2 + ‖PiPi+1xn − Pi+1xn‖

2)/(m− 1).(b) Let x be a weak sequential cluster point of (xn)n∈N. Show that

x ∈ C.(c) Show that (xn)n∈N converges weakly to a point in C.

Index

3∗ monotone, 354–356

accretive operator, 293acute cone, 105addition of functions, 6addition of set-valued operators, 2adjoint of a linear operator, 31affine constraint, 386affine hull, 1, 93affine minorant, 133, 134, 168, 184,

191, 192, 207, 223, 224affine operator, 3, 35, 44, 418affine projector, 48, 62affine subspace, 1, 43, 108almost surely, 29alternating minimizations, 162alternating projection method,

406, 432Anderson–Duffin theorem, 361angle bounded operator, 361, 362Apollonius’s identity, 30, 46approximating curve, 64asymptotic center, 117, 159, 195asymptotic regularity, 78at most single-valued, 2Attouch–Brezis condition, 210,

216Attouch–Brezis theorem, 207, 209autoconjugate function, 190, 239,

303averaged nonexpansive operator,

67, 72, 80–82, 294, 298, 440

Baillon–Haddad theorem, 270ball, 43Banach–Alaoglu theorem, 34Banach–Picard theorem, 20Banach–Steinhaus theorem, 31barrier cone, 103, 156base of a topology, 7, 9, 16, 33

base of neighborhoods, 23best approximation, 44, 278, 410best approximation algorithm,

410, 411, 431biconjugate function, 181, 185,

190, 192biconjugation theorem, 190bilinear form, 39, 384, 385Bishop–Phelps theorem, 107Boltzmann–Shannon entropy, 137,

139boundary of a set, 7bounded below, 134, 184Brezis–Haraux theorem, 358Bregman distance, 258Browder–Gohde–Kirk theorem, 64Brøndsted–Rockafellar theorem,

236Bunt–Motzkin theorem, 318Burg’s entropy, 138, 151, 244

Cantor’s theorem, 17Cauchy sequence, 17, 24, 34, 46Cauchy–Schwarz inequality, 29Cayley transform, 349chain, 3chain rule, 40characterization of minimizers,

381Chebyshev center, 249, 250Chebyshev problem, 47Chebyshev set, 44–47, 318closed ball, 16closed convex hull, 43closed range, 220closed set, 7, 8closure of a set, 7, 22cluster point, 7, 8

460

INDEX 461

cocoercive operator, 60, 61, 68, 70,270, 294, 298, 325, 336, 339,355, 370, 372, 377, 379, 438

coercive function, 158–161, 165,202–204, 210

cohypomonotone operator, 337common fixed points, 71compact set, 7, 8, 13complementarity problem, 376complementary slackness, 291, 422complete metric space, 17composition of set-valued opera-

tors, 2concave function, 113cone, 1, 87, 285, 287, 376, 387, 389,

425conical hull, 87conjugate function, 181conjugation, 181, 197, 226, 230constrained minimization prob-

lem, 283, 285, 383continuity, 9continuous affine minorant, 168continuous convex function, 123,

136continuous function, 11continuous linear functional, 31continuous operator, 9, 63convergence of a net, 7convex combination, 44convex cone, 87, 89, 179, 183, 285,

425convex feasibility problem, 81, 84convex function, 113, 155convex hull, 43, 44convex integrand, 118, 138, 193,

238convex on a set, 114, 125convex programming problem, 290convex set, 43convexity with respect to a cone,

285core, 90, 95, 123, 207, 210, 214,

216, 241, 271counting measure, 140

cyclically monotone operator, 326

Debrunner–Flor theorem, 315decreasing function, 5decreasing sequence of convex sets,

48, 417demiclosedness principle, 63dense hyperplane, 123dense set, 7, 33, 123, 232descent direction, 248, 249descent lemma, 270diameter of a set, 16directed set, 3, 4, 22, 27directional derivative, 241, 247discontinuous linear functional,

32, 123, 169discrete entropy, 140distance, 27distance to a set, 16, 20, 24, 32, 34,

44, 49, 98, 167, 170, 173, 177,183, 185, 188, 238, 271, 272

domain of a function, 5, 6, 113domain of a set-valued operator, 2domain of continuity, 11Douglas–Rachford algorithm, 366,

376, 401, 404dual cone, 96dual optimal value, 214dual problem, 212, 214, 275, 279,

408dual solution, 275, 279duality, 211, 213, 275duality gap, 212, 214–216, 221Dykstra’s algorithm, 431, 432

Eberlein–Smulian theorem, 35effective domain of a function, 6Ekeland variational principle, 19Ekeland–Lebourg theorem, 263enlargement of a monotone opera-

tor, 309entropy of a random variable, 139epi-sum, 167epigraph, 5, 6, 12, 15, 113, 119,

133, 168equality constraint, 283

462 INDEX

Euclidean space, 28even function, 186eventually in a set, 4evolution equation, 313exact infimal convolution, 167,

170, 171, 207, 209, 210exact infimal postcomposition, 178exact modulus of convexity, 144–

146existence of minimizers, 157, 159expected value, 29extended real line, 4extension, 297

Fσ set, 24Farkas’s lemma, 99, 106farthest-point operator, 249, 296Fejer monotone, 75, 83, 86, 160,

400Fenchel conjugate, 181Fenchel duality, 211Fenchel–Moreau theorem, 190Fenchel–Rockafellar duality, 213,

275, 282, 408Fenchel–Young inequality, 185,

226Fermat’s rule, 223, 235, 381firmly nonexpansive operator, 59,

61–63, 68, 69, 73, 80, 81, 176,270, 294, 298, 335, 337, 436

first countable space, 23Fitzpatrick function, 304, 311, 351Fitzpatrick function of order n,

330fixed point, 62, 79–81fixed point iterations, 75fixed point set, 20, 62–64, 436forward–backward algorithm, 370,

377, 405, 438, 439forward–backward–forward algo-

rithm, 375Frechet derivative, 38, 39, 257Frechet differentiability, 38, 176,

177, 243, 253, 254, 268–270,320

Frechet gradient, 38Frechet topological space, 23frequently in a set, 4function, 5

Gδ set, 19, 263, 320Gateaux derivative, 37Gateaux differentiability, 37–39,

243, 244, 246, 251, 252, 254,257, 267

gauge, 120, 124, 202generalized inverse, 50, 251, 360,

361, 395, 418generalized sequence, 4global minimizer, 223gradient, 38, 176, 243, 244, 266,

267, 382gradient operator, 38graph, 5graph of a set-valued operator, 2

Holder continuous gradient, 269half-space, 32, 33, 43, 419, 420Hamel basis, 32hard thresholder, 61Haugazeau’s algorithm, 436, 439Hausdorff distance, 25Hausdorff space, 7, 16, 33hemicontinous operator, 298, 325Hessian, 38, 243, 245, 246Hilbert direct sum, 28, 226Hilbert space, 27Huber’s function, 124hyperplane, 32, 34, 48, 123

increasing function, 5increasing sequence of convex sets,

416indicator function, 12, 113, 173,

227inequality constraint, 285, 389infimal convolution, 167, 187, 207,

210, 237, 266, 359infimal postcomposition, 178, 187,

199, 218, 237infimum, 5, 157, 159, 184, 188

INDEX 463

infimum of a function, 6infinite sum, 27initial condition, 295integral function, 118, 138, 193,

238interior of a set, 7, 22, 90, 123inverse of a monotone operator,

295inverse of a set-valued operator, 2,

231inverse strongly monotone opera-

tor, 60

Jensen’s inequality, 135

Karush–Kuhn–Tucker conditions,393

Kenderov theorem, 320kernel of a linear operator, 32Kirszbraun–Valentine theorem,

337Krasnosel’skiı–Mann algorithm,

78, 79

Lagrange multiplier, 284, 287, 291,386–388, 391

Lagrangian, 280, 282Lax–Milgram theorem, 385least element, 3least-squares solution, 50Lebesgue measure, 29Legendre function, 273Legendre transform, 181Legendre–Fenchel transform, 181level set, 5, 6, 12, 15, 132, 158, 203,

383limit inferior, 5limit superior, 5line segment, 1, 43, 54, 132linear convergence, 20, 78, 372,

377, 406, 407linear equations, 50linear functional, 32linear monotone operator, 296–

298, 355

Lipschitz continuous, 20, 31, 59,123, 176, 229, 339

Lipschitz continuous gradient,269–271, 405–407, 439

local minimizer, 156locally bounded operator, 316,

319, 344locally Lipschitz continuous, 20,

122lower bound, 3lower level set, 5, 6, 148, 427lower semicontinuity, 10, 129lower semicontinuous, 10, 12lower semicontinuous convex enve-

lope, 130, 185, 192, 193, 207lower semicontinuous convex func-

tion, 122, 129, 132, 185lower semicontinuous envelope, 14,

23lower semicontinuous function,

129lower semicontinuous infimal con-

volution, 170, 210

marginal function, 13, 120, 152max formula, 248maximal element, 3maximal monotone operator, 297maximal monotonicity and conti-

nuity, 298maximal monotonicity of a sum,

351maximally cyclically monotone op-

erator, 326maximally monotone extension,

316, 337maximally monotone operator,

297, 298, 311, 335, 336, 338,339, 438

maximum of a function, 6measure space, 28, 295metric space, 16metric topology, 16, 33, 34metrizable topology, 16, 23, 34midpoint convex function, 141

464 INDEX

midpoint convex set, 57minimax, 218minimization in a product space,

403minimization problem, 13, 156,

381, 393, 401, 402, 404–406minimizer, 156, 157, 159, 163, 243,

384minimizing sequence, 6, 13, 160,

399minimum of a function, 6, 243Minkowski gauge, 120, 124, 202Minty’s parametrization, 340Minty’s theorem, 311modulus of convexity, 144monotone extension, 297monotone linear operator, 296monotone operator, 244, 293, 311,

351, 363monotone set, 293Moore–Penrose inverse, 50Moreau envelope, 173, 175, 176,

183, 185, 187, 197, 198, 270,271, 276, 277, 334, 339, 342

Moreau’s conical decomposition,98

Moreau’s decomposition, 198Moreau–Rockafellar theorem, 204

negative orthant, 5negative real number, 4neighborhood, 7net, 4, 5, 22, 27, 53, 314nonexpansive operator, 59, 60, 62,

63, 79, 159, 270, 294, 298,336, 348, 439

nonlinear equation, 325norm, 27, 35, 40, 115, 118, 144,

147, 150, 151, 183, 199, 231,252

norm topology, 33normal cone, 101, 227, 230, 238,

272, 304, 334, 354, 383, 389normal equation, 50normal vector, 107

obtuse cone, 105odd operator, 79, 379open ball, 16open set, 7operator splitting algorithm, 366Opial’s condition, 41optimal value, 214order, 3orthogonal complement, 27orthonormal basis, 27, 37, 161,

301, 313, 344orthonormal sequence, 34outer normal, 32

parallel linear subspace, 1parallel projection algorithm, 82parallel splitting algorithm, 369,

404parallel sum of monotone opera-

tors, 359parallelogram identity, 29parametric duality, 279paramonotone operator, 323, 385partial derivative, 259partially ordered set, 3Pasch–Hausdorff envelope, 172,

179periodicity condition, 295perspective function, 119, 184POCS algorithm, 84pointed cone, 88, 105pointwise bounded operator fam-

ily, 31polar cone, 96, 110polar set, 110, 202, 206, 428polarization identity, 29polyhedral cone, 388, 389polyhedral function, 216–218, 381,

383, 388, 389polyhedral set, 216, 383polyhedron, 419positive operator, 60positive orthant, 5, 426positive real number, 4positive semidefinite matrix, 426

INDEX 465

positively homogeneous function,143, 201, 229, 278

positively homogeneous operator,3

power set, 2primal optimal value, 214primal problem, 212, 214, 275, 279,

408primal solution, 275, 279primal–dual algorithm, 408probability simplex, 426probability space, 29, 139product topology, 7projection algorithm, 431, 439projection onto a ball, 47projection onto a convex cone, 97,

98, 425projection onto a half-space, 419projection onto a hyperplane, 48,

419projection onto a hyperslab, 419projection onto a linear subspace,

49projection onto a lower level set,

427projection onto a polar set, 428projection onto a ray, 426projection onto a set, 44projection onto an affine subspace,

48, 77, 417projection onto an epigraph, 133,

427projection operator, 44, 61, 62,

175, 177, 334, 360, 361, 415projection theorem, 46, 238projection-gradient algorithm, 406projector, 44, 61proper function, 6, 132proximal average, 199, 205, 271,

307proximal mapping, 175proximal minimization, 399proximal-gradient algorithm, 405,

439

proximal-point algorithm, 345,399, 438

proximinal set, 44–46proximity operator, 175, 198, 199,

233, 243, 244, 271, 334, 339,342–344, 375, 381, 382, 401,402, 404, 405, 415, 428

pseudocontractive operator, 294pseudononexpansive operator, 294

quadratic function, 251quasiconvex function, 148, 157,

160, 165quasinonexpansive operator, 59,

62, 71, 75quasirelative interior, 91

random variable, 29, 135, 139, 194range of a set-valued operator, 2range of a sum of operators, 357,

358recession cone, 103recession function, 152recovery of primal solutions, 275,

408reflected resolvent, 336, 363, 366regularization, 393regularized minimization problem,

393relative interior, 90, 96, 123, 210,

216, 234resolvent, 333, 335, 336, 366, 370,

373reversal of a function, 186, 236,

342reversal of an operator, 3, 340Riesz–Frechet representation, 31right-shift operator, 330, 356Radstrom’s cancellation, 58

saddle point, 280–282scalar product, 27second Frechet derivative, 38second Gateaux derivative, 38second-order derivative, 245, 246

466 INDEX

selection of a set-valued operator,2

self-conjugacy, 183, 185self-dual cone, 96, 186self-polar cone, 186separable Hilbert space, 27, 194separated sets, 55separation, 55sequential cluster point, 7, 15, 33sequential topological space, 16, 23sequentially closed, 15, 16, 53, 231,

300, 301sequentially compact, 15, 16, 36sequentially continuous, 15, 16sequentially lower semicontinuous,

15, 129set-valued operator, 2shadow sequence, 76sigma-finite measure space, 194Slater condition, 391slope, 168soft thresholder, 61, 199solid cone, 88, 105span of a set, 1splitting algorithm, 375, 401, 402,

404, 405Stampacchia’s theorem, 384, 395standard unit vectors, 28, 89, 92steepest descent direction, 249strict contraction, 64strict epigraph, 180strictly convex function, 114, 144,

161, 267, 324strictly convex on a set, 114strictly convex set, 157strictly decreasing function, 5strictly increasing function, 5strictly monotone operator, 323,

344strictly negative real number, 4strictly nonexpansive operator,

325strictly positive operator, 246strictly positive orthant, 5strictly positive real number, 4

strictly quasiconvex function, 149,157

strictly quasinonexpansive opera-tor, 59, 71

string-averaged relaxed projec-tions, 82

strong convergence, 33, 37strong relative interior, 90, 95, 96,

209, 210, 212, 215, 217, 234,236, 381

strong separation, 55strong topology, 33strongly convex function, 144, 159,

188, 197, 270, 276, 324, 406strongly monotone operator, 323,

325, 336, 344, 372subadditive function, 143subdifferentiable function, 223,

247subdifferential, 223, 294, 304, 312,

324, 326, 354, 359, 381, 383subdifferential of a maximum, 264subgradient, 223sublinear function, 143, 153, 156,

241subnet, 4, 8, 22sum of linear subspaces, 33sum of monotone operators, 351sum rule for subdifferentials, 234summable family, 27supercoercive function, 158, 159,

172, 203, 210, 229support function, 109, 156, 183,

195, 201, 229, 240support point, 107, 109, 164supporting hyperplane, 107, 109supremum, 5, 129, 188supremum of a function, 6surjective monotone operator, 318,

320, 325, 358

tangent cone, 100time-derivative operator, 295, 312,

334Toland–Singer duality, 205

INDEX 467

topological space, 7topology, 7totally ordered set, 3trace of a matrix, 28translation of an operator, 3Tseng’s splitting algorithm, 373,

378, 407Tykhonov regularization, 393

unbounded net, 314uniform boundedness principle, 31uniformly convex function, 144,

147, 324, 394, 399uniformly convex on a set, 144,

147, 324, 407uniformly convex set, 164, 165uniformly monotone on a set, 324uniformly monotone on bounded

sets, 346, 367, 373, 376, 378,408

uniformly monotone operator,323, 325, 344, 354, 358, 367,373, 376, 378, 408

uniformly quasiconvex function,149, 163

upper bound, 3upper semicontinuous function,

11, 124, 281

value function, 279, 289variational inequality, 375–378,

383Volterra integration operator, 308von Neumann’s minimax theorem,

218von Neumann–Halperin theorem,

85

weak closure, 53weak convergence, 33, 36, 79–81weak sequential closure, 53weak topology, 33weakly closed, 33–35, 45, 53weakly compact, 33–35weakly continuous operator, 33,

35, 62, 418

weakly lower semicontinuous, 35,129

weakly lower semicontinuous func-tion, 33

weakly open, 33weakly sequentially closed, 33–35,

53weakly sequentially compact, 33,

35weakly sequentially continuous op-

erator, 343, 426weakly sequentially lower semicon-

tinuous, 129Weierstrass theorem, 13

Yosida approximation, 333, 334,336, 339, 345, 347, 348

zero of a monotone operator, 344,345, 347, 381, 438

zero of a set-valued operator, 2zero of a sum of operators, 363,

366, 369, 375