0976401304.pdf

CONVEX

OPTIMIZATION

&

EUCLIDEAN

DISTANCE

GEOMETRY

DATTORRO

M

Dattorro

CONVEX

OPTIMIZATION

&

EUCLIDEAN

DISTANCE

GEOMETRY

Meboo

Convex Optimization

&

Euclidean Distance Geometry

Jon Dattorro

Moo Publishing

Meboo Publishing USA

PO Box 12

Palo Alto, California 94302

Dattorro, Convex Optimization & Euclidean Distance Geometry,Moo, 2005, v2013.03.30.

ISBN 0976401304 (English) ISBN 9780615193687 (International III)

This is version 2013.03.30: available in print, as conceived, in color.

cybersearch:

I. convex optimization

II. semidefinite program

III. rank constraint

IV. convex geometry

V. distance matrix

VI. convex cones

VII. cardinality minimization

programs by Matlab

typesetting by

with donations from SIAM and AMS.

This searchable electronic color pdfBook is click-navigable within the text bypage, section, subsection, chapter, theorem, example, definition, cross reference,citation, equation, figure, table, and hyperlink. A pdfBook has no electronic copyprotection and can be read and printed by most computers. The publisher herebygrants the right to reproduce this work in any format but limited to personal use.

2001-2013 MebooAll rights reserved.

for Jennie Columba

Antonio

& Sze Wan

EDM = Sh (Sc S+

)

Prelude

The constant demands of my department and university and theever increasing work needed to obtain funding have stolen much ofmy precious thinking time, and I sometimes yearn for the halcyondays of Bell Labs.

Steven Chu, Nobel laureate [82]

Convex Analysis is the calculus of inequalities while Convex Optimizationis its application. Analysis is inherently the domain of a mathematicianwhile Optimization belongs to the engineer. A convex optimization problemis conventionally regarded as minimization of a convex objective functionsubject to an artificial convex domain imposed upon it by the problemconstraints. The constraints comprise equalities and inequalities of convexfunctions whose simultaneous solution set generally constitutes the imposedconvex domain: called feasible set.

It is easy to minimize a convex function over any convex subset of itsdomain because any locally minimum value must be the global minimum.But it is difficult to find the maximum of a convex function over someconvex domain because there can be many local maxima. Althoughthis has practical application (Eternity II 4.6.0.0.15), it is not a convexproblem. Tremendous benefit accrues when a mathematical problem can betransformed to an equivalent convex optimization problem, primarily becauseany locally optimal solution is then guaranteed globally optimal.0.1

0.1Solving a nonlinear system, for example, by instead solving an equivalent convexoptimization problem is therefore highly preferable and what motivates geometricprogramming ; a form of convex optimization invented in 1960s [60] [80] that has drivengreat advances in the electronic circuit design industry. [34, 4.7] [250] [388] [391] [102][182] [191] [192] [193] [194] [195] [266] [267] [312]

2001 Jon Dattorro. co&edg version 2013.03.30. All rights reserved.citation: Dattorro, Convex Optimization & Euclidean Distance Geometry,Moo Publishing USA, 2005, v2013.03.30.

7

Recognizing a problem as convex is an acquired skill; that being, toknow when an objective function is convex and when constraints specifya convex feasible set. The challenge, which is indeed an art, is how toexpress difficult problems in a convex way: perhaps, problems previouslybelieved nonconvex. Practitioners in the art of Convex Optimization engagethemselves with discovery of which hard problems can be transformed intoconvex equivalents; because, once convex form of a problem is found, then aglobally optimal solution is close at hand the hard work is finished: Findingconvex expression of a problem is itself, in a very real sense, its solution.

Yet, that skill acquired by understanding the geometry and applicationof Convex Optimization will remain more an art for some time to come; thereason being, there is generally no unique transformation of a given problemto its convex equivalent. This means, two researchers pondering the sameproblem are likely to formulate a convex equivalent differently; hence, onesolution is likely different from the other although any convex combinationof those two solutions remains optimal. Any presumption of only one rightor correct solution becomes nebulous. Study of equivalence & sameness,uniqueness, and duality therefore pervade study of Optimization.

It can be difficult for the engineer to apply convex theory without anunderstanding of Analysis. These pages comprise my journal over a ten yearperiod bridging gaps between engineer and mathematician; they constitutea translation, unification, and cohering of about four hundred papers, books,and reports from several different fields of mathematics and engineering.Beacons of historical accomplishment are cited throughout. Much of whatis written here will not be found elsewhere. Care to detail, clarity, accuracy,consistency, and typography accompanies removal of ambiguity and verbosityout of respect for the reader. Consequently there is much cross-referencingand background material provided in the text, footnotes, and appendices soas to be more self-contained and to provide understanding of fundamentalconcepts.

Jon DattorroStanford, California

2011

8

Convex Optimization&Euclidean Distance Geometry

In my career, I found that the best people are the ones that really understand the

content, and theyre a pain in the butt to manage. But you put up with it because theyre

so great at the content. And thats what makes great products; its not process, its content.

Steve Jobs, 1995

1 Overview 21

2 Convex geometry 352.1 Convex set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.2 Vectorized-matrix inner product . . . . . . . . . . . . . . . . . 502.3 Hulls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612.4 Halfspace, Hyperplane . . . . . . . . . . . . . . . . . . . . . . 732.5 Subspace representations . . . . . . . . . . . . . . . . . . . . . 852.6 Extreme, Exposed . . . . . . . . . . . . . . . . . . . . . . . . . 932.7 Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 972.8 Cone boundary . . . . . . . . . . . . . . . . . . . . . . . . . . 1062.9 Positive semidefinite (PSD) cone . . . . . . . . . . . . . . . . . 1142.10 Conic independence (c.i.) . . . . . . . . . . . . . . . . . . . . . 1412.11 When extreme means exposed . . . . . . . . . . . . . . . . . . 1472.12 Convex polyhedra . . . . . . . . . . . . . . . . . . . . . . . . . 1482.13 Dual cone & generalized inequality . . . . . . . . . . . . . . . 156

3 Geometry of convex functions 2193.1 Convex function . . . . . . . . . . . . . . . . . . . . . . . . . . 2203.2 Practical norm functions, absolute value . . . . . . . . . . . . 225

9

10 CONVEX OPTIMIZATION & EUCLIDEAN DISTANCE GEOMETRY

3.3 Inverted functions and roots . . . . . . . . . . . . . . . . . . . 2353.4 Affine function . . . . . . . . . . . . . . . . . . . . . . . . . . 2373.5 Epigraph, Sublevel set . . . . . . . . . . . . . . . . . . . . . . 2413.6 Gradient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2503.7 Convex matrix-valued function . . . . . . . . . . . . . . . . . . 2623.8 Quasiconvex . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2693.9 Salient properties . . . . . . . . . . . . . . . . . . . . . . . . . 271

4 Semidefinite programming 2734.1 Conic problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 2744.2 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2844.3 Rank reduction . . . . . . . . . . . . . . . . . . . . . . . . . . 2994.4 Rank-constrained semidefinite program . . . . . . . . . . . . . 3084.5 Constraining cardinality . . . . . . . . . . . . . . . . . . . . . 3334.6 Cardinality and rank constraint examples . . . . . . . . . . . . 3544.7 Constraining rank of indefinite matrices . . . . . . . . . . . . . 4014.8 Convex Iteration rank-1 . . . . . . . . . . . . . . . . . . . . . 407

5 Euclidean Distance Matrix 4135.1 EDM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4145.2 First metric properties . . . . . . . . . . . . . . . . . . . . . . 4155.3 fifth Euclidean metric property . . . . . . . . . . . . . . . . 4155.4 EDM definition . . . . . . . . . . . . . . . . . . . . . . . . . . 4205.5 Invariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4535.6 Injectivity of D & unique reconstruction . . . . . . . . . . . . 4585.7 Embedding in affine hull . . . . . . . . . . . . . . . . . . . . . 4645.8 Euclidean metric versus matrix criteria . . . . . . . . . . . . . 4695.9 Bridge: Convex polyhedra to EDMs . . . . . . . . . . . . . . . 4775.10 EDM-entry composition . . . . . . . . . . . . . . . . . . . . . 4845.11 EDM indefiniteness . . . . . . . . . . . . . . . . . . . . . . . . 4875.12 List reconstruction . . . . . . . . . . . . . . . . . . . . . . . . 4955.13 Reconstruction examples . . . . . . . . . . . . . . . . . . . . . 4995.14 Fifth property of Euclidean metric . . . . . . . . . . . . . . . 505

6 Cone of distance matrices 5156.1 Defining EDM cone . . . . . . . . . . . . . . . . . . . . . . . . 5176.2 Polyhedral bounds . . . . . . . . . . . . . . . . . . . . . . . . 5196.3

EDM cone is not convex . . . . . . . . . . . . . . . . . . . . 520

CONVEX OPTIMIZATION & EUCLIDEAN DISTANCE GEOMETRY 11

6.4 EDM definition in 11T . . . . . . . . . . . . . . . . . . . . . . 521

6.5 Correspondence to PSD cone SN1+ . . . . . . . . . . . . . . . 5296.6 Vectorization & projection interpretation . . . . . . . . . . . . 535

6.7 A geometry of completion . . . . . . . . . . . . . . . . . . . . 540

6.8 Dual EDM cone . . . . . . . . . . . . . . . . . . . . . . . . . . 547

6.9 Theorem of the alternative . . . . . . . . . . . . . . . . . . . . 561

6.10 Postscript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563

7 Proximity problems 565

7.1 First prevalent problem: . . . . . . . . . . . . . . . . . . . . . 573

7.2 Second prevalent problem: . . . . . . . . . . . . . . . . . . . . 584

7.3 Third prevalent problem: . . . . . . . . . . . . . . . . . . . . . 596

7.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606

A Linear algebra 609

A.1 Main-diagonal operator, , tr , vec . . . . . . . . . . . . . 609

A.2 Semidefiniteness: domain of test . . . . . . . . . . . . . . . . . 614

A.3 Proper statements . . . . . . . . . . . . . . . . . . . . . . . . . 617

A.4 Schur complement . . . . . . . . . . . . . . . . . . . . . . . . 629

A.5 Eigenvalue decomposition . . . . . . . . . . . . . . . . . . . . 634

A.6 Singular value decomposition, SVD . . . . . . . . . . . . . . . 638

A.7 Zeros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643

B Simple matrices 651

B.1 Rank-one matrix (dyad) . . . . . . . . . . . . . . . . . . . . . 652

B.2 Doublet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657

B.3 Elementary matrix . . . . . . . . . . . . . . . . . . . . . . . . 658

B.4 Auxiliary V -matrices . . . . . . . . . . . . . . . . . . . . . . . 660

B.5 Orthomatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 665

C Some analytical optimal results 671

C.1 Properties of infima . . . . . . . . . . . . . . . . . . . . . . . . 671

C.2 Trace, singular and eigen values . . . . . . . . . . . . . . . . . 672

C.3 Orthogonal Procrustes problem . . . . . . . . . . . . . . . . . 680

C.4 Two-sided orthogonal Procrustes . . . . . . . . . . . . . . . . 682

C.5 Nonconvex quadratics . . . . . . . . . . . . . . . . . . . . . . 687

12 CONVEX OPTIMIZATION & EUCLIDEAN DISTANCE GEOMETRY

D Matrix calculus 689D.1 Directional derivative, Taylor series . . . . . . . . . . . . . . . 689D.2 Tables of gradients and derivatives . . . . . . . . . . . . . . . 710

E Projection 719E.1 Idempotent matrices . . . . . . . . . . . . . . . . . . . . . . . 723E.2 IP , Projection on algebraic complement . . . . . . . . . . . 728E.3 Symmetric idempotent matrices . . . . . . . . . . . . . . . . . 729E.4 Algebra of projection on affine subsets . . . . . . . . . . . . . 735E.5 Projection examples . . . . . . . . . . . . . . . . . . . . . . . 736E.6 Vectorization interpretation, . . . . . . . . . . . . . . . . . . . 743E.7 Projection on matrix subspaces . . . . . . . . . . . . . . . . . 751E.8 Range/Rowspace interpretation . . . . . . . . . . . . . . . . . 754E.9 Projection on convex set . . . . . . . . . . . . . . . . . . . . . 755E.10 Alternating projection . . . . . . . . . . . . . . . . . . . . . . 769

F Notation, Definitions, Glossary 791

Bibliography 811

Index 841

List of Figures

1 Overview 211 Orion nebula . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 Application of trilateration is localization . . . . . . . . . . . . 233 Molecular conformation . . . . . . . . . . . . . . . . . . . . . . 244 Face recognition . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Swiss roll . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 USA map reconstruction . . . . . . . . . . . . . . . . . . . . . 287 Honeycomb, Hexabenzocoronene molecule . . . . . . . . . . . 308 Robotic vehicles . . . . . . . . . . . . . . . . . . . . . . . . . . 319 Reconstruction of David . . . . . . . . . . . . . . . . . . . . . 3410 David by distance geometry . . . . . . . . . . . . . . . . . . . 34

2 Convex geometry 3511 Slab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3812 Open, closed, convex sets . . . . . . . . . . . . . . . . . . . . . 4113 Intersection of line with boundary . . . . . . . . . . . . . . . . 4214 Tangentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4515 Inverse image . . . . . . . . . . . . . . . . . . . . . . . . . . . 4916 Inverse image under linear map . . . . . . . . . . . . . . . . . 4917 Tesseract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5218 Linear injective mapping of Euclidean body . . . . . . . . . . 5419 Linear noninjective mapping of Euclidean body . . . . . . . . 5620 Convex hull of a random list of points . . . . . . . . . . . . . . 6121 Hulls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6322 Two Fantopes . . . . . . . . . . . . . . . . . . . . . . . . . . . 6623 Nuclear Norm Ball . . . . . . . . . . . . . . . . . . . . . . . . 67

13

14 LIST OF FIGURES

24 Convex hull of rank-1 matrices . . . . . . . . . . . . . . . . . . 6925 A simplicial cone . . . . . . . . . . . . . . . . . . . . . . . . . 7226 Hyperplane illustrated H is a partially bounding line . . . . 7427 Hyperplanes in R2 . . . . . . . . . . . . . . . . . . . . . . . . 7628 Affine independence . . . . . . . . . . . . . . . . . . . . . . . . 7929 {z C | aTz= i} . . . . . . . . . . . . . . . . . . . . . . . . . 8130 Hyperplane supporting closed set . . . . . . . . . . . . . . . . 8231 Minimizing hyperplane over affine set in nonnegative orthant . 8932 Maximizing hyperplane over convex set . . . . . . . . . . . . . 9033 Closed convex set illustrating exposed and extreme points . . 9534 Two-dimensional nonconvex cone . . . . . . . . . . . . . . . . 9835 Nonconvex cone made from lines . . . . . . . . . . . . . . . . 9836 Nonconvex cone is convex cone boundary . . . . . . . . . . . . 9937 Union of convex cones is nonconvex cone . . . . . . . . . . . . 9938 Truncated nonconvex cone X . . . . . . . . . . . . . . . . . . 10039 Cone exterior is convex cone . . . . . . . . . . . . . . . . . . . 10040 Not a cone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10241 Minimum element, Minimal element . . . . . . . . . . . . . . . 10542 K is a pointed polyhedral cone not full-dimensional . . . . . . 10843 Exposed and extreme directions . . . . . . . . . . . . . . . . . 11244 Positive semidefinite cone . . . . . . . . . . . . . . . . . . . . 11645 Convex Schur-form set . . . . . . . . . . . . . . . . . . . . . . 11746 Projection of truncated PSD cone . . . . . . . . . . . . . . . . 12047 Circular cone showing axis of revolution . . . . . . . . . . . . 13148 Circular section . . . . . . . . . . . . . . . . . . . . . . . . . . 13249 Polyhedral inscription . . . . . . . . . . . . . . . . . . . . . . 13450 Conically (in)dependent vectors . . . . . . . . . . . . . . . . . 14351 Pointed six-faceted polyhedral cone and its dual . . . . . . . . 14452 Minimal set of generators for halfspace about origin . . . . . . 14653 Venn diagram for cones and polyhedra . . . . . . . . . . . . . 15054 Simplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15255 Two views of a simplicial cone and its dual . . . . . . . . . . . 15456 Two equivalent constructions of dual cone . . . . . . . . . . . 15857 Dual polyhedral cone construction by right angle . . . . . . . 15958 K is a halfspace about the origin . . . . . . . . . . . . . . . . 16059 Iconic primal and dual objective functions . . . . . . . . . . . 16160 Orthogonal cones . . . . . . . . . . . . . . . . . . . . . . . . . 16661 Blades K and K . . . . . . . . . . . . . . . . . . . . . . . . . 168

LIST OF FIGURES 15

62 Membership w.r.t K and orthant . . . . . . . . . . . . . . . . 17863 Shrouded polyhedral cone . . . . . . . . . . . . . . . . . . . . 18264 Simplicial cone K in R2 and its dual . . . . . . . . . . . . . . . 18765 Monotone nonnegative cone KM+ and its dual . . . . . . . . . 19866 Monotone cone KM and its dual . . . . . . . . . . . . . . . . . 19967 Two views of monotone cone KM and its dual . . . . . . . . . 20068 First-order optimality condition . . . . . . . . . . . . . . . . . 204

3 Geometry of convex functions 21969 Convex functions having unique minimizer . . . . . . . . . . . 22170 Minimum/Minimal element, dual cone characterization . . . . 22371 1-norm ball B1 from compressed sensing/compressive sampling 22772 Cardinality minimization, signed versus unsigned variable . . 22973 1-norm variants . . . . . . . . . . . . . . . . . . . . . . . . . . 22974 Affine function . . . . . . . . . . . . . . . . . . . . . . . . . . 23975 Supremum of affine functions . . . . . . . . . . . . . . . . . . 24176 Epigraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24277 Gradient in R2 evaluated on grid . . . . . . . . . . . . . . . . 25078 Quadratic function convexity in terms of its gradient . . . . . 25779 Contour plot of convex real function at selected levels . . . . . 25980 Iconic quasiconvex function . . . . . . . . . . . . . . . . . . . 269

4 Semidefinite programming 27381 Venn diagram of convex program classes . . . . . . . . . . . . 27782 Visualizing positive semidefinite cone in high dimension . . . . 28083 Primal/Dual transformations . . . . . . . . . . . . . . . . . . 29084 Projection versus convex iteration . . . . . . . . . . . . . . . . 31185 Trace heuristic . . . . . . . . . . . . . . . . . . . . . . . . . . 31286 Sensor-network localization . . . . . . . . . . . . . . . . . . . . 31687 2-lattice of sensors and anchors for localization example . . . . 31988 3-lattice of sensors and anchors for localization example . . . . 32089 4-lattice of sensors and anchors for localization example . . . . 32190 5-lattice of sensors and anchors for localization example . . . . 32291 Uncertainty ellipsoids orientation and eccentricity . . . . . . . 32392 2-lattice localization solution . . . . . . . . . . . . . . . . . . . 32693 3-lattice localization solution . . . . . . . . . . . . . . . . . . . 32694 4-lattice localization solution . . . . . . . . . . . . . . . . . . . 327

16 LIST OF FIGURES

95 5-lattice localization solution . . . . . . . . . . . . . . . . . . . 32796 10-lattice localization solution . . . . . . . . . . . . . . . . . . 32897 100 randomized noiseless sensor localization . . . . . . . . . . 32898 100 randomized sensors localization . . . . . . . . . . . . . . . 32999 Regularization curve for convex iteration . . . . . . . . . . . . 332100 1-norm heuristic . . . . . . . . . . . . . . . . . . . . . . . . . . 335101 Sparse sampling theorem . . . . . . . . . . . . . . . . . . . . . 339102 Signal dropout . . . . . . . . . . . . . . . . . . . . . . . . . . 343103 Signal dropout reconstruction . . . . . . . . . . . . . . . . . . 344104 Simplex with intersecting line problem in compressed sensing . 346105 Geometric interpretations of sparse-sampling constraints . . . 350106 Permutation matrix column-norm and column-sum constraint 358107 max cut problem . . . . . . . . . . . . . . . . . . . . . . . . 367108 Shepp-Logan phantom . . . . . . . . . . . . . . . . . . . . . . 373109 MRI radial sampling pattern in Fourier domain . . . . . . . . 378110 Aliased phantom . . . . . . . . . . . . . . . . . . . . . . . . . 380111 Neighboring-pixel stencil on Cartesian grid . . . . . . . . . . . 382112 Differentiable almost everywhere . . . . . . . . . . . . . . . . . 383113 Eternity II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386114 Eternity II game-board grid . . . . . . . . . . . . . . . . . . . 388115 Eternity II demo-game piece illustrating edge-color ordering . 389116 Eternity II vectorized demo-game-board piece descriptions . . 390117 Eternity II difference and boundary parameter construction . 392118 Sparsity pattern for composite Eternity II variable matrix . . 393119 MIT logo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403120 One-pixel camera . . . . . . . . . . . . . . . . . . . . . . . . . 404121 One-pixel camera - compression estimates . . . . . . . . . . . 405

5 Euclidean Distance Matrix 413122 Convex hull of three points . . . . . . . . . . . . . . . . . . . . 414123 Complete dimensionless EDM graph . . . . . . . . . . . . . . . 417124 Fifth Euclidean metric property . . . . . . . . . . . . . . . . . 419125 Fermat point . . . . . . . . . . . . . . . . . . . . . . . . . . . 428126 Arbitrary hexagon in R3 . . . . . . . . . . . . . . . . . . . . . 430127 Kissing number . . . . . . . . . . . . . . . . . . . . . . . . . . 431128 Trilateration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436129 This EDM graph provides unique isometric reconstruction . . 441

LIST OF FIGURES 17

130 Two sensors and three anchors . . . . . . . . . . . . . . . 441131 Two discrete linear trajectories of sensors . . . . . . . . . . . . 442132 Coverage in cellular telephone network . . . . . . . . . . . . . 444133 Contours of equal signal power . . . . . . . . . . . . . . . . . . 445134 Depiction of molecular conformation . . . . . . . . . . . . . . 447135 Square diamond . . . . . . . . . . . . . . . . . . . . . . . . . . 456136 Orthogonal complements in SN abstractly oriented . . . . . . . 459137 Elliptope E3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478138 Elliptope E2 interior to S2+ . . . . . . . . . . . . . . . . . . . . 479139 Smallest eigenvalue of V TNDVN . . . . . . . . . . . . . . . . 485140 Some entrywise EDM compositions . . . . . . . . . . . . . . . 485141 Map of United States of America . . . . . . . . . . . . . . . . 498142 Largest ten eigenvalues of V TNOVN . . . . . . . . . . . . . . 501143 Relative-angle inequality tetrahedron . . . . . . . . . . . . . . 508144 Nonsimplicial pyramid in R3 . . . . . . . . . . . . . . . . . . . 512

6 Cone of distance matrices 515145 Relative boundary of cone of Euclidean distance matrices . . . 518146 Example of VX selection to make an EDM . . . . . . . . . . . 523147 Vector VX spirals . . . . . . . . . . . . . . . . . . . . . . . . . 526148 Three views of translated negated elliptope . . . . . . . . . . . 533149 Halfline T on PSD cone boundary . . . . . . . . . . . . . . . . 537150 Vectorization and projection interpretation example . . . . . . 538151 Intersection of EDM cone with hyperplane . . . . . . . . . . . 541152 Neighborhood graph . . . . . . . . . . . . . . . . . . . . . . . 543153 Trefoil knot untied . . . . . . . . . . . . . . . . . . . . . . . . 544154 Trefoil ribbon . . . . . . . . . . . . . . . . . . . . . . . . . . . 546155 Orthogonal complement of geometric center subspace . . . . . 551156 EDM cone construction by flipping PSD cone . . . . . . . . . 552157 Decomposing a member of polar EDM cone . . . . . . . . . . 556158 Ordinary dual EDM cone projected on S3h . . . . . . . . . . . 562

7 Proximity problems 565159 Pseudo-Venn diagram . . . . . . . . . . . . . . . . . . . . . . 568160 Elbow placed in path of projection . . . . . . . . . . . . . . . 569161 Convex envelope . . . . . . . . . . . . . . . . . . . . . . . . . 589

18 LIST OF FIGURES

A Linear algebra 609162 Geometrical interpretation of full SVD . . . . . . . . . . . . . 642

B Simple matrices 651163 Four fundamental subspaces of any dyad . . . . . . . . . . . . 653164 Four fundamental subspaces of a doublet . . . . . . . . . . . . 657165 Four fundamental subspaces of elementary matrix . . . . . . . 658166 Gimbal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667

D Matrix calculus 689167 Convex quadratic bowl in R2 R . . . . . . . . . . . . . . . . 700

E Projection 719168 Action of pseudoinverse . . . . . . . . . . . . . . . . . . . . . . 720169 Nonorthogonal projection of xR3 on R(U)=R2 . . . . . . . 726170 Biorthogonal expansion of point x aff K . . . . . . . . . . . . 738171 Dual interpretation of projection on convex set . . . . . . . . . 757172 Projection on orthogonal complement . . . . . . . . . . . . . . 761173 Projection on dual cone . . . . . . . . . . . . . . . . . . . . . 763174 Projection product on convex set in subspace . . . . . . . . . 768175 von Neumann-style projection of point b . . . . . . . . . . . . 771176 Alternating projection on two halfspaces . . . . . . . . . . . . 772177 Distance, optimization, feasibility . . . . . . . . . . . . . . . . 774178 Alternating projection on nonnegative orthant and hyperplane 777179 Geometric convergence of iterates in norm . . . . . . . . . . . 777180 Distance between PSD cone and iterate in A . . . . . . . . . . 782181 Dykstras alternating projection algorithm . . . . . . . . . . . 784182 Polyhedral normal cones . . . . . . . . . . . . . . . . . . . . . 785183 Normal cone to elliptope . . . . . . . . . . . . . . . . . . . . . 786184 Normal-cone progression . . . . . . . . . . . . . . . . . . . . . 788

List of Tables2 Convex geometry

Table 2.9.2.3.1, rank versus dimension of S3+ faces 123

Table 2.10.0.0.1, maximum number of c.i. directions 142

Cone Table 1 195

Cone Table S 195

Cone Table A 196

Cone Table 1* 201

4 Semidefinite programmingFaces of S3+ corresponding to faces of S3+ 281

5 Euclidean Distance MatrixPrecis 5.7.2: affine dimension in terms of rank 467

B Simple matricesAuxiliary V -matrix Table B.4.4 664

D Matrix calculusTable D.2.1, algebraic gradients and derivatives 711

Table D.2.2, trace Kronecker gradients 712

Table D.2.3, trace gradients and derivatives 713

Table D.2.4, logarithmic determinant gradients, derivatives 715

Table D.2.5, determinant gradients and derivatives 716

Table D.2.6, logarithmic derivatives 716

Table D.2.7, exponential gradients and derivatives 717


19

Chapter 1

Overview

Convex Optimization

Euclidean Distance Geometry

People are so afraid of convex analysis.

Claude Lemarechal, 2003

In laymans terms, the mathematical science of Optimization is the studyof how to make a good choice when confronted with conflicting requirements.The qualifier convex means: when an optimal solution is found, then it isguaranteed to be a best solution; there is no better choice.

Any convex optimization problem has geometric interpretation. If a givenoptimization problem can be transformed to a convex equivalent, then thisinterpretive benefit is acquired. That is a powerful attraction: the ability tovisualize geometry of an optimization problem. Conversely, recent advancesin geometry and in graph theory hold convex optimization within their proofscore. [399] [322]

This book is about convex optimization, convex geometry (withparticular attention to distance geometry), and nonconvex, combinatorial,and geometrical problems that can be relaxed or transformed into convexproblems. A virtual flood of new applications follows by epiphany that manyproblems, presumed nonconvex, can be so transformed. [10] [11] [59] [94][151] [153] [280] [301] [309] [366] [367] [396] [399] [34, 4.3, p.316-322]


21

22 CHAPTER 1. OVERVIEW

Figure 1: Orion nebula. (astrophotography by Massimo Robberto)

Euclidean distance geometry is, fundamentally, a determination of pointconformation (configuration, relative position or location) by inference frominterpoint distance information. By inference we mean: e.g., given onlydistance information, determine whether there corresponds a realizableconformation of points; a list of points in some dimension that attains thegiven interpoint distances. Each point may represent simply location or,abstractly, any entity expressible as a vector in finite-dimensional Euclideanspace; e.g., distance geometry of music [109].

It is a common misconception to presume that some desired pointconformation cannot be recovered in absence of complete interpoint distanceinformation. We might, for example, want to realize a constellation givenonly interstellar distance (or, equivalently, parsecs from our Sun and relativeangular measurement; the Sun as vertex to two distant stars); called stellarcartography, an application evoked by Figure 1. At first it may seem O(N 2)data is required, yet there are many circumstances where this can be reducedto O(N ).

x2

x4

x3

Figure 2: Application of trilateration (5.4.2.3.5) is localization (determiningposition) of a radio signal source in 2 dimensions; more commonly known byradio engineers as the process triangulation. In this scenario, anchorsx2 , x3 , x4 are illustrated as fixed antennae. [211] The radio signal source(a sensor x1) anywhere in affine hull of three antenna bases can beuniquely localized by measuring distance to each (dashed white arrowed linesegments). Ambiguity of lone distance measurement to sensor is representedby circle about each antenna. Trilateration is expressible as a semidefiniteprogram; hence, a convex optimization problem. [323]

If we agree that a set of points may have a shape (three points can forma triangle and its interior, for example, four points a tetrahedron), thenwe can ascribe shape of a set of points to their convex hull. It should beapparent: from distance, these shapes can be determined only to within arigid transformation (rotation, reflection, translation).

Absolute position information is generally lost, given only distanceinformation, but we can determine the smallest possible dimension in whichan unknown list of points can exist; that attribute is their affine dimension(a triangle in any ambient space has affine dimension 2, for example). Incircumstances where stationary reference points are also provided, it becomespossible to determine absolute position or location; e.g., Figure 2.

23


Figure 3: [190] [118] Distance data collected via nuclear magnetic resonance(NMR) helped render this 3-dimensional depiction of a protein molecule.At the beginning of the 1980s, Kurt Wuthrich [Nobel laureate] developedan idea about how NMR could be extended to cover biological moleculessuch as proteins. He invented a systematic method of pairing each NMRsignal with the right hydrogen nucleus (proton) in the macromolecule. Themethod is called sequential assignment and is today a cornerstone of all NMRstructural investigations. He also showed how it was subsequently possible todetermine pairwise distances between a large number of hydrogen nuclei anduse this information with a mathematical method based on distance-geometryto calculate a three-dimensional structure for the molecule. [284] [383] [185]

Geometric problems involving distance between points can sometimes bereduced to convex optimization problems. Mathematics of this combinedstudy of geometry and optimization is rich and deep. Its applicationhas already proven invaluable discerning organic molecular conformationby measuring interatomic distance along covalent bonds; e.g., Figure 3.[89] [356] [141] [46] Many disciplines have already benefitted and simplifiedconsequent to this theory; e.g., distance-based pattern recognition (Figure 4),localization in wireless sensor networks by measurement of intersensordistance along channels of communication, [47] [394] [45] wireless locationof a radio-signal source such as a cell phone by multiple measurements ofsignal strength, the global positioning system (GPS), and multidimensionalscaling which is a numerical representation of qualitative data by finding alow-dimensional scale.

25

Figure 4: This coarsely discretized triangulated algorithmically flattenedhuman face (made by Kimmel & the Bronsteins [228]) represents a stagein machine recognition of human identity; called face recognition. Distancegeometry is applied to determine discriminating-features.

Euclidean distance geometry together with convex optimization have alsofound application in artificial intelligence:

to machine learning by discerning naturally occurring manifolds in:

Euclidean bodies (Figure 5, 6.7.0.0.1),

Fourier spectra of kindred utterances [214],

and photographic image sequences [377],

to robotics ; e.g., automatic control of vehicles maneuvering information. (Figure 8)

by chapter

We study pervasive convex Euclidean bodies; their many manifestationsand representations. In particular, we make convex polyhedra, cones, anddual cones visceral through illustration in chapter 2 Convex geometrywhere geometric relation of polyhedral cones to nonorthogonal bases


Figure 5: Swiss roll from Weinberger & Saul [377]. The problem of manifoldlearning, illustrated for N= 800 data points sampled from a Swiss roll 1O.A discretized manifold is revealed by connecting each data point and its k=6nearest neighbors 2O. An unsupervised learning algorithm unfolds the Swissroll while preserving the local geometry of nearby data points 3O. Finally, thedata points are projected onto the two dimensional subspace that maximizestheir variance, yielding a faithful embedding of the original manifold 4O.

27

(biorthogonal expansion) is examined. It is shown that coordinates areunique in any conic system whose basis cardinality equals or exceeds spatialdimension; for high cardinality, a new definition of conic coordinate isprovided in Theorem 2.13.12.0.1. The conic analogue to linear independence,called conic independence, is introduced as a tool for study, analysis,and manipulation of cones; a natural extension and next logical step inprogression: linear, affine, conic. We explain conversion between halfspace-and vertex-description of a convex cone, we motivate the dual cone andprovide formulae for finding it, and we show how first-order optimalityconditions or alternative systems of linear inequality or linear matrixinequality can be explained by dual generalized inequalities with respect toconvex cones. Arcane theorems of alternative generalized inequality are,in fact, simply derived from cone membership relations ; generalizations ofalgebraic Farkas lemma translated to geometry of convex cones.

Any convex optimization problem can be visualized geometrically. Desireto visualize in high dimension [Sagan, CosmosThe Edge of Forever, 22:55]is deeply embedded in the mathematical psyche. [1] Chapter 2 providestools to make visualization easier, and we teach how to visualize in highdimension. The concepts of face, extreme point, and extreme direction of aconvex Euclidean body are explained here; crucial to understanding convexoptimization. How to find the smallest face of any pointed closed convexcone containing convex set C , for example, is divulged; later shown to havepractical application to presolving convex programs. The convex cone ofpositive semidefinite matrices, in particular, is studied in depth:

We interpret, for example, inverse image of the positive semidefinitecone under affine transformation. (Example 2.9.1.0.2)

Subsets of the positive semidefinite cone, discriminated by rankexceeding some lower bound, are convex. In other words, high-ranksubsets of the positive semidefinite cone boundary united with itsinterior are convex. (Theorem 2.9.2.9.3) There is a closed form forprojection on those convex subsets.

The positive semidefinite cone is a circular cone in low dimension,while Gersgorin discs specify inscription of a polyhedral cone into thatpositive semidefinite cone. (Figure 49)


original reconstruction

Figure 6: About five thousand points along borders constituting UnitedStates were used to create an exhaustive matrix of interpoint distance for eachand every pair of points in an ordered set (a list); called Euclidean distancematrix. From that noiseless distance information, it is easy to reconstructmap exactly via Schoenberg criterion (943). (5.13.1.0.1, confer Figure 141)Map reconstruction is exact (to within a rigid transformation) given anynumber of interpoint distances; the greater the number of distances, thegreater the detail (as it is for all conventional map preparation).

Chapter 3 Geometry of convex functions observes Fenchels analogybetween convex sets and functions: We explain, for example, how the realaffine function relates to convex functions as the hyperplane relates to convexsets. Partly a toolbox of practical useful convex functions and a cookbookfor optimization problems, methods are drawn from the appendices aboutmatrix calculus for determining convexity and discerning geometry.

Semidefinite programming is reviewed in chapter 4 with particularattention to optimality conditions for prototypical primal and dual problems,their interplay, and a perturbation method for rank reduction of optimalsolutions (extant but not well known). Positive definite Farkas lemmais derived, and we also show how to determine if a feasible set belongsexclusively to a positive semidefinite cone boundary. An arguably goodthree-dimensional polyhedral analogue to the positive semidefinite cone of33 symmetric matrices is introduced: a new tool for visualizing coexistence

29

of low- and high-rank optimal solutions in six isomorphic dimensions and amnemonic aid for understanding semidefinite programs. We find a minimalcardinality Boolean solution to an instance of Ax= b :

minimizex

x0subject to Ax = b

xi {0, 1} , i=1 . . . n(679)

The sensor-network localization problem is solved in any dimension in thischapter. We introduce a method of convex iteration for constraining rankin the form rankG and cardinality in the form cardx k . Cardinalityminimization is applied to a discrete image-gradient of the Shepp-Loganphantom, from Magnetic Resonance Imaging (MRI) in the field of medicalimaging, for which we find a new lower bound of 1.9% cardinality. Weshow how to handle polynomial constraints, and how to transform arank-constrained problem to a rank-1 problem.

The EDM is studied in chapter 5 Euclidean Distance Matrix; itsproperties and relationship to both positive semidefinite and Gram matrices.We relate the EDM to the four classical properties of Euclidean metric;thereby, observing existence of an infinity of properties of the Euclideanmetric beyond triangle inequality. We proceed by deriving the fifth Euclideanmetric property and then explain why furthering this endeavor is inefficientbecause the ensuing criteria (while describing polyhedra in angle or area,volume, content, and so on ad infinitum) grow linearly in complexity andnumber with problem size.

Reconstruction methods are explained and applied to a map of theUnited States; e.g., Figure 6. We also experimentally test a conjecture ofBorg & Groenen by reconstructing a distorted but recognizable isotonic mapof the USA using only ordinal (comparative) distance data: Figure 141e-f.We demonstrate an elegant method for including dihedral (or torsion)angle constraints into a molecular conformation problem. We explain whytrilateration (a.k.a localization) is a convex optimization problem. Weshow how to recover relative position given incomplete interpoint distanceinformation, and how to pose EDM problems or transform geometricalproblems to convex optimizations; e.g., kissing number of packed spheresabout a central sphere (solved in R3 by Isaac Newton).


(a) (b)

Figure 7: (a) These bees construct a honeycomb by solving a convexoptimization problem (5.4.2.3.4). The most dense packing of identicalspheres about a central sphere in 2 dimensions is 6. Sphere centersdescribe a regular lattice. (b) A hexabenzocoronene molecule (diameter :1.4nm) imaged by noncontact atomic force microscopy using a microscope tipterminated with a single carbon monoxide molecule. The carbon-carbon bondsin the imaged molecule appear with different contrast and apparent lengths.Based on these disparities, the bond orders and lengths of the individual bondscan be distinguished. (Image by Leo Gross)

The set of all Euclidean distance matrices forms a pointed closed convexcone called the EDM cone: EDMN . We offer a new proof of Schoenbergsseminal characterization of EDMs:

D EDMN { V TNDVN 0

D SNh(943)

Our proof relies on fundamental geometry; assuming, any EDM mustcorrespond to a list of points contained in some polyhedron (possibly atits vertices) and vice versa. It is known, but not obvious, this Schoenbergcriterion implies nonnegativity of the EDM entries; proved herein.

We characterize the eigenvalue spectrum of an EDM, and then devisea polyhedral spectral cone for determining membership of a given matrix(in Cayley-Menger form) to the convex cone of Euclidean distance matrices;id est, a matrix is an EDM if and only if its nonincreasingly ordered vectorof eigenvalues belongs to a polyhedral spectral cone for EDMN ;

D EDMN

([0 1T

1 D])

[RN+R

] H

D SNh(1159)

We will see: spectral cones are not unique.

31

Figure 8: Robotic vehicles in concert can move larger objects, guard aperimeter, perform surveillance, find land mines, or localize a plume of gas,liquid, or radio waves. [140]

In chapter 6 Cone of distance matrices we explain a geometricrelationship between the cone of Euclidean distance matrices, two positivesemidefinite cones, and the elliptope. We illustrate geometric requirements,in particular, for projection of a given matrix on a positive semidefinite conethat establish its membership to the EDM cone. The faces of the EDM coneare described, but still open is the question whether all its faces are exposedas they are for the positive semidefinite cone.

The Schoenberg criterion

D EDMN { V TNDVN SN1+

D SNh(943)

for identifying a Euclidean distance matrix is revealed to be a discretizedmembership relation (dual generalized inequalities, a new Farkas-like lemma)

between the EDM cone and its ordinary dual: EDMN. A matrix criterion

for membership to the dual EDM cone is derived that is simpler than the


Schoenberg criterion:

D EDMN (D1)D 0 (1309)

There is a concise equality, relating the convex cone of Euclidean distancematrices to the positive semidefinite cone, apparently overlooked in theliterature; an equality between two large convex Euclidean bodies:

EDMN = SNh (SNc SN+

)(1303)

Seemingly innocuous problems in terms of point position xi Rn like

minimize{xi}

i , j I

(xi xj hij)2 (1343)

minimize{xi}

i , j I

(xi xj2 hij)2 (1344)

are difficult to solve. So, in chapter 7 Proximity problems, we insteadexplore methods of solution to a few fundamental and prevalent Euclideandistance matrix proximity problems; the problem of finding that distancematrix closest, in some sense, to a given matrix H= [hij] :

minimizeD

V (D H)V 2Fsubject to rankV DV

D EDMN

minimizeD

D H2Fsubject to rankV DV

D

EDMN

minimizeD

D H2Fsubject to rankV DV

D EDMN

minimizeD

V ( D H)V 2Fsubject to rankV DV

D

EDMN

(1345)

We apply a convex iteration method for constraining rank. Known heuristicsfor rank minimization are also explained. We offer new geometrical proof, ofa famous discovery by Eckart & Young in 1936 [131], with particular regardto Euclidean projection of a point on that generally nonconvex subset ofthe positive semidefinite cone boundary comprising all semidefinite matriceshaving rank not exceeding a prescribed bound . We explain how thisproblem is transformed to a convex optimization for any rank .

33

appendices

We presume a reader already comfortable with elementary vector operations;[13, 3] formally known as analytic geometry. [385] Sundry toolboxes areprovided, in the form of appendices, so as to be more self-contained:

linear algebra (appendix A is primarily concerned with properstatements of semidefiniteness for square matrices),

simple matrices (dyad, doublet, elementary, Householder, Schoenberg,orthogonal, etcetera, in appendix B),

collection of known analytical solutions to some important optimizationproblems (appendix C),

matrix calculus remains somewhat unsystematized when comparedto ordinary calculus (appendix D concerns matrix-valued functions,matrix differentiation and directional derivatives, Taylor series, andtables of first- and second-order gradients and matrix derivatives),

an elaborate exposition offering insight into orthogonal andnonorthogonal projection on convex sets (the connection betweenprojection and positive semidefiniteness, for example, or betweenprojection and a linear objective function in appendix E),

Matlab code on Wmization to discriminate EDMs, to determineconic independence, to reduce or constrain rank of an optimal solutionto a semidefinite program, compressed sensing (compressive sampling)for digital image and audio signal processing, and two distinct methodsof reconstructing a map of the United States: one given only distancedata, the other given only comparative distance data.


Figure 9: Three-dimensional reconstruction of David from distance data.

Figure 10: Digital Michelangelo Project , Stanford University. Measuringdistance to David by laser rangefinder. Spatial resolution is 0.29mm.

Chapter 2

Convex geometry

Convexity has an immensely rich structure and numerousapplications. On the other hand, almost every convex idea canbe explained by a two-dimensional picture.

Alexander Barvinok [26, p.vii]

We study convex geometry because it is the easiest of geometries. Forthat reason, much of a practitioners energy is expended seeking invertibletransformation of problematic sets to convex ones.

As convex geometry and linear algebra are inextricably bonded byasymmetry (linear inequality), we provide much background material onlinear algebra (especially in the appendices) although a reader is assumedcomfortable with [334] [336] [204] or any other intermediate-level text.The essential references to convex analysis are [201] [310]. The readeris referred to [333] [26] [376] [42] [61] [307] [363] for a comprehensivetreatment of convexity. There is relatively less published pertaining to convexmatrix-valued functions. [217] [205, 6.6] [297]


35

36 CHAPTER 2. CONVEX GEOMETRY

2.1 Convex set

A set C is convex iff for all Y , Z C and 01Y + (1 )Z C (1)

Under that defining condition on , the linear sum in (1) is called a convexcombination of Y and Z . If Y and Z are points in real finite-dimensionalEuclidean vector space [229] [385] Rn or Rmn (matrices), then (1) representsthe closed line segment joining them. Line segments are thereby convex sets;C is convex iff the line segment connecting any two points in C is itself in C .Apparent from this definition: a convex set is a connected set. [260, 3.4, 3.5][42, p.2] A convex set can, but does not necessarily, contain the origin 0.

An ellipsoid centered at x= a (Figure 13 p.42), given matrix CRmn{xRn | C(x a)2 = (x a)TCTC(x a) 1} (2)

is a good icon for a convex set.2.1

2.1.1 subspace

A nonempty subset R of real Euclidean vector space Rn is called a subspace(2.5) if every vector2.2 of the form x + y , for , R , is inR whenevervectors x and y are. [252, 2.3] A subspace is a convex set containing theorigin, by definition. [310, p.4] Any subspace is therefore open in the sensethat it contains no boundary, but closed in the sense [260, 2]

R+ R = R (3)It is not difficult to show

R = R (4)as is true for any subspace R , because xR xR . Given any xR

R = x + R (5)The intersection of an arbitrary collection of subspaces remains a

subspace. Any subspace not constituting the entire ambient vector space Rn

is a proper subspace; e.g.,2.3 any line through the origin in two-dimensionalEuclidean space R2. The vector space Rn is itself a conventional subspace,inclusively, [229, 2.1] although not proper.

2.1This particular denition is slablike (Figure 11) in Rnwhen C has nontrivial nullspace.2.2A vector is assumed, throughout, to be a column vector.2.3We substitute abbreviation e.g. in place of the Latin exempli gratia.

2.1. CONVEX SET 37

2.1.2 linear independence

Arbitrary given vectors in Euclidean space {iRn, i=1 . . . N} are linearlyindependent (l.i.) if and only if, for all RN (iR)

1 1 + + N1 N1 N N = 0 (6)has only the trivial solution = 0 ; in other words, iff no vector from thegiven set can be expressed as a linear combination of those remaining.

Geometrically, two nontrivial vector subspaces are linearly independentiff they intersect only at the origin.

2.1.2.1 preservation of linear independence

(confer 2.4.2.4, 2.10.1) Linear transformation preserves linear dependence.[229, p.86] Conversely, linear independence can be preserved under lineartransformation. Given Y = [ y1 y2 yN ]RNN , consider the mapping

T () : RnN RnN , Y (7)whose domain is the set of all matrices RnN holding a linearlyindependent set columnar. Linear independence of {yiRn, i=1 . . . N}demands, by definition, there exist no nontrivial solution RN to

y1 i + + yN1 N1 yN N = 0 (8)By factoring out , we see that triviality is ensured by linear independenceof {yiRN}.

2.1.3 Orthant:

name given to a closed convex set that is the higher-dimensionalgeneralization of quadrant from the classical Cartesian partition of R2 ; aCartesian cone. The most common is the nonnegative orthant Rn+ or R

nn+

(analogue to quadrant I) to which membership denotes nonnegative vector-or matrix-entries respectively; e.g.,

Rn+ , {xRn | xi 0 i} (9)The nonpositive orthant Rn or R

nn (analogue to quadrant III) denotes

negative and 0 entries. Orthant convexity2.4 is easily verified bydefinition (1).

2.4All orthants are selfdual simplicial cones. (2.13.5.1, 2.12.3.1.1)


{yR2 | c aTy b} (2005)

Figure 11: A slab is a convex Euclidean body infinite in extent but notaffine. Illustrated in R2, it may be constructed by intersecting two opposinghalfspaces whose bounding hyperplanes are parallel but not coincident.Because number of halfspaces used in its construction is finite, slab is apolyhedron (2.12). (Cartesian axes + and vector inward-normal, to eachhalfspace-boundary, are drawn for reference.)

2.1.4 affine set

A nonempty affine set (from the word affinity) is any subset of Rn thatis a translation of some subspace. Any affine set is convex and open socontains no boundary: e.g., empty set , point, line, plane, hyperplane(2.4.2), subspace, etcetera. The intersection of an arbitrary collection ofaffine sets remains affine.

2.1.4.0.1 Definition. Affine subset.We analogize affine subset to subspace,2.5 defining it to be any nonemptyaffine set of vectors (2.1.4); an affine subset of Rn.

For some parallel 2.6 subspace R and any point xA

A is affine A = x +R= {y | y xR}

(10)

Affine hull of a set CRn (2.3.1) is the smallest affine set containing it.2.5The popular term affine subspace is an oxymoron.2.6Two ane sets are parallel when one is a translation of the other. [310, p.4]

2.1. CONVEX SET 39

2.1.5 dimension

Dimension of an arbitrary set S is Euclidean dimension of its affine hull;[376, p.14]

dimS , dim aff S = dim aff(S s) , sS (11)the same as dimension of the subspace parallel to that affine set aff S whennonempty. Hence dimension (of a set) is synonymous with affine dimension.[201, A.2.1]

2.1.6 empty set versus empty interior

Emptiness of a set is handled differently than interior in the classicalliterature. It is common for a nonempty convex set to have empty interior;e.g., paper in the real world:

An ordinary flat sheet of paper is a nonempty convex set having emptyinterior in R3 but nonempty interior relative to its affine hull.

2.1.6.1 relative interior

Although it is always possible to pass to a smaller ambient Euclidean spacewhere a nonempty set acquires an interior [26, II.2.3], we prefer the qualifierrelative which is the conventional fix to this ambiguous terminology.2.7 Sowe distinguish interior from relative interior throughout: [333] [376] [363]

Classical interior int C is defined as a union of points: x is an interiorpoint of CRn if there exists an open ball of dimension n and nonzeroradius centered at x that is contained in C .

Relative interior rel int C of a convex set CRn is interior relative toits affine hull.2.8

2.7Superuous mingling of terms as in relatively nonempty set would be an unfortunateconsequence. From the opposite perspective, some authors use the term full orfull-dimensional to describe a set having nonempty interior.2.8Likewise for relative boundary (2.1.7.2), although relative closure is superuous.

[201, A.2.1]


Thus defined, it is common (though confusing) for int C the interior of C tobe empty while its relative interior is not: this happens whenever dimensionof its affine hull is less than dimension of the ambient space (dim aff C

2.1. CONVEX SET 41

(a)

(b)

(c)

R2

Figure 12: (a) Closed convex set. (b) Neither open, closed, or convex. YetPSD cone can remain convex in absence of certain boundary components(2.9.2.9.3). Nonnegative orthant with origin excluded (2.6) and positiveorthant with origin adjoined [310, p.49] are convex. (c) Open convex set.

int C = C \ C a bounded open set has boundary defined but not contained in the set

interior of an open set is equivalent to the set itself;

from which an open set is defined: [260, p.109]

C is open int C = C (19)C is closed int C = C (20)

The set illustrated in Figure 12b is not open because it is not equivalentto its interior, for example, it is not closed because it does not containits boundary, and it is not convex because it does not contain all convexcombinations of its boundary points.

2.1.7.1 Line intersection with boundary

A line can intersect the boundary of a convex set in any dimension at apoint demarcating the lines entry to the set interior. On one side of that


(a)

(b)

(c)

R

R2

R3

Figure 13: (a) Ellipsoid in R is a line segment whose boundary comprises twopoints. Intersection of line with ellipsoid in R , (b) in R2, (c) in R3. Eachellipsoid illustrated has entire boundary constituted by zero-dimensionalfaces; in fact, by vertices (2.6.1.0.1). Intersection of line with boundaryis a point at entry to interior. These same facts hold in higher dimension.

2.1. CONVEX SET 43

entry-point along the line is the exterior of the set, on the other side is theset interior. In other words,

starting from any point of a convex set, a move toward the interior isan immediate entry into the interior. [26, II.2]

When a line intersects the interior of a convex body in any dimension, theboundary appears to the line to be as thin as a point. This is intuitivelyplausible because, for example, a line intersects the boundary of the ellipsoidsin Figure 13 at a point in R , R2, and R3. Such thinness is a remarkablefact when pondering visualization of convex polyhedra (2.12, 5.14.3) infour Euclidean dimensions, for example, having boundaries constructed fromother three-dimensional convex polyhedra called faces.

We formally define face in (2.6). For now, we observe the boundary ofa convex body to be entirely constituted by all its faces of dimension lowerthan the body itself. Any face of a convex set is convex. For example: Theellipsoids in Figure 13 have boundaries composed only of zero-dimensionalfaces. The two-dimensional slab in Figure 11 is an unbounded polyhedronhaving one-dimensional faces making its boundary. The three-dimensionalbounded polyhedron in Figure 20 has zero-, one-, and two-dimensionalpolygonal faces constituting its boundary.

2.1.7.1.1 Example. Intersection of line with boundary in R6.The convex cone of positive semidefinite matrices S3+ (2.9), in the ambientsubspace of symmetric matrices S3 (2.2.2.0.1), is a six-dimensional Euclideanbody in isometrically isomorphic R6 (2.2.1). Boundary of the positivesemidefinite cone, in this dimension, comprises faces having only thedimensions 0, 1, and 3 ; id est, {(+1)/2, =0, 1, 2}.

Unique minimum-distance projection PX (E.9) of any point X S3 onthat cone S3+ is known in closed form (7.1.2). Given, for example, intR3+and diagonalization (A.5.1) of exterior point

X = QQT S3, , 1 02

0 3

(21)

where QR33 is an orthogonal matrix, then the projection on S3+ in R6 is

PX = Q

1 02

0 0

QT S3+ (22)


This positive semidefinite matrix PX nearest X thus has rank 2, found bydiscarding all negative eigenvalues in . The line connecting these two pointsis {X + (PXX)t | tR} where t=0 X and t=1 PX . Becausethis line intersects the boundary of the positive semidefinite cone S3+ atpoint PX and passes through its interior (by assumption), then the matrixcorresponding to an infinitesimally positive perturbation of t there shouldreside interior to the cone (rank 3). Indeed, for an arbitrarily small positiveconstant,

X + (PXX)t|t=1+ = Q(+(P)(1+))QT = Q 1 02

0 3

QT intS3+

(23)2

2.1.7.1.2 Example. Tangential line intersection with boundary.A higher-dimensional boundary C of a convex Euclidean body C is simplya dimensionally larger set through which a line can pass when it does notintersect the bodys interior. Still, for example, a line existing in five ormore dimensions may pass tangentially (intersecting no point interior to C[221, 15.3]) through a single point relatively interior to a three-dimensionalface on C . Lets understand why by inductive reasoning.

Figure 14a shows a vertical line-segment whose boundary comprisesits two endpoints. For a line to pass through the boundary tangentially(intersecting no point relatively interior to the line-segment), it must exist inan ambient space of at least two dimensions. Otherwise, the line is confinedto the same one-dimensional space as the line-segment and must pass alongthe segment to reach the end points.

Figure 14b illustrates a two-dimensional ellipsoid whose boundary isconstituted entirely by zero-dimensional faces. Again, a line must exist inat least two dimensions to tangentially pass through any single arbitrarilychosen point on the boundary (without intersecting the ellipsoid interior).

Now lets move to an ambient space of three dimensions. Figure 14cshows a polygon rotated into three dimensions. For a line to pass through itszero-dimensional boundary (one of its vertices) tangentially, it must existin at least the two dimensions of the polygon. But for a line to passtangentially through a single arbitrarily chosen point in the relative interiorof a one-dimensional face on the boundary as illustrated, it must exist in atleast three dimensions.

2.1. CONVEX SET 45

R2(a) (b)

(c) (d)

R3

Figure 14: Line tangential: (a) (b) to relative interior of a zero-dimensionalface in R2, (c) (d) to relative interior of a one-dimensional face in R3.


Figure 14d illustrates a solid circular cone (drawn truncated) whoseone-dimensional faces are halflines emanating from its pointed end (vertex ).This cones boundary is constituted solely by those one-dimensional halflines.A line may pass through the boundary tangentially, striking only onearbitrarily chosen point relatively interior to a one-dimensional face, if itexists in at least the three-dimensional ambient space of the cone.

From these few examples, way deduce a general rule (without proof):

A line may pass tangentially through a single arbitrarily chosen pointrelatively interior to a k-dimensional face on the boundary of a convexEuclidean body if the line exists in dimension at least equal to k+2.

Now the interesting part, with regard to Figure 20 showing a boundedpolyhedron in R3 ; call it P : A line existing in at least four dimensionsis required in order to pass tangentially (without hitting intP) through asingle arbitrary point in the relative interior of any two-dimensional polygonalface on the boundary of polyhedron P . Now imagine that polyhedron P isitself a three-dimensional face of some other polyhedron in R4. To pass aline tangentially through polyhedron P itself, striking only one point fromits relative interior rel intP as claimed, requires a line existing in at leastfive dimensions.2.10

It is not too difficult to deduce:

A line may pass through a single arbitrarily chosen point interior to ak-dimensional convex Euclidean body (hitting no other interior point)if that line exists in dimension at least equal to k+1.

In laymans terms, this means: a being capable of navigating four spatialdimensions (one Euclidean dimension beyond our physical reality) could seeinside three-dimensional objects. 2

2.1.7.2 Relative boundary

The classical definition of boundary of a set C presumes nonempty interior:

C = C \ int C (17)2.10This rule can help determine whether there exists unique solution to a convexoptimization problem whose feasible set is an intersecting line; e.g., the trilaterationproblem (5.4.2.3.5).

2.1. CONVEX SET 47

More suitable to study of convex sets is the relative boundary ; defined[201, A.2.1.2]

rel C , C \ rel int C (24)boundary relative to affine hull of C .

In the exception when C is a single point {x} , (12)

rel {x} = {x}\{x} = , xRn (25)

A bounded convex polyhedron (2.3.2, 2.12.0.0.1) in subspace R , forexample, has boundary constructed from two points, in R2 from atleast three line segments, in R3 from convex polygons, while a convexpolychoron (a bounded polyhedron in R4 [378]) has boundary constructedfrom three-dimensional convex polyhedra. A halfspace is partially boundedby a hyperplane; its interior therefore excludes that hyperplane. An affineset has no relative boundary.

2.1.8 intersection, sum, difference, product

2.1.8.0.1 Theorem. Intersection. [310, 2, thm.6.5]Intersection of an arbitrary collection of convex sets {Ci} is convex. For afinite collection of N sets, a necessarily nonempty intersection of relativeinterior

Ni=1rel int Ci = rel int

Ni=1Ci equals relative interior of intersection.

And for a possibly infinite collection, Ci = Ci .

In converse this theorem is implicitly false insofar as a convex set can beformed by the intersection of sets that are not. Unions of convex sets aregenerally not convex. [201, p.22]

Vector sum of two convex sets C1 and C2 is convex [201, p.24]

C1+ C2 = {x + y | x C1 , y C2} (26)

but not necessarily closed unless at least one set is closed and bounded.By additive inverse, we can similarly define vector difference of two convex

sets

C1 C2 = {x y | x C1 , y C2} (27)which is convex. Applying this definition to nonempty convex set C1 , itsself-difference C1 C1 is generally nonempty, nontrivial, and convex; e.g.,


for any convex cone K , (2.7.2) the set K K constitutes its affine hull.[310, p.15]

Cartesian product of convex sets

C1 C2 ={[

xy

]| x C1 , y C2

}=

[ C1C2]

(28)

remains convex. The converse also holds; id est, a Cartesian product isconvex iff each set is. [201, p.23]

Convex results are also obtained for scaling C of a convex set C ,rotation/reflection Q C , or translation C+ ; each similarly defined.

Given any operator T and convex set C , we are prone to write T (C)meaning

T (C) , {T (x) | x C} (29)Given linear operator T , it therefore follows from (26),

T (C1 + C2) = {T (x + y) | x C1 , y C2}= {T (x) + T (y) | x C1 , y C2}= T (C1) + T (C2)

(30)

2.1.9 inverse image

While epigraph (3.5) of a convex function must be convex, it generallyholds that inverse image (Figure 15) of a convex function is not. The mostprominent examples to the contrary are affine functions (3.4):

2.1.9.0.1 Theorem. Inverse image. [310, 3]Let f be a mapping from Rpk to Rmn.

The image of a convex set C under any affine functionf(C) = {f(X) | X C} Rmn (31)

is convex.

Inverse image of a convex set F ,f1(F ) = {X | f(X)F} Rpk (32)

a single- or many-valued mapping, under any affine function f isconvex.

2.1. CONVEX SET 49

f

f

f1(F ) F

C

f(C)

(b)

(a)

Figure 15: (a) Image of convex set in domain of any convex function f isconvex, but there is no converse. (b) Inverse image under convex function f .

xp

Rn AAx = xp

Axp = b

Ax = b

R(AT) R(A)

N (A) N (AT)

A = 0

x=xp+

b

Rm

{b}

xp =Ab

00{x}

Figure 16:(confer Figure 168) Action of linear map represented byARmn :Component of vector x in nullspace N (A) maps to origin while componentin rowspace R(AT) maps to range R(A). For any ARmn, AAAx= b(E) and inverse image of b R(A) is a nonempty affine set: xp+N (A).


In particular, any affine transformation of an affine set remains affine.[310, p.8] Ellipsoids are invariant to any [sic] affine transformation.

Although not precluded, this inverse image theorem does not require auniquely invertible mapping f . Figure 16, for example, mechanizes inverseimage under a general linear map. Example 2.9.1.0.2 and Example 3.5.0.0.2offer further applications.

Each converse of this two-part theorem is generally false; id est, givenf affine, a convex image f(C) does not imply that set C is convex, andneither does a convex inverse image f1(F ) imply set F is convex. Acounterexample, invalidating a converse, is easy to visualize when the affinefunction is an orthogonal projector [334] [252]:

2.1.9.0.2 Corollary. Projection on subspace.2.11 (1977) [310, 3]Orthogonal projection of a convex set on a subspace or nonempty affine setis another convex set.

Again, the converse is false. Shadows, for example, are umbral projectionsthat can be convex when the body providing the shade is not.

2.2 Vectorized-matrix inner product

Euclidean space Rn comes equipped with a vector inner-product

y , z , yTz = yz cos (33)where (952) represents angle (in radians) between vectors y and z . Weprefer those angle brackets to connote a geometric rather than algebraicperspective; e.g., vector y might represent a hyperplane normal (2.4.2).Two vectors are orthogonal (perpendicular) to one another if and only iftheir inner product vanishes (iff is an odd multiple of

2);

y z y , z = 0 (34)When orthogonal vectors each have unit norm, then they are orthonormal.A vector inner-product defines Euclidean norm (vector 2-norm, A.7.1)

y2 = y ,yTy , y = 0 y = 0 (35)

2.11For hyperplane representations see 2.4.2. For projection of convex sets on hyperplanessee [376, 6.6]. A nonempty ane set is called an affine subset (2.1.4.0.1). Orthogonalprojection of points on ane subsets is reviewed in E.4.

2.2. VECTORIZED-MATRIX INNER PRODUCT 51

For linear operator A , its adjoint AT is a linear operator defined by[229, 3.10]

y ,ATz , Ay, z (36)For linear operation on a vector, represented by real matrix A , the adjointoperator AT is its transposition. This operator is self-adjoint when A=AT.

Vector inner-product for matrices is calculated just as it is for vectors;by first transforming a matrix in Rpk to a vector in Rpk by concatenatingits columns in the natural order. For lack of a better term, we shall callthat linear bijective (one-to-one and onto [229, App.A1.2]) transformationvectorization. For example, the vectorization of Y = [ y1 y2 yk ]Rpk[168] [330] is

vecY ,

y1y2...yk

Rpk (37)

Then the vectorized-matrix inner-product is trace of matrix inner-product;for Z Rpk, [61, 2.6.1] [201, 0.3.1] [387, 8] [370, 2.2]

Y , Z , tr(Y TZ) = vec(Y )TvecZ (38)where (A.1.1)

tr(Y TZ) = tr(ZY T) = tr(YZT) = tr(ZTY ) = 1T(Y Z)1 (39)and where denotes the Hadamard product 2.12 of matrices [161, 1.1.4]. Theadjoint AT operation on a matrix can therefore be defined in like manner:

Y , ATZ , AY , Z (40)Take any element C1 from a matrix-valued set in Rpk, for example, andconsider any particular dimensionally compatible real vectors v and w .Then vector inner-product of C1 with vwT isvwT, C1 = v , C1w = vTC1w = tr(wvTC1) = 1T

((vwT) C1

)1 (41)

Further, linear bijective vectorization is distributive with respect toHadamard product; id est,

vec(Y Z) = vec(Y ) vec(Z) (42)2.12Hadamard product is a simple entrywise product of corresponding entries from twomatrices of like size; id est, not necessarily square. A commutative operation, theHadamard product can be extracted from within a Kronecker product. [204, p.475]


(a) (b)

R2 R3

Figure 17: (a) Cube in R3 projected on paper-plane R2. Subspace projectionoperator is not an isomorphism because new adjacencies are introduced.(b) Tesseract is a projection of hypercube in R4 on R3.

2.2.0.0.1 Example. Application of inverse image theorem.Suppose set C Rpk were convex. Then for any particular vectors vRpand wRk, the set of vector inner-products

Y , vTCw = vwT, C R (43)

is convex. It is easy to show directly that convex combination of elementsfrom Y remains an element of Y .2.13 Instead given convex set Y , C mustbe convex consequent to inverse image theorem 2.1.9.0.1.

More generally, vwT in (43) may be replaced with any particular matrixZRpk while convexity of set Z , CR persists. Further, by replacingv and w with any particular respective matrices U and W of dimensioncompatible with all elements of convex set C , then set UTCW is convex bythe inverse image theorem because it is a linear mapping of C . 2

2.13To verify that, take any two elements C1 and C2 from the convex matrix-valued set C ,and then form the vector inner-products (43) that are two elements of Y by denition.Now make a convex combination of those inner products; videlicet, for 01

vwT, C1 + (1 ) vwT, C2 = vwT, C1 + (1 )C2

The two sides are equivalent by linearity of inner product. The right-hand side remainsa vector inner-product of vwT with an element C1 + (1 )C2 from the convex set C ;hence, it belongs to Y . Since that holds true for any two elements from Y , then it mustbe a convex set.


2.2.1 Frobenius

2.2.1.0.1 Definition. Isomorphic.An isomorphism of a vector space is a transformation equivalent to a linearbijective mapping. Image and inverse image under the transformationoperator are then called isomorphic vector spaces.

Isomorphic vector spaces are characterized by preservation of adjacency ;id est, if v and w are points connected by a line segment in one vectorspace, then their images will be connected by a line segment in the other.Two Euclidean bodies may be considered isomorphic if there exists anisomorphism, of their vector spaces, under which the bodies correspond.[372, I.1] Projection (E) is not an isomorphism, Figure 17 for example;hence, perfect reconstruction (inverse projection) is generally impossiblewithout additional information.

When Z=Y Rpk in (38), Frobenius norm is resultant from vectorinner-product; (confer (1725))

Y 2F = vecY 22 = Y , Y = tr(Y TY )=i, j

Y 2ij =i

(Y TY )i =i

(Y )2i(44)

where (Y TY )i is the ith eigenvalue of Y TY , and (Y )i the i

th singularvalue of Y . Were Y a normal matrix (A.5.1), then (Y )= |(Y )|[398, 8.1] thus

Y 2F =i

(Y )2i = (Y )22 = (Y ) , (Y ) = Y , Y (45)

The converse (45) normal matrix Y also holds. [204, 2.5.4]Frobenius norm is the Euclidean norm of vectorized matrices. Because

the metrics are equivalent, for X Rpk

vecXvecY 2 = XY F (46)

and because vectorization (37) is a linear bijective map, then vector spaceRpk is isometrically isomorphic with vector space Rpk in the Euclidean senseand vec is an isometric isomorphism of Rpk. Because of this Euclideanstructure, all the known results from convex analysis in Euclidean space Rn

carry over directly to the space of real matrices Rpk.


R2R3

T

dim domT = dimR(T )Figure 18: Linear injective mapping Tx=Ax : R2R3 of Euclidean bodyremains two-dimensional under mapping represented by skinny full-rankmatrix AR32 ; two bodies are isomorphic by Definition 2.2.1.0.1.

2.2.1.1 Injective linear operators

Injective mapping (transformation) means one-to-one mapping; synonymouswith uniquely invertible linear mapping on Euclidean space.

Linear injective mappings are fully characterized by lack of nontrivialnullspace.

2.2.1.1.1 Definition. Isometric isomorphism.An isometric isomorphism of a vector space having a metric defined on it is alinear bijective mapping T that preserves distance; id est, for all x, ydomT

Tx Ty = x y (47)Then isometric isomorphism T is called a bijective isometry.

Unitary linear operator Q : Rk Rk, represented by orthogonal matrixQRkk (B.5.2), is an isometric isomorphism; e.g., discrete Fouriertransform via (837). Suppose T (X)= UXQ is a bijective isometry whereU is a dimensionally compatible orthonormal matrix.2.14 Then we also say

2.14 any matrix U whose columns are orthonormal with respect to each other (UTU= I );these include the orthogonal matrices.


Frobenius norm is orthogonally invariant ; meaning, for X,Y Rpk

U(XY )QF = XY F (48)

Yet isometric operator T : R2 R3, represented by A = 1 00 1

0 0

on R2,

is injective but not a surjective map to R3. [229, 1.6, 2.6] This operator Tcan therefore be a bijective isometry only with respect to its range.

Any linear injective transformation on Euclidean space is uniquelyinvertible on its range. In fact, any linear injective transformation has arange whose dimension equals that of its domain. In other words, for anyinvertible linear transformation T [ibidem]

dim dom(T ) = dimR(T ) (49)

e.g., T represented by skinny-or-square full-rank matrices. (Figure 18) Animportant consequence of this fact is:

Affine dimension, of any n-dimensional Euclidean body in domain ofoperator T , is invariant to linear injective transformation.

2.2.1.1.2 Example. Noninjective linear operators.Mappings in Euclidean space created by noninjective linear operators can becharacterized in terms of an orthogonal projector (E). Consider noninjectivelinear operator Tx=Ax : RnRm represented by fat matrix ARmn(m


R3R3PT

B

PT (B)

x PTx

Figure 19: Linear noninjective mapping PTx=AAx : R3R3 ofthree-dimensional Euclidean body B has affine dimension 2 under projectionon rowspace of fat full-rank matrix AR23. Set of coefficients of orthogonalprojection TB= {Ax |xB} is isomorphic with projection P (TB) [sic].

2.2.2 Symmetric matrices

2.2.2.0.1 Definition. Symmetric matrix subspace.Define a subspace of RMM : the convex set of all symmetric MM matrices;

SM ,{ARMM | A=AT} RMM (50)

This subspace comprising symmetric matrices SM is isomorphic with thevector space RM(M+1)/2 whose dimension is the number of free variables in asymmetric MM matrix. The orthogonal complement [334] [252] of SM is

SM ,{ARMM | A=AT} RMM (51)

the subspace of antisymmetric matrices in RMM ; id est,

SM SM= RMM (52)where unique vector sum is defined on page 794.

All antisymmetric matrices are hollow by definition (have 0 maindiagonal). Any square matrix ARMM can be written as a sum of its


symmetric and antisymmetric parts: respectively,

A =1

2(A+AT) +

1

2(AAT) (53)

The symmetric part is orthogonal in RM2

to the antisymmetric part; videlicet,

tr((A+AT)(AAT)) = 0 (54)

In the ambient space of real matrices, the antisymmetric matrix subspacecan be described

SM ={

1

2(AAT) | ARMM

} RMM (55)

because any matrix in SM is orthogonal to any matrix in SM. Furtherconfined to the ambient subspace of symmetric matrices, because ofantisymmetry, SM would become trivial.

2.2.2.1 Isomorphism of symmetric matrix subspace

When a matrix is symmetric in SM , we may still employ the vectorizationtransformation (37) to RM

2

; vec , an isometric isomorphism. We mightinstead choose to realize in the lower-dimensional subspace RM(M+1)/2 byignoring redundant entries (below the main diagonal) during transformation.Such a realization would remain isomorphic but not isometric. Lack ofisometry is a spatial distortion due now to disparity in metric between RM

2

and RM(M+1)/2. To realize isometrically in RM(M+1)/2, we must make acorrection: For Y = [Yij] SM we take symmetric vectorization [217, 2.2.1]

svecY ,

Y112Y12Y22

2Y132Y23Y33...YMM

RM(M+1)/2 (56)

where all entries off the main diagonal have been scaled. Now for Z SM

Y , Z , tr(Y TZ) = vec(Y )TvecZ = svec(Y )TsvecZ (57)


Then because the metrics become equivalent, for X SM

svecX svecY 2 = X Y F (58)and because symmetric vectorization (56) is a linear bijective mapping, thensvec is an isometric isomorphism of the symmetric matrix subspace. In otherwords, SM is isometrically isomorphic with RM(M+1)/2 in the Euclidean senseunder transformation svec .

The set of all symmetric matrices SM forms a proper subspace in RMM ,so for it there exists a standard orthonormal basis in isometrically isomorphicRM(M+1)/2

{Eij SM} =

eieTi , i = j = 1 . . . M

12

(eie

Tj + eje

Ti

), 1 i < j M

(59)

where M(M + 1)/2 standard basis matrices Eij are formed from thestandard basis vectors

ei =

[{1 , i = j0 , i 6= j , j = 1 . . . M

] RM (60)

Thus we have a basic orthogonal expansion for Y SM

Y =Mj=1

ji=1

Eij , Y Eij (61)

whose unique coefficients

Eij , Y ={

Yii , i = 1 . . . M

Yij

2 , 1 i < j M(62)

correspond to entries of the symmetric vectorization (56).

2.2.3 Symmetric hollow subspace

2.2.3.0.1 Definition. Hollow subspaces. [357]Define a subspace of RMM : the convex set of all (real) symmetric MMmatrices having 0 main diagonal;

RMMh ,{ARMM | A=AT, (A) = 0} RMM (63)


where the main diagonal of ARMM is denoted (A.1)(A) RM (1450)

Operating on a vector, linear operator naturally returns a diagonal matrix;((A)) is a diagonal matrix. Operating recursively on a vector RN ordiagonal matrix SN , operator (()) returns itself;

2() (()) = (1452)The subspace RMMh (63) comprising (real) symmetric hollow matrices isisomorphic with subspace RM(M1)/2 ; its orthogonal complement is

RMMh ,{ARMM | A=AT + 22(A)} RMM (64)

the subspace of antisymmetric antihollow matrices in RMM ; id est,

RMMh RMMh = RMM (65)Yet defined instead as a proper subspace of ambient SM

SMh ,{A SM | (A) = 0} RMMh SM (66)

the orthogonal complement SMh of symmetric hollow subspace SMh ,

SMh ,{A SM | A=2(A)} SM (67)

called symmetric antihollow subspace, is simply the subspace of diagonalmatrices; id est,

SMh SMh = SM (68)

Any matrix ARMM can be written as a sum of its symmetric hollowand antisymmetric antihollow parts: respectively,

A =

(1

2(A+AT) 2(A)

)+

(1

2(AAT) + 2(A)

)(69)

The symmetric hollow part is orthogonal to the antisymmetric antihollowpart in RM

2

; videlicet,

tr

((1

2(A+AT) 2(A)

)(1

2(AAT) + 2(A)

))= 0 (70)


because any matrix in subspace RMMh is orthogonal to any matrix in theantisymmetric antihollow subspace

RMMh ={

1

2(AAT) + 2(A) | ARMM

} RMM (71)

of the ambient space of real matrices; which reduces to the diagonal matricesin the ambient space of symmetric matrices

SMh ={2(A) | ASM} = {(u) | uRM} SM (72)

In anticipation of their utility with Euclidean distance matrices (EDMs)in 5, for symmetric hollow matrices we introduce the linear bijectivevectorization dvec that is the natural analogue to symmetric matrixvectorization svec (56): for Y = [Yij] SMh

dvecY ,

2

Y12Y13Y23Y14Y24Y34...

YM1,M

RM(M1)/2 (73)

Like svec , dvec is an isometric isomorphism on the symmetric hollowsubspace. For X SMh

dvecX dvecY 2 = X Y F (74)The set of all symmetric hollow matrices SMh forms a proper subspace

in RMM , so for it there must be a standard orthonormal basis inisometrically isomorphic RM(M1)/2

{Eij SMh } ={

12

(eie

Tj + eje

Ti

), 1 i < j M

}(75)

where M(M1)/2 standard basis matrices Eij are formed from the standardbasis vectors eiRM .

The symmetric hollow majorization corollary A.1.2.0.2 characterizeseigenvalues of symmetric hollow matrices.

2.3. HULLS 61

Figure 20: Convex hull of a random list of points in R3. Some pointsfrom that generating list reside interior to this convex polyhedron (2.12).[378, Convex Polyhedron] (Avis-Fukuda-Mizukoshi)

2.3 Hulls

We focus on the affine, convex, and conic hulls: convex sets that may beregarded as kinds of Euclidean container or vessel united with its interior.

2.3.1 Affine hull, affine dimension

Affine dimension of any set in Rn is the dimension of the smallest affine set(empty set, point, line, plane, hyperplane (2.4.2), translated subspace, Rn)that contains it. For nonempty sets, affine dimension is the same as dimensionof the subspace parallel to that affine set. [310, 1] [201, A.2.1]

Ascribe the points in a list {x Rn, =1 . . . N} to the columns ofmatrix X :

X = [x1 xN ] RnN (76)In particular, we define affine dimension r of the N -point list X asdimension of the smallest affine set in Euclidean space Rn that contains X ;

r , dim aff X (77)

Affine dimension r is a lower bound sometimes called embedding dimension.[357] [187] That affine set A in which those points are embedded is uniqueand called the affine hull [333, 2.1];

A , aff {xRn, =1 . . . N} = aff X= x1 + R{x x1 , =2 . . . N} = {Xa | aT1 = 1} Rn (78)


for which we call list X a set of generators. Hull A is parallel to subspace

R{x x1 , =2 . . . N} = R(X x11T) Rn (79)

where R(A) = {Ax | x} (142)Given some arbitrary set C and any x C

aff C = x + aff(C x) (80)

where aff(Cx) is a subspace.

aff , (81)The affine hull of a point x is that point itself;

aff{x} = {x} (82)

Affine hull of two distinct points is the unique line through them. (Figure 21)The affine hull of three noncollinear points in any dimension is that uniqueplane containing the points, and so on. The subspace of symmetric matricesSm is the affine hull of the cone of positive semidefinite matrices; (2.9)

aff Sm+ = Sm (83)

2.3.1.0.1 Example. Affine hull of rank-1 correlation matrices. [220]The set of all mm rank-1 correlation matrices is defined by all the binaryvectors y in Rm (confer 5.9.1.0.1)

{yyT Sm+ | (yyT)=1} (84)

Affine hull of the rank-1 correlation matrices is equal to the set of normalizedsymmetric matrices; id est,

aff{yyT Sm+ | (yyT)=1} = {A Sm | (A)=1} (85)2

2.3.1.0.2 Exercise. Affine hull of correlation matrices.Prove (85) via definition of affine hull. Find the convex hull instead. H

2.3. HULLS 63

affine hull (drawn truncated)

convex hull

conic hull (truncated)

range or span is a plane (truncated)

A

C

K

R

Figure 21: Given two points in Euclidean vector space of any dimension,their various hulls are illustrated. Each hull is a subset of range; generally,A , C ,K R 0. (Cartesian axes drawn for reference.)


2.3.1.1 Partial order induced by RN+ and SM+

Notation a 0 means vector a belongs to nonnegative orthant RN+ whilea 0 means vector a belongs to the nonnegative orthants interior intRN+ .a b denotes comparison of vector a to vector b on RN with respect to thenonnegative orthant; id est, a b means a b belongs to the nonnegativeorthant but neither a or b is necessarily nonnegative. With particularrespect to the nonnegative orthant, a b ai bi i (369).

More generally, aK b or aK b denotes comparison with respect topointed closed convex cone K , but equivalence with entrywise comparisondoes not hold and neither a or b necessarily belongs to K . (2.7.2.2)

The symbol is reserved for scalar comparison on the real line R withrespect to the nonnegative real line R+ as in aTy b . Comparison ofmatrices with respect to the positive semidefinite cone SM+ , like I A 0in Example 2.3.2.0.1, is explained in 2.9.0.1.

2.3.2 Convex hull

The convex hull [201, A.1.4] [310] of any bounded2.15 list or set of N pointsX RnN forms a unique bounded convex polyhedron (confer 2.12.0.0.1)whose vertices constitute some subset of that list;

P , conv {x , =1 . . . N} = convX = {Xa | aT1 = 1, a 0} Rn(86)

Union of relative interior and relative boundary (2.1.7.2) of the polyhedroncomprise its convex hull P , the smallest closed convex set that contains thelist X ; e.g., Figure 20. Given P , the generating list {x} is not unique.But because every bounded polyhedron is the convex hull of its vertices,[333, 2.12.2] the vertices of P comprise a minimal set of generators.

Given some arbitrary set CRn, its convex hull conv C is equivalent tothe smallest convex set containing

Documents

0976401304.pdf