Convex Functions Handout

7/28/2019 Convex Functions Handout

1/17

Semidenite matrices &Convex functions

Rudi Pendavingh

Eindhoven Technical University

Optimization in R n , lecture 7

Rudi Pendavingh (TUE) Semidenite matrices & Convex functions ORN7 1 / 17


2/17

Semidenite matricesA symmetric matrix A is positive semidenite (PSD) if and only if x t Ax 0 for all x R n .

Theorem

Let A be a symmetric matrix. The following are equivalent:1 A is PSD, i.e. x t Ax 0 for all x R n .2

all eigenvalues of A are nonnegative.3 A = Z t Z for some real matrix Z .

CorollaryLet p R [X 1, . . . , X n ] be a homogeneous quadratic polynomial. Thenp (x 1, . . . , x n ) 0 for all x 1, . . . , x n R if and only if

p = s 21 + + s 2n

for some s i R [X 1, . . . , X n ]



3/17

Recognizing PSD matricesDenitionsymmetric matrix operations on a matrix are:

1 multiplying both the i -th row and i -th column by = 02 swapping the i -th and j -th column; and swapping the i -th and j -th

row3 adding i -th column to j -th column and adding i -th row to

j -th rowA = B : B is obtained from A by zero or more symmetric matrixoperations.

LemmaA = B if and only if B = Y t AY for some invertible Y .

LemmaLet A, B be symmetric. If A = B , then A is PSD B is PSD .



4/17

Recognizing PSD matrices

To determine whether A is PSD, compute a diagonal matrix D = A.

Example

A = 1 22 3 =1 00 1 =

D .

D is not PSD, hence A is not PSD.

Example

A =1 2 12 5 1

1 1 12=

1 0 00 1 30 3 11

=1 0 00 1 00 0 2

= D .

D is PSD, hence A is PSD.



5/17

Positive denite matrices

A matrix A is positive denite (PD) if x t Ax > 0 for all nonzero x R n .

LemmaA is PD A is PSD and det( A) = 0 .

Theorem

Let A be a symmetric matrix. The following are equivalent:1 A is PD, i.e. x t Ax > 0 for all nonzero x R n .2 all eigenvalues of A are positive.3 A = Z t Z for some real matrix Z so that det( Z ) = 0 .

LemmaLet A, B be symmetric. If A = B , then A is PD B is PD .



6/17

Further characterizations

Let A be a square matric and let I { 1, . . . , n}. Then AI denotes therestriction of A to the rows and columns indexed by I .

Lemma

Let A be an n n matrix. Then A is PSD det( AI ) 0 for all I { 1, . . . , n}.

LemmaLet A be an n n matrix. Then A is PD det( AI ) > 0 for I = {1}, . . . , {1, . . . , n}.



7/17

Gersgorins TheoremGiven a complexn n matrix A, we put p := j = p |apj | for p = 1 , . . . , n.

Theorem (Gersgorin, 1931)Let A be a complex n n matrix and let be an eigenvalue of A. Then| app | p for some p { 1, . . . , n}.

Proof.Let x C n be a nonzero vector such that x = Ax .Let p be such that |x p | = max i |x i |.We have x p = n j =1 apj x j , hence ( app )x p = j = p apj x j .

Taking norms, | app || x p | = | j = p apj x j | j = p |apj || x j | p |x p | .Dividing by |x p |, the Theorem follows.

For each p , we have the Gersgorin disk { C | | app | p }. TheTheorem states that each eigenvalue is in one of the Gersgorin disks.



8/17

Convex and concave functions

DenitionA function f : R n R is convex if Dom(f ) is convex, and

f (x ) + (1 )f (y ) f (x + (1 )y ),

for all x , y Dom(f ) and for all [0, 1].

Denitionf is concave if f is convex.

Lemmaf is convex and concave if and only if f is affine.



9/17

Norms

DenitionA function f : R n R is a norm if

f (x ) 0 for all x R n

f (x ) = 0 if and only if x = 0f (x ) = | |f (x ) for all x R n , R

f (x + y ) f (x ) + f (y ) for all x , y R n

TheoremLet f : R n R be a norm. Then f is convex.

LemmaLet f : R n R be a convex function. Then {x R n | f (x ) } is aconvex set for any R .

So the norm ball {x R n | x 1} is convex for any norm . .Rudi Pendavingh (TUE) Semidenite matrices & Convex functions ORN7 9 / 17


10/17

DenitionThe epigraph of a function f : R n R is

epi(f ) := {(x , t ) Rn+1

| x Dom(f ), t f (x )}.

Lemmaf is a convex function if and only if epi (f ) is a convex set.

DenitionA subgradient of f at x is a row vector w such that

f (y ) f (x ) w (y x ) for all y .

Theoremf is convex Dom (f ) is convex, and there exists a subgradient of f at each x in the interior of Dom (f ).



11/17

Operations that preserve convexity

LemmaIf f : R n R is convex, then the following functions are convex as well:x f (x ), for any > 0x f (x + t ), for any xed t R n

x f (Ax ), for any n m matrix A

LemmaIf f 1, . . . , f m : R n R are convex, then the following functions are convex as well:

x max{f 1(x ), . . . , f m (x )}x f 1(x ) + + f m (x )



12/17

Differentiable convex functions

Denition

The gradient of f : Rn

R at x is f (x ) := ( f

x 1 (x ), . . . , f

x n (x )) .

LemmaIf w is a subgradient of f at x , and f is differentiable at x , then

w = f (x ).

Theorem (First-order condition for convexity)A differentiable function f is convex if and only if

f (y ) f (x ) + f (x )(y x )

for all x , y Dom (f ).

Note: x is a minimizer of f if and only if 0 is a subgradient of f at x .



13/17

Twice differentiable convex functions

DenitionThe Hessian of a function f : R n R at x is the matrix

2f (x ) :=

2f x 21

(x ) 2f

x 1 x n (x )...

... 2f

x n x 1 (x ) 2f x 2n (x )

.

Theorem (Second-order condition for convexity)Let f be a twice differentiable function. Then f is convex Dom (f ) is

convex and 2f (x ) is positive semidenite

for all x in the interior of Dom (f ).



14/17

Quadratic functions

ExampleLet f : x x

t Qx 2 + px , where Q is a symmetric matrix, p a row vector.

f (x ) = x t Q + p for all x 2f (x ) = Q for all x

f is convex if and only if Q is PSDif f is convex, then f (y ) = min {f (x ) | x R n } if and only if

0 = f (y ) = y t Q + p

Note: f is convex if and only if the second-order approximation of f isconvex everywhere.



15/17

Some convex functions

Example (Functions of one variable)x e ax is convex on R , for any a Rx x a is

convex on {x R | x > 0} when a > 1 or a < 0concave on {x R | x > 0} when 0 a 1

x log(x ) is concave on {x R | x > 0}x x log(x ) is convex on {x R | x > 0}



16/17

More convex functions

Example (Functions of several variables)(x , y ) x 2/ y is convex on {(x , y ) R 2 | y > 0}(x 1, . . . , x n ) max{x 1, . . . , x n } is convex on R n

(x 1, . . . , x n ) log(e x 1 + + e x n ) is convex on R n(x 1, . . . , x n ) ( ni =1 x i )

1n is concave on {x R n | x > 0}

X log(det( X )) is concave on

{X | X is PSD, det( X ) = 0 }



17/17

The function f : X log(det(X )) is convexProof.It suffices to show that for any PD X and symmetric Y , the function

g : t log(det( X + tY ))

is convex. AsX is PD, there exists a PD matrix Z so that Z 2 = X . Hence

g (t ) = log(det( ZZ )) log(det( Z 1

(X + tY )Z 1

)) == log(det( X )) log(det( I + tY ))

where Y = Z 1YZ 1. Let 1, . . . , n be the eigenvalues of Y .Then det( I + tY ) = i (1 + t i ), hence

g (t ) = log(det( X )) i

log(1 + t i ).

This is a sum of a constant and convex funtions t log(1 + t ).


Documents

Convex Functions Handout