1

OR IIOR IIGSLM 52800GSLM 52800

2

OutlineOutline

some terminology

differences between LP and NLP

basic questions in NLP

gradient and Hessian

quadratic form

contour, graph, and tangent plane

3

Feasible Points, Solution Set, Feasible Points, Solution Set, and Neighborhoodand Neighborhood

feasible point: a point that satisfies all the constraints

solution set (feasible set, feasible region): the collection of all feasible points

neighborhood of x0 = {x| |xx0| < } for some pre-specified

feasible region

the neighborhood of a point for a given

A

C

B

D

only the neighborhood of D is completely

feasible for this

4

Weak and Strong; Weak and Strong; Local and Global Local and Global

local minima: x1, any point in [s, t], x3

strict (strong) local minima: x1, x3

weak local minima: any point in [s, t] strict global minimum: x1

weak local maxima: any point in [s, t]

x

f(x )

12x3

x2

x1 ts

5

Differences Between Differences Between

Linear and Non-Linear ProgrammingLinear and Non-Linear Programming linear programming

there exists an optimal extreme point (a corner point) direction of improvement keeps on being so unless

hitting a constraint a local optimum point is also globally optimal

direction of improvement

optimal point

6

Differences Between Differences Between

Linear and Non-Linear ProgrammingLinear and Non-Linear Programming

none of these necessarily holds for a non-linear program

x

f(x )

12x3

x2

x1 ts

min x2 + y2,s.t. -2 x, y 2

7

Basic Questions Basic Questions in Non-Linear Programming in Non-Linear Programming

main question: given an initial location x0, how to get to a local minimum, or, better, a global minimum (a) the direction of improvement? (b) the necessary conditions of an optimal point? (c) the sufficient conditions of an optimal point? (d) any conditions to simplify the processes in (a),

(b), and (c)? (e) any algorithmic procedures to solve a NLP

problem?

8

Basic Questions Basic Questions in Non-Linear Programming in Non-Linear Programming

calculus required for (a) to (e)

direction of improvement of f = gradient of f shaped by constraints

convexity for (d), and also (b) and (c) identification of convexity: definiteness of

matrices, especially for Hessians

9

Gradient and HessianGradient and Hessian

gradient of f: f(x) = in short

Hessian =

1

( ) ( ),...,n

Tf fx x

x x

( )

j

fx

x

2 2 2

21 2 11

2 2

22 1 2

2 2

21

( ) ( ) ( )

( ) ( )

( ) ( )

n

n n

f f fx x x xx

f fx x x

f fx x x

x x x

x x

x x

L

M

M O M

L L

f and gj usually assumed to be

twice differentiable functions Hessian is a

symmetric matrix

10

Gradient and HessianGradient and Hessian

ej: (0, …, 0, 1, 0, …, 0)T, where “1” at the jth position

for small , f(x+ej) f(x) +

in general, x = (x1, …, xn)T from x,

f(x+x) f(x) +

( ) ( )( )

0lim j

j

f ffx

x e xx

( )

j

fx

x

( )

j

fjx

jx

x

11

Example 1.6.1Example 1.6.1

(a). f(x) = x2; f(3.5+) ? for small

(b). f(x, y) = x2 + y2, f((1, 1) + (x, y)) ? for small x, y

gradient f : direction of steepest accent of the objective fucntion

12

Example 1.6.2Example 1.6.2

find the Hessian of (a). f(x, y) = x2 + 7y2

(b). f(x, y) = x2 + 5xy + 7y2

(c). f(x, y) = x3 + 7y2

13

Quadratic FormQuadratic Form

general form: xTQx/2 + cTx + a, where x is an n-dimensional vector; Q an nn square matrix; c and a are matrices of appropriate dimensions how to derive the gradient and Hessian?

gradient f(x) = Qx+c

Hessian H = Q

14

Quadratic FormQuadratic Form

relate the two forms xTQx/2 + cTx + a and f(x, y) = 1x2+2xy+3y2+4x+5y+6

Example 1.6.3

15

Example 1.6.4Example 1.6.4

Find the first two derivatives of the following f(x) f(x) = x2 for x [-2, 2]

f(x) = -x2 for x [-2, 2]

16

Contour and Graph Contour and Graph (i.e., Surface) of Function (i.e., Surface) of Function ff

Example 1.7.1: f(x1, x2) = 2 21 2x x

17

Contour and Graph Contour and Graph (i.e., Surface) of Function (i.e., Surface) of Function ff

an n-dimensional function

a contour of f: a diagram f(x) = c in the n-dimensional space for a given value c

the graph (surface function) of f: the diagram z = f(x) in the (n+1)st dimensional space as x and z vary

18

Contour and Graph Contour and Graph (i.e., Surface) of Function (i.e., Surface) of Function ff

how do the contours of the one-dimensional function f(x) = x2 look like?

19

An Important Property Between the Gradient An Important Property Between the Gradient and the Tangent Plane at a Contourand the Tangent Plane at a Contour

the gradient of f at point x0 is orthogonal to the tangent of the contour f(x) = c at x0

many optimization results are related to the above property

20

Gradient of f at x0 Being Orthogonal to the Tangent of the Contour f(x) = c at x0

Example 1.7.3: f(x1, x2) = x1+2x2

gradient at (4, 2)?

tangent of contour at (4, 2)?

21

Gradient of f at x0 Being Orthogonal to the Tangent of the Contour f(x) = c at x0

Example 1.7.2: f(x1, x2) =

point (x10, x20) on a contour f(x1, x2) = c

2 21 2x x

22

Tangent at a Contour and the Corresponding Tangent at a Contour and the Corresponding Tangent Plane at a SurfaceTangent Plane at a Surface

the above two are related

for contour of f(x, y) = x2+y2, the tangent at (x0, y0)

(x-x0, y- y0)T(2x0, 2y0) = 0

two orthogonal vectors u and v:

uTv = 0

23

Tangent at a Contour and the Corresponding Tangent at a Contour and the Corresponding Tangent Plane at a SurfaceTangent Plane at a Surface

the tangent place at (x0, y0) for the surface of f(x, y) = x2+y2

the surface: z = x2+y2

defining a contour at a higher dimension: F(x, y, z) = x2+y2z tangent plane at (x0, y0, ) of the surface: 2 2

0 0x y

what happens when z = 2 2

0 0 ?x y