View
214
Download
0
Category
Preview:
Citation preview
1
15.053 Thursday, April 25
• Nonlinear Programming Theory
• Separable programming
Handouts: Lecture Notes
3
Graphical Analysis of Non-linear programsin two dimensions: An example
• Minimize
• subject to (x - 8)2 + (y - 9)2 ≤ 49 x ≥2 x ≤13 x + y ≤24
4
Where is the optimal solution?
Note: the optimalsolution is not at acorner point.It is where theisocontour first hitsthe feasible region.
5
Another example: Minimize(x-8)2 + (y-8)2
Then the globalunconstrainedminimum is alsofeasible.
The optimalsolution is not onthe boundary ofthe feasible region.
6
Local vs. Global Optima
Def’n: Let x be a feasible solution, then
– x is a global max if f(x) ≥ f(y) for every feasible y.
– x is a local max if f(x) ≥ f(y) for every feasible y sufficiently close to x (i.e., xj-ε ≤ yj ≤ xj+ ε for all j and some small ε).
There may be several locally optimal solutions.
7
When is a locally optimal solutionalso globally optimal?
• We are minimizing. The objective function is convex. The feasible region is convex.
8
Convexity and Extreme Points
We say that a set S is convex, if for everytwo points x and y in S, and for every realnumber λ in [0,1], λx + (1-λ)y ε S.
The feasible region of a linear program is convex.
We say that an element w ε S is anextreme point (vertex, corner point), if wis not the midpoint of any line segmentcontained in S.
9
Recognizing convex feasible regions
• If all constraints are linear, then the feasible region is convex
• The intersection of convex regions is convex
• If for all feasible x and y, the midpoint of x and y is feasible, then the region is convex (except in totally non-realistic examples. )
11
Convex FunctionsConvex Functions: f( y + (1- )z) f(y) + (1- )f(z)for every y and z and for 0. e.g., f((y+z)/2) f(y)/2 + f(z)/2
We say “strict” convexity if signis “<” for 0.
Line joining any pointsis above the curve
12
Convex FunctionsConvex Functions: f( y + (1- )z) ≥ f(y) + (1- )f(z)for every y and z and for 0. e.g., f((y+z)/2) ≥ f(y)/2 + f(z)/2
We say “strict” convexity if signis “<” for 0.
Line joining any pointsis above the curve
14
What functions are convex?
• f(x) = 4x + 7 all linear functions• f(x) = 4x2 – 13 some quadratic functions• f(x) = ex
• f(x) = 1/x for x > 0• f(x) = |x|• f(x) = - ln(x) for x > 0
Sufficient condition: f”(x) > 0 for all x.
15
What functions are convex?
• If f(x) is convex, and g(x) is convex. Then so is h(x) = a f(x) + b g(x) for a>0, b>0.
• If y = f(x) is convex, then {(x,y) : f(x) ≤ y} is a convex set
16
Local Maximum (Minimum)Property
• A local max of a concave function on a convex feasible region is also a global max.• A local min of a convex function on a convex feasible region is also a global min.• Strict convexity or concavity implies that the global optimum is unique.• Given this, we can exactly solve: – Maximization Problems with a concave objective function and linear constraints – Minimization Problems with a convex objective function and linear constraints
18
More on local optimality
• The techniques for non-linear optimization minimization usually find local optima.• This is useful when a locally optimal solution is a globally optimal solution• It is not so useful in many situations.
• Conclusion: if you solve an NLP, try to find out how good the local optimal solutions are.
20
Finding a local optimal for a singlevariable NLP
Solving NLP's with One Variable:
max f(θ)s.t. a ≤ θ ≤ b
Optimal solution is either a boundary point or satisfies f′(θ∗) = 0 and f ″(θ∗) < 0.
21
Solving Single Variable NLP (contd.)
If f(θ) is concave (or simply unimodal) and differentiable
max f(θ)s.t. a ≤ θ ≤ b
Bisection (or Bolzano) Search:
• Step 1. Begin with the region of uncertainty for θ as [a,b]. Evaluate f ′ (θ) at the midpoint θΜ =(a+b)/2.
• Step 2. If f ′ (θΜ) > 0, then eliminate the interval up to θΜ.If f′(θΜ) < 0, then eliminate the interval beyond θΜ.
• Step 3. Evaluate f′(θ) at the midpoint of the newinterval. Return to Step 2 until the interval of uncertaintyis sufficiently small.
22
Unimodal Functions
• A single variable function f is unimodal if there is at most one local maximum (or at most one local
minimum) .
23
Other Search Techniques
• Instead of taking derivatives (which may be computationally intensive), use two function evaluations to determine updated interval.• Fibonacci Search• Step 1. Begin with the region of uncertainty for θ as [a, b]. Evaluate f (θ1) and f (θ2) for 2 symmetric points θ1<θ2.• Step 2. If f (θ1) ≤ f (θ2), then eliminate the interval up tθ1. If f (θ1) > f (θ2), then eliminate the interval beyond θ2.• Step 3. Select a second point symmetric to the point already in the new interval, rename these points θ1 and θ2 such that θ1<θ2 and evaluate f (θ1) and f (θ2). Return to Step 2 until the interval is sufficiently small.
24
On Fibonacci search
1, 1, 2, 3, 5, 8, 13, 21, 34At iteration 1, the length of the search interval is the kth fibonacci number for some kAt iteration j, the length of the search interval is the k-j+1 fibonacci number.The technique converges to the optimal when the function is unimodal.
25
Finding a local maximum usingFibonacci Search.
Length of search
interval
Where the maximum may be
28
Number of function evaluations inFibonacci Search
• As new point is chosen symmetrically, the length lk of successive search intervals is given by: lk= lk+1 + lk+2 .• Solving for these lengths given a final interval length of 1, ln = 1, gives the Fibonacci numbers: 1, 2, 3, 5, 8, 13, 21, 34,…• Thus, if the initial interval has length 34, it takes 8 function calculations to reduce the interval length to 1.
• Remark: if the function is convex or unimodal, then fibonacci search converges to the global maximum
29
Separable Programming
• Separable programs have the form:
Max
st
Each variable xj appears separately, one ineach function gij and one in each function fj inthe objective.
Each non-linear function is of a single variable.
30
Examples of SeparablePrograms
f(x1,x2) = x1(30-x1)+x2(35-x2)-2x12-3x2
2
f(x1,x2) = x15 + -18e-x2-4x2
3
x1
f(x1,x2,x2) = lnx15-sinx2-x3e-x3)+7x1
-4
31
Approximating a non-linear functionwith a piecewise linear function.
• Aspect 1. Choosing the approximation.• Aspect 2. When is the piecewise linear approximation a linear program in disguise?
33
Approximating a non-linear functionof 1 variable: the λ method
Choosedifferentvalues of x toapproximatethe x-axis
Approximateusingpiecewiselinearsegments
34
More on the method
Suppose that for –3 ≤ x ≤ -1,
we represent x hasλ1 (-3) +λ2 (-1)
whereλ1 +λ2 = 1 andλ1,λ2 ≥ 0
Then we approximate f(x) as
λ1 (-20) +λ2 (-7 1/3)
a1 = -3, f(a1) = -20
a2 = -1 f(a2) = -7 1/3
35
More on the method
a2 = -1, f(a2) = -7 1 /3
a3 = 1 f(a3) = -2 2/3
Suppose that for –1 ≤ x ≤ 1,
we represent x hasλ2 (-3) +λ3 (-1)whereλ2 +λ3 = 1 andλ2,λ3 ≥ 0
How do weapproximate f( ) inthis interval?
What if –3 ≤ x ≤ 1?
36
Almost the method
Original problem:
min x3/3 + 2x – 5 + more terms
s.t. -3 ≤ x ≤ 3 + many more constraints
a1 = -3; a2 = - 1; a3 = 1; a4 = 3f(a1) = -20; f(a2) = -7 1/3; f(a3) = -2 2/3; f(a4) = 4
Approximated problem:
minλ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)
more linear terms
s.t. λ1 + λ2 + λ3 + λ4= 1 ; λ ≥ 0
+ many more constraints
37
Why the approximation is incorrect
Approximated problem:
minλ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)
more linear terms
s.t. λ1+λ2 +λ3+λ4 = 1 ; λ ≥ 0Considerλ1= 1/2 ;λ2=0 ;λ3=1/2 ;λ4 = 0;
The method givesthe correctapproximation ifonly twoconsecutiveλs arepositive.
38
Adjacency Condition
1. At most two weights (λs) are positive2. If exactly two weights (λs) are positive, then they are λj and λj+1 for some j3. The same condition applies to every approximated function.
39
Approximating a non-linear objectivefunction for a minimization NLP. original problem: minimize {f(y): y ∈P} Suppose that y = Σjjaj , where Σjj = 1 and >= 0 . Approximate f(y). minimize {Σjjf(aj): Σjjaj∈P}
• Note: when given a choice of representing y in alternative ways, the LP will choose one that leads to the least objective value for the approximation.
40
For minimizing a convex function, the-method automatically satisfies theadditional adjacency property.
min Z =λ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)+λ5f(a5) s.t. λ1 + λ2 + λ3 + λ4 +λ5 = 1; ≥0 + adjacency condition + other constraints
41
Feasible approximated objective functions without the adjacency conditions
min Z =λ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)+λ5f(a5) s.t. λ1 + λ2 + λ3 + λ4 +λ5 = 1; ≥0 + other constraints
42
But a minimum in this case always occurson the piecewise linear curve.
min Z =λ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)+λ5f(a5) s.t. λ1 + λ2 + λ3 + λ4 +λ5 = 1; ≥0 + other constraints
43
Separable Programming (in the case oflinear constraints)
• Begin with an NLP:
• Transform to Separable:
• Approximate using the Method:
Max f ( )xs.t. Dx d x 0
Max ∑fj (xj) s.t. Dx d x
0
n
j=1
44
Approximation
• Re-express in terms of variables:
If the original problem is concave, then you can drop theadjacency conditions (they are automatically satisfied.)
and the adjacency conditions
45
How can one construct separable functions?
Term Substitution Constraints Restriction
None
None
None
46
Transformation Examples
Ex : (x1 + x2 + x3)6
Substitute y6 and let y = x1 + x2 + x3
x1x22
1+ x3
Ex : let y1and let y2
and add the constraint
47
NLP Summary
• Convex and Concave functions as well as convex sets are important properties• Bolzano and Fibonacci search techniques – used to solve single variable unimodal functions• Separable Programming – nonlinear objective and nonlinear constraints that are separable – General approximation technique
Recommended