View
24
Download
0
Category
Preview:
Citation preview
Convex Optimization
CMU-10725Penalty Methods
Barnabás Póczos & Ryan Tibshirani
2
Outline
� Penalty functions
3
Books to Read
David G. Luenberger, Yinyu Ye: Linear and Nonlinear Programming
Boyd and Vandenberghe: Convex Optimization
4
Penalty Methods
5
Penalty Methods
Penalty program: replace (P) with the unconstrained problem:
Penalty term: high cost for violation of the constraints
6
Inequality Constraints
A useful penalty function in this case is:
7
Penalty Methods
David G. Luenberger, Yinyu Ye: Linear and Nonlinear Programming
8
Inequality and Equality Constraints
Inequality and Equality constraints:
Rewrite them as:
9
Penalty Method
Penalty parameter:
Penalty program:
Penalty Lemma:
10
Proof of Penalty Lemma (1)
11
Proof of Penalty Lemma (2)
12
Proof of Penalty Lemma (3)
13
Proof of Penalty Lemma (4)
14
Convergence of Penalty Method
Theorem: [Penalty convergence]
15
Proof of Penalty Convergence
16
Penalty functions
Often used penalty functions
Polynomial penalty:
Linear penalty:
Quadratic penalty:
17
Inequality and Equality constraints
Problem (P)
Definition [Penalty function]
Example [Penalty function]
18
Derivative of the penalty function
Penalty program:
Penalty function:
Assumptions:
Derivatives:
19
Derivative of the penalty function
Difficulties: max is not differentiable
This is not perfectly correct, because
Solution:
Example:
20
KKT in Penalty methods
Penalty program:
Penalty function:
Derivatives:
1st order condition in local minimum:
21
KKT and Penalty method multipliers
Penalty program:
Problem (P)
Theorem: Under some mild conditions
KKT multipliers:
Recommended