Lecture Static

7/31/2019 Lecture Static

1/31

The ProfessorThe Class Overview

The Class...Numerical Optimization

Math 693A: Advanced Numerical Analysis Numerical Optimization

Lecture Notes #1 Introduction

Peter Blomgren,[email protected]

Department of Mathematics and StatisticsDynamical Systems Group

Computational Sciences Research Center

San Diego State UniversitySan Diego, CA 92182-7720

http://terminus.sdsu.edu/

Fall 2012

Peter Blomgren, [email protected] Lecture Notes #1 Introduction (1/31)
http://terminus.sdsu.edu/http://terminus.sdsu.edu/


2/31


3/31



Academic LifeContact Information, Office Hours

Academic Life

MSc. Engineering Physics, Royal Institute of Technology (KTH), Stockholm, Sweden. Thesis Advisers:Michael Benedicks, Department of Mathematics KTH, andErik Aurell, Stockholm University, Department of Mathematics. Thesis Topic: A Renormalization Technique for Families with Flat Maxima.

PhD. UCLA Department of Mathematics. Adviser:Tony F. Chan. PDE-Based Methods for Image Processing.Thesis title: Total Variation Methods for Restoration of Vector Valued Images.



4/31




Academic Life

Research Associate. Stanford University,

Department of Mathematics. Main Focus: Time Reversal andImaging in Random Media (with George Papanicolaou, et. al.)

Professor, San Diego State University, Departmentof Mathematics and Statistics. Projects: ComputationalCombustion, Biomedical Imaging (Mitochondrial Structures,Heartcell Contractility, Skin Cancer Classication).



5/31




Contact Information

Office GMCS-587Email [email protected]

Web http://terminus.sdsu.edu/SDSU/Math693a f2012/Phone (619)594-2602Office Hours TuTh: 10:45 11:30a, 3:00 3:45p

and by appointment


h f
http://terminus.sdsu.edu/SDSU/Math693a_f2012/http://terminus.sdsu.edu/SDSU/Math693a_f2012/


6/31



Literature & SyllabusGradingExpectations and Procedures

Math 693A: Literature

Required (A Modern Treatment of NumericalOptimization)Numerical Optimization, 2nd Edition, Jorge Nocedal and StephenJ. Wright, Springer Series in Operations Research, Springer Verlag,

2006. ISBN-10: 0387303030; ISBN-13: 978-0387303031Required (Supplemental)Class notes and class web-page.

Optional (A Classic in the eld; Source for classprojects)Numerical Methods for Unconstrained Optimization and Nonlinear Equations , J. E. Dennis, Jr. and Robert B. Schnabel, Classics inApplied Mathematics 16, Society for Industrial and AppliedMathematics (SIAM), 1996. ISBN 0-89871-364-1.


Th P f


7/31




Math 693A: Introduction What we will cover

NW-2 Unconstrained Optimization

NW-3 Line Search Methods

NW-4 Trust Region Methods

NW-5 Conjugate Gradient Methods

NW-6 Quasi-Newton Methods

NW-7 Calculating Derivatives

NW-10 Least Squares Problems

NW-11 Nonlinear Equations


The Professor


8/31




Math 693A: Introduction Grading etc.

40% Homework: both theoretical, and implementation (program-ming) C/C++ or Fortran are the recommended languages,but feel free to program in 6510 assembler, Java, M$-D , orMatlab... Class accounts will be available.

60% Project: Implementation of several interacting parts of anoptimization package. By the end of the semester you should

have a working toolbox of optimization algorithms whichwill be useful in your current and future research projects.[Complete details TBA].


The Professor


9/31




Expectations and Procedures, I

Most class attendance is OPTIONAL Homework andannouncements will be posted on the class web page. If/whenyou attend class:

Please be on time.

Please pay attention.

Please turn off mobile phones.

Please be courteous to other students and the instructor.Abide by university statutes, and all applicable local, state, andfederal laws.


The Professor


10/31




Expectations and Procedures, II

Please, turn in assignments on time. (The instructor reservesthe right not to accept late assignments.)The instructor will make special arrangements for studentswith documented learning disabilities and willtry to makeaccommodations for other unforeseen circumstances, e.g.illness, personal/family crises, etc. in a way that is fair to allstudents enrolled in the class. Please contact the instructorEARLY regarding special circumstances.

Students are expected and encouraged to ask questions inclass!Students are expected and encouraged to to make use of office hours! If you cannot make it to the scheduled officehours: contact the instructor to schedule an appointment!


The Professor Li & S ll b


11/31




Expectations and Procedures, III

Missed midterm exams: Dont miss exams! The instructorreserves the right to schedule make-up exams, make such

exams oral presentation, and/or base the grade solely on otherwork (including the nal exam).

Missed nal exam: Dont miss the nal! Contact theinstructor ASAP or a grade of WUor F will be assigned.

Academic honesty : submit your own work but feel free todiscuss homework with other students in the class!


The Professor Lit t & S ll b


12/31

The Class OverviewThe Class...

Numerical Optimization


Honesty Pledges, I

The following Honesty Pledge must be included in allprograms you submit (as part of homework and/or projects):

I, (your name), pledge that this program is completely my ownwork, and that I did not take, borrow or steal code from anyother person, and that I did not allow any other person to use,have, borrow or steal portions of my code. I understand that if I violate this honesty pledge, I am subject to disciplinary actionpursuant to the appropriate sections of the San Diego StateUniversity Policies.

Work missing the honesty pledge may not be graded!


The Professor Literature & Syllabus


13/31




Honesty Pledges, II

Larger reports must contain the following text:

I, (your name), pledge that this report is completely my own

work, and that I did not take, borrow or steal any portionsfrom any other person. Any and all references I used areclearly cited in the text. I understand that if I violate thishonesty pledge, I am subject to disciplinary action pursuant tothe appropriate sections of the San Diego State UniversityPolicies. Your signature .

Work missing the honesty pledge may not be graded!


The Professor


14/31



ResourcesFormal Prerequisites

Math 693A: Computer Resources

You need access to a computing environment in which to writeyour code; you may want to use a combination of Matlab (forquick prototyping and short homework assignments) and C/C++or Fortran (or something completely different).

Class accounts for the computer lab(s) will be available.

You can also use the Rohan Sun Enterprise system or anothercapable system. [http://www-rohan.sdsu.edu/raccts.html ]

Free C/C++ ( gcc ) and Fortran ( f77 ) compilers are available forLinux/UNIX.

You may also want to consider buying the student version of Matlab: http://www.mathworks.com/


The Professor
http://www-rohan.sdsu.edu/raccts.htmlhttp://www-rohan.sdsu.edu/raccts.htmlhttp://www.mathworks.com/http://www.mathworks.com/http://www-rohan.sdsu.edu/raccts.html


15/31



ResourcesFormal Prerequisites

Math 693A: Introduction What you should know already

Math 524 and (Math 542 or Math 543 )

524 Linear Algebra Vector spaces, linear transformations, orthogonality, eigenvalues

and eigenvectors, normal forms for complex matrices, positive

denite matrices.542 Numerical Solutions of Differential Equations

Initial and boundary value problems for ODEs. PDEs. Iterativemethods, nite difference methods, the method of lines.

543 Numerical Matrix Analysis Gaussian elimination, LU-factorizations and pivoting strategies.

Direct and iterative methods for linear systems. Iterative methodsfor diagonalization and eigensystem computation. Tridiagonal,Hessenberg, and Householder matrices. The QR algorithm.


The ProfessorTh Cl O i The What? Why? and How?


16/31



The What? Why? and How?Concepts & TermsMathematical Formulation

Math 693A: Introduction Why???

Q: Why do we need numerical optimization methods?

A: Many problems in applications are formulated as optimizationproblems:

Optimal trajectories for airplanes, space craft, robotic motion,etc.

Optimal shape for cars, airfoils, aerodynamic bicycle wheels,etc.

Risk management investment portfolios; insurance premi-ums.

Circuit and network design.


The ProfessorTh Cl O i The What? Why? and How?


17/31



yConcepts & TermsMathematical Formulation

Numerical Optimization: Concepts and Term 1 of 2

Students optimize: minimize study time T , such that GPA isacceptable. ( :-)

Nature optimizes: Physical systems settle in a state of minimalenergy A ball rolls down to the bottom of a slope; DNA molecules fold to minimize some measure of energy; Light rays follow the path that minimizes travel time. Chemical reactions are energy-driven, etc, etc, etc...

In order to understand physical (economic, etc.) systems we mustoptimize: rst we must identify the objective (the measure of performance, or energy). The objective depends on a number of variables (the characteristics of the system).


The ProfessorThe Class Overview The What? Why? and How?


18/31



yConcepts & TermsMathematical Formulation

Numerical Optimization: Concepts and Term 2 of 2

Our goal is to nd the values of the variables that optimize (eitherminimize, or maximize) the objective . Often the variables are

constrained (restricted) in some way (e.g. densities, and interestrates are non-negative).

The process of identifying the objective , variables , andconstraints is non-trivial and will essentially be completely ignoredin this class. (See Mathematical Modeling )

Our discussion starts after the modeling is done!




19/31



Concepts & TermsMathematical Formulation

Mathematical Formulation 1 of 2

From the point of view of a mathematician, optimization is theminimization (or maximization) of a function subject to constraintson its variables.

Notation:

x the vector of variables (a.k.a. unknowns, or parameters)f (x) the objective functionc the vector of constraints that the unknowns must satisfy

The Optimization Problem can be written

minxR n

f (x) subject to c i (x) = 0 , i E c i (x) 0, i I




20/31




Mathematical Formulation 2 of 2

The Optimization Problem

minxR n


Here E is the set of equality constraints , and I the set of inequality constraints .

Note that a maximization problem can be converted into aminimization problem:

maxxR n

f (x) minxR n

[ f (x)]

and a less-than-or-equal-to constraint can similarly be convertedinto a greater-than-or-equal-to constraint.


The ProfessorThe Class Overview The What? Why? and How?C & T


21/31




Feasible Region

The set of all x R n which satisfy the constraints c is called thefeasible region , e.g. if

c 1(x 1 , x 2) = x 21 + x 22 1c 2(x 1 , x 2) = (x 21 + x 22 ) 4

then the feasible region is the annulus:

Figure: The annulus 1 r 2.


The ProfessorThe Class Overview The What? Why? and How?C t & T


22/31



Constrained vs. Unconstrained Optimization 1 of 2

Optimization problems of the form

minxR n


can be classied according to the nature of the function andconstraints (linear, non-linear, convex, etc.) the key distinctionis between problems that have constraints, and problems that donot:

Constrained Optimization Problems : arise from models thatinclude explicit constraints on the variables. They can be relativelysimple, or nasty non-linear inequalities expressing complexrelationships between the variables.


The ProfessorThe Class Overview The What? Why? and How?Concepts & Terms


23/31



Constrained vs. Unconstrained Optimization 2 of 2

Unconstrained Optimization Problems arise directly in someapplications; if the constraints are natural it may be safe todisregard them during the solution process and verify that they are

satised in the solution.Further, constrained problems can be restated as unconstrainedproblems the constraints are replaced by penalizing terms in theobjective which discourage violation of the constraints.

The more complicated the constraints, the more difficult it is tond the optimal solution. The absence of constraints is the easiestcase.




24/31



Continuous vs. Discrete Optimization

In many applications, the variables (x) can only take integer values very few people would be interested in buying 3/4 of a TV set,or receive 1/3 of a package; the electrons in an atom can only existin certain quantum states, etc, etc.

The Discrete Optimization Problem is harder than the ContinuousOptimization Problem. One way to get close to solving thediscrete problem is to solve the problem as if it is continuous andthen round or truncate the solution to integer values. This will

often give a sub-optimal integer solution and/or a solution that isinfeasible .

Here we will only consider the easier Continuous OptimizationProblem.




25/31



Global and Local Optimization

The fastest optimization algo-rithms seek a local solution a point where the objective issmaller than all other feasiblepoints in its vicinity.

The best solution the globalminimum is usually hard to nd,but is often desirable.

-10 0 10-1

0

1

2

3

Figure: A function with multiple local minima, and one global minimum.

Under certain circumstances (e.g. see convexity) there is only oneminimum.

We will focus on local optimization algorithms, but note that(most) global algorithms will solve a sequence of local optimizationproblems.



Th Cl

The What? Why? and How?Concepts & Terms


26/31



Stochastic and Deterministic Optimization

In many applications it is impossible to fully specify all parameters

at the time of formulation; in quantum physics, the stock market,or the game of risk some quantities are random and are bestmodeled using some probability model.

We will focus ondeterministic optimization problems, where themodel can be fully specied when we formulate the problem.

However, in many cases the solutions to stochastic optimizationproblems are formulated as sequences or collections of deterministic problems.



Th Cl



27/31


pMathematical Formulation

Convexity: Denitions

There are two types of convexity which impact optimizationproblems:

S R n is a convex set if the straight line segment connectingany two points in S lies entirely insideS . Formally, for any two

points x S and y S , we have ( x + (1 )y) S for all [0, 1].

f is a convex function if its domain is a convex set and if forany two points x and y in this domain, the graph of f lies below

the straight line connecting ( x, f (x)) to ( y, f (y)) in R n+1

. Thatis, we have

f ( x + (1 )y) f (x) + (1 )f (y) , [0, 1]



The Class



28/31


pMathematical Formulation

Convexity: Illustrations

Figure: A convex (left) and a non-convex set (right) in R 2 .

Figure: A convex function.



The Class



29/31

The Class...Numerical Optimization Mathematical Formulation

Convexity: Notes

A function f is said to be concave if f is convex.

Optimization algorithms for unconstrained problems are usuallyguaranteed to converge to a stationary point (maximum,minimum, or inection point) of the objective f .

If f is convex, then the algorithm has converged to a globaloptimum.

Bottom line: Convexity simplies the problem.


The ProfessorThe Class OverviewThe Class


h l l


30/31


Summary

Easier HarderUnconstrained Constrained Continuous Discrete Local Optimization Global OptimizationDeterministic Stochastic Convex Non-Convex

Table: Summary of some factors impacting the difficulty of the optimization problem.

In this class we will mainly look at Local Optimization methods forDeterministic, Unconstrained, Continuous, Convex functions overConvex sets. Still, it will be a challenging semester!


The ProfessorThe Class OverviewThe Class...

The What? Why? and How?Concepts & TermsM h i l F l i


31/31


Algorithms

Optimization algorithms are iterative and generate a sequence of successively better estimates of the solution.Three key attributes characterize each (good) algorithm:

Robustness: Algorithm performance on a wide variety of problems

(of the same type), for a range of reasonable choices of initial values.

Efficiency: We prefer fast algorithms that do not require excessiveamounts of storage.

Accuracy: The algorithm should nd the solution without beingoverly sensitive to errors in the data or roundoff errors inthe computations.

These goals are often conicting hence careful consideration of trade-off between the goals is a key part of this course.


Documents

Lecture Static