Slide 1

Unconstrained optimizationGradient based algorithmsSteepest descentConjugate gradientsNewton and quasi-NewtonPopulation based algorithmsNelder Meads sequential simplex Stochastic algorithms

Unconstrained local minimizationThe necessity for one dimensional searches

The most intuitive choice of sk is the direction of steepest descent

This choice, however is very poorMethods are based on the dictum that all functions of interest are locally quadratic

Conjugate gradients

What are the unlabeled axes?Newton and quasi-Newton methodsNewton

Quasi-Newton methods use successive evaluations of gradients to obtain approximation to Hessian or its inverseEarliest was DFP, currently best known is BFGSLike conjugate gradients guaranteed to converge in n steps or less for a quadratic function.

Matlab fminfuncX=FMINUNC(FUN,X0,OPTIONS) minimizes with the default optimization parameters replaced by values in the structure OPTIONS, an argumentcreated with the OPTIMSET function. See OPTIMSET for details. Used options are Display, TolX, TolFun, DerivativeCheck, Diagnostics, FunValCheck GradObj, HessPattern, Hessian, HessMult, HessUpdate, InitialHessType, InitialHessMatrix, MaxFunEvals, MaxIter,DiffMinChange and DiffMaxChange, LargeScale, MaxPCGIter,PrecondBandWidth, TolPCG, TypicalX.

Rosenbrock Banana function.

Vanderplaatss version

My version

Matlab output [x,fval,exitflag,output] = fminunc(@banana,[-1.2, 1])Warning: Gradient must be provided for trust-region algorithm; using line-search algorithm instead.

Local minimum found.Optimization completed because the size of the gradient is less thanthe default value of the function tolerance.x =1.0000 1.0000fval =2.8336e-011exitflag =1output = iterations: 36, funcCount: 138algorithm: 'medium-scale: Quasi-Newton line searchHow would we reduce the number of iterations?

Sequential Simplex Method (section 4.2.1)In n dimensional space start with n+1 particles at vertices of a regular (e.g., equilateral) simplex.

Reflect worst point about c.g.

Read about expansion and contraction

Matlab commandsfunction [y]=banana(x)global z1global z2global ygglobal county=100*(x(2)-x(1)^2)^2+(1-x(1))^2;z1(count)=x(1);z2(count)=x(2);yg(count)=y;count=count+1; global z2>> global yg>> global z1>> global count>> count =1;>> options=optimset('MaxFunEvals',20)[x,fval] = fminsearch(@banana,[-1.2, 1],options) >> mat=[z1;z2;yg]

mat =

Columns 1 through 8

-1.200 -1.260 -1.200 -1.140 -1.080 -1.080 -1.020 -0.960 1.000 1.000 1.050 1.050 1.075 1.125 1.1875 1.150 24.20 39.64 20.05 10.81 5.16 4.498 6.244 9.058

Columns 9 through 16

-1.020 -1.020 -1.065 -1.125 -1.046 -1.031 -1.007 -1.013 1.125 1.175 1.100 1.100 1.119 1.094 1.078 1.113 4.796 5.892 4.381 7.259 4.245 4.218 4.441 4.813

fminsearch Banana function.

-1.26-1.24-1.22-1.2-1.18-1.16-1.14-1.12-1.1-1.08-1.0611.011.021.031.041.051.061.071.081.0939.624,2

20.0510.815.16

Next iteration

Completed search[x,fval,exitflag,output] = fminsearch(@banana,[-1.2, 1])

x =1.0000 1.0000fval =8.1777e-010

exitflag =1

output = iterations: 85 funcCount: 159 algorithm: 'Nelder-Mead simplex direct search

Why is the number of iterations large compared to function evaluations (36 and 138 for fminunc)?