Solving Optimization Problems with MATLAB · 93 Global Optimization Toolbox Solvers GlobalSearch, MultiStart – Well suited for smooth objective and constraints – Return the location

1© 2015 The MathWorks, Inc.

Solving Optimization Problems with

MATLAB

LOREN SHURE

2

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

3

Optimization Problems

Minimize Risk

Maximize Profits

Maximize Fuel Efficiency

4

Design Process

Initial

Design

Variables

System

Modify

Design

Variables

Optimal

DesignObjectives

met?

No

Yes

5

Why use Optimization?

Manually (trial-and-error or iteratively)

Initial

Guess

6


Automatically (using optimization techniques)

Initial

Guess

7


Finding better (optimal) designs

Faster Design Evaluations

Useful for trade-off analysis

Non-intuitive designs may be found

Antenna Design Using Genetic Algorithmhttp://ic.arc.nasa.gov/projects/esg/research/antenna.htm

http://ic.arc.nasa.gov/projects/esg/research/antenna.htm

8

Introduction




Global optimization

Topics

9

Curve Fitting Demo

Given some data:

Fit a curve of the form:

teccty 21

)(

t = [0 .3 .8 1.1 1.6 2.3];

y = [.82 .72 .63 .60 .55 .50];

0 0.5 1 1.5 2 2.50.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

t

y

10

How to solve?

Ecc

cey

eccty

t

t

2

1

21

1

)(

As a linear system of equations:

2

2min yEc

Ecy

c

Can’t solve this exactly

(6 eqns, 2 unknowns)

An optimization problem!

2

1

3.2

6.1

1.1

8.0

3.0

0

1

1

1

1

1

1

50.0

55.0

60.0

63.0

72.0

82.0

c

c

e

e

e

e

e

e

11

Introduction




Global optimization

Topics

12

Nonlinear Optimization

min𝑥

log 1 + 𝑥1 −4

3

2

+ 3 𝑥1 + 𝑥2 − 𝑥13 2

13

Nonlinear Optimization - Modeling Gantry Crane

Determine acceleration profile

that minimizes payload swing

s25s4

s20s1

s20s1

:sConstraint

2

1

21

f

p

p

ppf

t

t

t

ttt

14

Functions for analytical computations

Conveniently manage & document symbolic computations

in MuPAD Notebook

• Math notation, embedded text, graphics

• Share work as pdf

Perform exact computations using familiar MATLAB

syntax in MATLAB

Integrate with numeric computing – MATLAB,

Simulink and Simscape language

Perform Variable-precision arithmetic

Symbolic Math Toolbox

Integration Differentiation Solving equations Transforms

Simplification

15

Introduction




Global optimization

Topics

16

Mixed-Integer Programming

Many things exist in discrete amounts:

– Shares of stock

– Number of cars a factory produces

– Number of cows on a farm

Often have binary decisions:

– On/off

– Buy/don’t buy

Mixed-integer linear programming:

– Solve optimization problem while enforcing that certain

variables need to be integer

17

Continuous and integer variables

𝑥1 ∈ 0, 100 𝑥2 ∈ 1,2,3,4,5

Linear objective and constraints

min𝑥

−𝑥1 − 2𝑥2

𝑥1 + 4𝑥2 ≤ 20𝑥1 + 𝑥2 = 10

such that

Mixed-Integer Linear Programming

18

Traveling Salesman Problem

Problem

How to find the shortest path through a series of points?

Solution

Calculate distances between all combinations of points

Solve an optimization problem where variables correspond to trips

between two points

1

11

0

1

1

0

0

00

19

Introduction




Global optimization

Topics

20

Example Global Optimization Problems

Why does fmincon have a hard time finding the

function minimum?

0 5 10

-10

-5

0

5

10

x

Starting at 10

0 5 10

-10

-5

0

5

10

x

Starting at 8

0 5 10

-10

-5

0

5

10

x

Starting at 6

x s

in(x

) +

x c

os(2

x)

0 5 10

-10

-5

0

5

10

x

Starting at 3

0 5 10

-10

-5

0

5

10

x

Starting at 1

0 5 10

-10

-5

0

5

10

x

Starting at 0x s

in(x

) +

x c

os(2

x)

21


Why didn’t fminunc find the maximum efficiency?

Revolutions Per Minute, RPM

Manifold

Pre

ssure

Ratio

Peak VE Value = 0.96144

Start

End

1000 2000 3000 4000 5000 60000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

22


Why didn’t nonlinear regression find a good fit?

0 20 40 60 80 100 1200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

t

c

c=b1e-b

4t+b

2e-b

5t+b

3e-b

6t

23

Global Optimization

Goal:

Want to find the lowest/largest value of

the nonlinear function that has many local

minima/maxima

Problem:

Traditional solvers often return one of the

local minima (not the global)

Solution:

A solver that locates globally optimal

solutionsRastrigin’s Function

24

Global Optimization Solvers Covered Today

Multi Start

Global Search

Simulated Annealing

Pattern Search

Genetic Algorithm

25

MultiStart Demo – Nonlinear Regression

lsqcurvefit solution MultiStart solution

26

MULTISTART

27

What is MultiStart?

Run a local solver from

each set of start points

Option to filter starting

points based on feasibility

Supports parallel

computing

28

MultiStart Demo – Peaks Function

29

GLOBAL SEARCH

30

What is GlobalSearch?

Multistart heuristic algorithm

Calls fmincon from

multiple start points to try

and find a global minimum

Filters/removes non-

promising start points

31

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview Schematic Problem

Peaks function

Three minima

Green, z = -0.065

Red, z= -3.05

Blue, z = -6.55

x

y

32

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 0Run from specified x0

x

y

33

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1

3

6

0

00

4

0

-2

x

y

34

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

6

0

00

4

0

-2

x

y

35

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

36

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

37

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

38

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

39

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

40

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


6

Current penalty

threshold value : 4

x

y

41

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

42

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


Current penalty

threshold value : 4

-3

x

y

43

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2Expand basin of attraction if minimum already found

Current penalty threshold value : 2

-0.1

x

y Basins can overlap

44

GlobalSearch Demo – Peaks Function

45

SIMULATED ANNEALING

46

What is Simulated Annealing?

A probabilistic metaheuristic

approach based upon the

physical process of annealing

in metallurgy.

Controlled cooling of a metal

allows atoms to realign from

a random higher energy state

to an ordered crystalline

(globally) lower energy state

47

Simulated Annealing Overview – Iteration 1Run from specified x0

x

y

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

0.9

48

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1

3

x

y 0.9

Possible New Points:

Standard Normal N(0,1) * Temperature

Temperature = 1

49

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

11.01

1/)9.03(

Taccepte

P

50

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

0.3

51

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

0.3

52

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

0.3

53

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 0.75

0.3

-1.2

54

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N-1

3

x

y 0.9

Temperature = 0.1

0.3

-1.2

-3

55

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration NReannealing

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

56

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

27.01

1/))3(2(

Taccepte

P

57

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

58

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N+1

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

59

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N+1

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

60

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration …

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

-6.5

61

Simulated Annealing – Peaks Function

62

PATTERN SEARCH

(DIRECT SEARCH)

63

What is Pattern Search?

An approach that uses a

pattern of search directions

around the existing points

Expands/contracts around

the current point when a

solution is not found

Does not rely on gradients:

works on smooth and

nonsmooth problems

64

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 1Run from specified x0

x

y

3

65

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 1Apply pattern vector, poll new points for improvement

x

y

3

Mesh size = 1

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

0_*_ xvectorpatternsizemeshPnew

0]0,1[*1 x1.6

0.4

4.6

2.8

First poll successful

Complete Poll (not default)

66

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 2

x

y

3

Mesh size = 2

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

Complete Poll

67

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

3

Mesh size = 4

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

68

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

3

Mesh size = 4*0.5 = 2

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

69

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration NContinue expansion/contraction until convergence…

x

y

31.6

0.4

4.6

2.8

-4

0.3-2.8

-6.5

70

Pattern Search – Peaks Function

71

Pattern Search Climbs Mount Washington

72

PARTICLE SWARM

73

What is Particle Swarm Optimization?

A collection of particles

move throughout the region

Particles have velocity and

are affected by the other

particles in the swarm

Does not rely on gradients:

works on smooth and

nonsmooth problems

74

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration 1Initialize particle locations and velocities, evaluate all

locations

x

y

75

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NUpdate velocities for each particle

x

yPrevious

Velocity

Best Location

for this Particle

Best Location for

Neighbor Particles

76

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NUpdate velocities for each particle

x

y

New

Velocity

77

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NMove particles based on new velocities

x

y

78

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NContinue swarming until convergence

x

y

79

Particle Swarm – Peaks Function

80

GENETIC ALGORITHM

81

What is a Genetic Algorithm?

Uses concepts from

evolutionary biology

Start with an initial generation

of candidate solutions that are

tested against the objective

function

Subsequent generations

evolve from the 1st through

selection, crossover and

mutation

82

How Evolution Works – Binary Case

Selection

– Retain the best performing bit strings from one generation to the next.

Favor these for reproduction

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

Crossover

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

– child = [ 1 0 0 0 0 1 1 0 1 0 ]

Mutation

– parent = [ 1 0 1 0 0 1 1 0 0 0 ]

– child = [ 0 1 0 1 0 1 0 0 0 1 ]

83

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1Evaluate initial population

x

y

84

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1Select a few good solutions for reproduction

x

y

85

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2Generate new population and evaluate

x

y

86

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2

x

y

87

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

88

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

89

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration NContinue process until stopping criteria are met

x

y

Solution found

90

Genetic Algorithm – Peaks Function

91

Genetic Algorithm – Integer Constraints

Mixed Integer Optimization

s.t. some x are integers

Examples

Only certain sizes of components

available

Can only purchase whole shares of stock

)(min xfx

92

Application: Circuit Component Selection

Thermistor Circuit 6 components to size

Only certain sizes

available

Objective:

– Match Voltage vs.

Temperature curve

300 Ω

330 Ω

360 Ω

…

180k Ω

200k Ω

220k Ω

Thermistors:

Resistance varies

nonlinearly with

temperature

?

𝑅𝑇𝐻 =𝑅𝑇𝐻,𝑁𝑜𝑚

𝑒𝛽𝑇−𝑇𝑁𝑜𝑚𝑇∗𝑇𝑁𝑜𝑚

93

Global Optimization Toolbox Solvers

GlobalSearch, MultiStart

– Well suited for smooth objective and constraints

– Return the location of local and global minima

ga, simulannealbnd

– Many function evaluations to sample the search space

– Work on both smooth and nonsmooth problems

patternsearch

– Fewer function evaluations than ga, simulannealbnd

– Does not rely on gradient calculation like GlobalSearch and

MultiStart

– Works on both smooth and nonsmooth problems

94

Speeding-up with Parallel Computing

Global Optimization Algorithms that support

Parallel Computing:– ga: Members of population evaluated in parallel at

each iteration

– patternsearch: Poll points evaluated in parallel at

each iteration

– MultiStart: Start points evaluated in parallel

Optimization Algorithms that support Parallel

Computing:– fmincon: parallel evaluation of objective function for

finite differences

– fminimax, fgoalattain: same as fmincon

Parallel Computing can also be used in the

Objective Function– parfor

95

Speed up parallel applications

Take advantage of GPUs

Prototype code for your cluster

Parallel Computing Toolbox for the Desktop

Simulink, Blocksets,

and Other Toolboxes

Local

Desktop Computer

MATLAB

……

96

Scale Up to Clusters and Clouds

Cluster

Scheduler

Computer Cluster

…

…

…

…

…

…

… … …

Simulink, Blocksets,

and Other Toolboxes

Local

Desktop Computer

MATLAB

……

97

Learn More about Optimization with MATLAB

MATLAB Digest: Using Symbolic

Gradients for Optimization

Recorded webinar: Optimization

in MATLAB: An Introduction to

Quadratic Programming

Optimization Toolbox Web demo:

Finding an Optimal Path using MATLAB

and Optimization Toolbox

MATLAB Digest: Improving

Optimization Performance with

Parallel Computing

98

Key Takeaways

Solve a wide variety of optimization problems in MATLAB

– Linear and Nonlinear

– Continuous and mixed-integer

– Smooth and Nonsmooth

Find better solutions to multiple minima and non-smooth problems

using global optimization

Use symbolic math for setting up problems and automatically

calculating gradients

Using parallel computing to speed up optimization problems

99© 2015 The MathWorks, Inc.

Questions?

Documents

Solving Optimization Problems with MATLAB · 93 Global Optimization Toolbox Solvers GlobalSearch, MultiStart – Well suited for smooth objective and constraints – Return the location