SURVEY OF DYNAMIC PROGRAMMING SOFTWARES

IE-5306- Dynamic OptimizationFall 2015Project ReportDate: 12/1/15

Instructor:

Dr. Herbert Corley

Submitted by:Ashkan AliabadiUkesh ChawalNilabh Ohol

AbstractThis report surveys some of the commercially available softwares to solve dynamic programming

problems. Specific software for solving dynamic programming problem is not available and due to its

vast application and versatility of problems a number of softwares can be used to solve different type of

problems. Keeping in mind the scope of the course and in an attempt for users to be able to solve some

simple problems, an overview of such solvers has been put forth with the hope that using this knowledge

more complex problems can be tackled by the users. Different types of solvers require different skill set

which is also being highlighted in the report.

Introduction

Background:Dynamic Programming solves optimization problems by breaking a complex problem into a sequences of

sub-problems, each having decisions such that they can be solved in like fashion to obtain an optimal

solution for each sub problem such that the optimal solution for a the original problem can be found out

from the optimal solutions of the sub problems. This method is based on Bellman’s Principal of

optimality which can be phrased as follows:

“An optimal policy has the property that whatever the initial state and initial decision are, the remaining

decisions must constitute an optimal policy with regard to the state resulting from the first decision.”(Lew

and Mauch, 2007)

A large variety of problems can be solved using dynamic programming, both discrete and continuous.

Some of the popular problems include allocation problems, optimal stopping rule which are discrete

problems while continuous problems include linear programming, non-linear programming, minimax

problems. Dynamic programming is popular since it is a generic approach to solving optimization

problems. This report proposes the work to be done over coming weeks to survey various dynamic

optimization softwares. The motivation behind the survey, the methodology to complete the study and the

expected results are discussed in the following sections.

Motivation:Curse of dimensionality refers to phenomena that arises when analyzing or organizing data with high

dimensional spaces. This is a common problem while solving dynamic optimization problems since there

are a number of problems where the dimension of state variables is large. Thus in order to improve

computational simplicity, a computer program, that can handle such a problem, is necessary. Also, apart

from handling high dimension spaces, it is always advantageous to have a tool which can solve a dynamic

optimization problem. Software tool to solving dynamic programming is much more difficult to design in

part because the problem specification task in itself presents difficulties. Unlike linear programming

where there are specific solvers that handle the problem, dynamic programming problems do not have a

dedicated solver largely due to its generality. However there are certain solvers available that can handle

dynamic programming problems. Hence it is necessary to investigate such softwares and highlight their

advantages and disadvantages and their applications in dynamic programming world. Following sections

cover the problem definition, enlists different softwares and solves some popular dynamic programming

problems. The algorithm embedded in different solvers is also highlighted to show different approaches to

solving dynamic programming problems. Finally the learning objectives are summarized in the

conclusion section. The flow of the report closely follows the course structure.

Problem Definition:The aim of the study is to survey various dynamic programming softwares, compare them to highlight

their advantages and disadvantages along with their usefulness. As a part of the study numerous examples

will be solved to illustrate the working of these softwares.

Survey of Softwares

LINDO

An Overview:The name LINDO is an abbreviation of Linear INteractive Discrete Optimization. With the LINDO, users

can easily create their own optimization applications. It allows them to plug the power of the LINDO

solvers right into customized applications and mathematical programs that they have written.

Like almost all LP packages, LINDO solves real world linear programs using the simplex method. Along

with the solution to the problem, the program will also provide ordinary sensitivity analysis of the

Objective Function Coefficients (called Cost Coefficients) and the Right-hand-side (RHS) of the

constraints.

Key Benefits of the LINDO:

Fast, Easy Application Development

The LINDO makes it easy for you to seamlessly integrate optimization into your own application. The

developer interface has been designed for maximum ease-of-use and flexibility.

Powerful Solvers

LINDO provides you with an arsenal of powerful solvers for Stochastic, Linear, Nonlinear (convex &

nonconvex/Global), Quadratic, Quadratically Constrained, Second Order Cone and Integer optimization.

All solvers incorporate numerous enhancements for maximum speed and robustness.

Comprehensive Set of Routines

Whether your application is big or small, simple or complex, the LINDO API provides the flexibility and

functionality that you'll need. It includes dozens of routines to formulate, solve, query, and modify your

problems.

Extensive Documentation and Help

LINDO API provides all of the tools you will need to get up and running quickly. You get the LINDO

API User Manual that includes detailed function definitions for all routines. Also included in the manual

is a discussion to assist you in writing your own applications.

Analyze Infeasible and Unbounded Models

LINDO API includes tools that allow you to track down what has caused a model to be infeasible or

unbounded. The tools isolate a portion of the original model as the source of the problem. This allows you

to focus your attention on a relatively small subsection of the model to look for formulation or data entry

errors. On infeasible linear, nonlinear, quadratic and integer models, the tools can find an irreducibly

inconsistent set of constraints (IIS), and on unbounded models, the tools can find an irreducibly

unbounded set of columns (IUS).

Create Web and Intranet Applications

The LINDO API is thread safe to allow you to create web and network applications that handle multiple

user sessions concurrently. Web and network applications require special licensing.

Assignment ProblemQ) Consider a school with 5 teachers and 5 courses. The performance of a teacher i in charge of course j

is xij. There will be fixed unit of performance done by each teacher in each course. For example, 7 units of

performance is done by teacher 1 in course 1. Similarly, 8 units of performance is done by teacher 5 in

course 4 and so on.

course 1 course 2 course 3 course 4 course 5

Teacher 1 7 5 4 4 5

Teacher 2 7 9 7 9 4

Teacher 3 4 6 5 8 5

Teacher 4 5 4 5 7 4

Teacher 5 4 5 5 8 9

Chair has assign teachers to courses to maximize performance.

Traditional Dynamic Programming Solution Method

Stages: teachers

Decision variables: the course chosen

States: courses left

Returns: performance

max∑i=1

n

∑j=1

n

x ij

s.t

∑i=1

n

x ij=1∀ i=1, ……, n

∑j=1

n

x ij=1∀ j=1 , ……, n

x ij ≥ 0∀ i , j

Since it is pretty much impossible to solve it by hand (Traditional Dynamic Programming Solution

Method), we took help from Dynamic Programming Software, LINDO to solve this problem.

Formulating the Problems:

Max 7 x11+5 x12+4 x13+4 x14+5 x15+7 x 21+9 x22+7 x23+9x 24+4 x25+4 x 31+6 x32+5 x33+8 x 34+5 x35+5 x 41+4 x42+5 x 43+7 x 44+4 x 45+4 x51+5 x52+5 x53+8x 54+9 x 55

st

x11+ x12+x13+x 14+x 15=1

x21+x 22+x23+x 24+x 25=1

x31+x 32+ x33+x 34+x35=1

x 41+x 42+x 43+x 44+x 45=1

x51+x 52+ x53+x54+x55=1

x11+ x21+x31+x 41+x51=1

x12+x 22+ x32+x 42+x52=1

x13+x 23+x33+ x43+ x53=1

x14+x24+x34+x 44+x54=1

x15+x 25+x35+ x45+ x55=1

x ij ≥ 0∀ i , j

T1T4 T3 T2

d1

s4 s3 s2

r4 r3r2 r1

d4 d3 d2

s1

T5s5

d5

r5

Input in LINDO

Solution from LINDO

From the solution window, we can see the number of iterations needed to solve it is 19.

We can interpret that maximum performance is 38units.

We can also see x11 =1 meaning 7 units of performance done by teacher 1 in course 1. Similarly, x22 = 1

meaning 9 units of performance done by teacher 2 in course 2. x34 = 1 meaning 8 units of performance

done by teacher 3 in course 4. x43 =1 meaning 5 units of performance done by teacher 4 in course 3. x55 =

1 meaning 9 units of performance done by teacher 5 in course 5 which sums up to be 38 units.

Limitations

1) Non – Negativity Conditions – Imposed on all variables by default

2) Shadow Prices – Non-consistent

3) Optimal Solution to non-standard form problem – May need conversion

4) Sensitivity Ranges – May need conversion which is not an easy task.

The maximum sized problem LINGO can handle depends on the version you have. The current limits for

the various versions are:

Version

Total

Variables

Integer

Variables

Nonlinear

Variables

Global

Variables Constraints

Demo/Web 300 30 30 5 150

Solver Suite 500 50 50 5 250

Super 2,000 200 200 10 1,000

Hyper 8,000 800 800 20 4,000

Industrial 32,000 3,200 3,200 50 16,000

Extended Unlimited Unlimited Unlimited Unlimited Unlimited

TORA

AN OVERVIEW OF TORA

The Temporary-Ordered Routing Algorithm (TORA), An Operations Research Software is an algorithm

i.e. a mathematical set of instructions or programs. It is an optimization system in the area of operations

research which is very easy to use. Further, TORA is menu-driven and Windows-based which makes it

very user friendly.

TORA is invented by Vincent Park and M.Scott Corson from University of Maryland. The main objective

of TORA is to limit control message propagation in the highly dynamic mobile computing environment.

Each node has to explicitly initiate a query when it needs to send data to a particular destination.

TORA software deals with the following algorithms:

•Solution of simultaneous linear equations

•Linear programming

•Transportation model

•Integer programming

•Network models

•Project analysis by CPM/PERT

•Poisson queuing models

•Zero-sum games

Like LINDO, TORA also solves linear programming programs using the simplex method.

Linear Programming - Transportation Problem

Q) There are four factories (sources) and four markets (sinks). Each of the factories have certain

maximum capacity to supply. Each of the markets have certain maximum demand. For example, Factory

1 has a capacity of producing 250 units, Factory 2 has a capacity of 300 and so on. Market 1 has a

demand of 250 units, Market 2 has 200 units and so on. There will be fixed cost of transporting 1 unit of

material from each factory to each market. For example, there is a 5 dollar cost for transporting a material

from Factory 1 to Market 1. Similarly there is a 6 dollar cost for transporting from Factory 4 to Market 3

and so on.

Market 1 Market 2 Market 3 Market 4 CapacityFactory 1 $5 $4 $6 $3 250Factory 2 $7 $8 $8 $1 300Factory 3 $4 $4 $3 $6 100Factory 4 $8 $7 $6 $8 200Demand 250 200 200 200

Minimize the total transportation cost.

Traditional Dynamic Programming Solution Method

Stages: factory

Decision variables: quantity of material shipped from a factory i to market j.

States: quantity left

Returns: transportation cost

Since it is pretty much impossible to solve it by hand (Traditional Dynamic Programming Solution

Method), we took help from Dynamic Programming Software, TORA to solve this problem.

Input in Tora

F1F4 F3 F2

d1

s4 s3 s2

r4 r3r2 r1

d4 d3 d2

s1

Solutions from TORA

From the solution window, we can interpret that total minimum cost to transport the required quantities is

$ 3550.

We would sell 50 units of materials from factory 1 to market 1, 200 units of material from factory 1 to

market 2, 100 units of materials from factory 2 to market 1, 200 units of materials from factory 2 to

market 4, 100 units of materials from factory 3 to market 1 and 200 units of materials from factory 4 to

market 3.

Limitations: Not able to solve following DP Problems

1) Non-linear Programming Problem

2) Probabilistic Problem

3) Mini-max Problem

4) Optimal Stopping Rule Problem

Cross Checking the TORA answer with LINDO.

Formulating the Problems in LINDO:

Min5 x 11+4 x12+6 x13+3 x14+7 x21+8 x 22+8 x23+x 24+4 x31+4 x32+3 x33+6 x 34+8 x 41+7 x 42+6 x 43+8 x 44

st

x11+ x12+x13+x 14≤250

x21+x 22+x23+x 24≤300

x31+x 32+ x33+x 34≤100

x 41+x 42+x 43+x 44≤200

x11+ x21+x31+x 41=250

x12+x 22+ x32+x 42=200

x13+x 23+x33+ x43=200

x14+x24+x34+x 44=200

Input in LINDO

Solution from LINDO

From the solution window, we can see total minimum transportation cost is $ 3550. We would sell 50

units of materials from factory 1 to market 1, 200 units of material from factory 1 to market 2, 100 units

of materials from factory 2 to market 1, 200 units of materials from factory 2 to market 4, 100 units of

materials from factory 3 to market 1 and 200 units of materials from factory 4 to market 3.

The Answer from LINDO exactly match with that of TORA.

MATLABMATLAB is a multi-paradigm numerical environment and a fourth generation programming language

intended primarily for numerical computing. A proprietary programming language developed by

MathWorks, MATLAB allows matrix manipulation, plotting of functions and data, implementation of

algorithms. It shares a robust interface with programs written in other languages, including C, C++, Java,

FORTRAN and Python. The application is built around the MATLAB scripting language and involves

using a command window as an interactive mathematical shell or executing text files containing

MATLAB codes. The ability to create a separate function file which can be called from a main program

makes it a very strong tool for solving custom made dynamic optimization problems.

The optimization toolbox in MATLAB contains an exhaustive list of functions that handles a variety of

optimization problems. Problems such as linear programming problems, non-linear programming, convex

and non-convex, problems, stochastic programming problems, minmax and multi-objective problems can

be solved in MATLAB using with the help of the optimization tool box. A typical dynamic programming

(DP) problem uses a backward recursive algorithm which in not necessarily embedded in all the solvers.

A custom DP problem can be coded in a MATLAB script file without the use of the existing toolbox in

the software. An example of this is a travelling salesman problem. A function was developed to solve the

travelling sales man problem using dynamic programming. The function code is attached in appendix and

is developed based on the paper by Held and Karp from 1962. The DP is guaranteed to provide accurate

result to travelling salesman problem, but due to the time complexity is limited to at most 15 cities. The

output is also attached in the appendix. The objective of this function is to highlight the fact that a custom

program can be developed to solve a DP problem in the traditional way.

Minimax Problem

Another problem that MATLAB effectively handles is a minimax problem. A minimax problem can be

converted into a linear programming problem and solved in a traditional way. However MATLAB

function fminimax finds the minimum of a problem specified by:

minx

maxi

F i ( x ) such that {c (x ) ≤ 0

ceq ( x )=0A ∙x ≤b

Aeq ∙ x=beqlb≤ x≤ ub

where b and beq are vectors, A and Aeq are matrices, and c(x), ceq(x), and F(x) are functions that return

vectors. F(x), c(x), and ceq(x) can be nonlinear functions. x, lb, and ub can be passed as vectors or

matrices. A maximin problem can be solved using the identity,

maxx

mini

F i ( x )=−minx

maxi

(−F¿¿i¿ ( x ))¿¿

This is illustrated by a problem,

Find the values of x that minimizesthe maximum value of [ f 1 ( x ) , f 2 (x ) , f 3 ( x ) , f 4 ( x ) , f 5 ( x ) ]where ,

f 1 ( x )=2 x12+x2

2−48 x1−40 x2+304f 2 ( x )=−x12−3 x2

2f 3 ( x )=x1+3 x2−18f 4 ( x )=−x1−x2

f 5 ( x )=x1+x2−8subject ¿ , x12+x2

2≤ 12

First a function file is written that computes all the functions at x. The code for the function file is given

below:

function f = minimax(x)%objective functions are defined heref(1) = 2*x(1)^2+x(2)^2-48*x(1)-40*x(2)+304; %notice some of the functions are nonlinearf(2) = -x(1)^2-3*x(2)^2;f(3) = x(1)+3*x(2)-18;f(4) = -x(1)-x(2);f(5) = x(1)+x(2)-18;

Next step is to define the non-linear constraint in a different function. This is written as follows:

function [c,ceq] = constr(x)%non-linear constraints are defined herec(1) = x(1)^2 + x(2)^2 - 12;%non-linear inequality constraintceq = [];%no non-linear equality constraint present

Notice that the ceq, which is the non-linear inequality constraint, is defined as a null set since it is not

present in the problem. It is imperative to define all the constraints which are missing in the problem as

null sets for the function fminimax to work. The syntax is discussed later on.

After the two functions are defined a main program is written which calls out these two functions and

defines other parameters. This is given below:

%%Minimax main Program which calls the two previously defined functions%%clc;clear all;x0 = [0;0];%Starting point of the algorithmA=[];%No linear constrainsb=[];ub=[];%Upper bound is not definedlb=[0;0];%x1 and x1 >=0Aeq=[];%no linear equality constraints for the problembeq=[];[x,fval] = fminimax(@minimax,x0,A,b,Aeq,beq,lb,ub,@constr);%fminmax calculates the minimax for the objective functions

The syntax fminimax(@minimax,x0,A,b,Aeq,beq,lb,ub,@constr) minimizes the maximum value of te

functions. In this syntax, @minimax calls the function that computes all the functions at x. x0 is te

starting point from which the algorithm begins. In this case it is defined as [0;0] for x 1 and x2. It makes

sense because the lower bound lb is defined as 0, A and b is the left and right hand side of the linear

inequality constraint respectively. Since these constraints are not present in the problem, they are defined

as null sets. Similarly Aeq and beq which are the left and right hand side of the linear equality constraints

are defined as null sets. lb and ub is the lower and upper bound respectively. @constr calls the function

that defines the nonlinear constraints. The output parameter x returns the value of x1 and x2 which

minimizes the maximum value of the functions and fval gives the values of the five functions. The output

of the program is shown below:

Figure: MATLAB Output of the Minimax Problem

fminimax internally reformulates the minimax problem into an equivalent Nonlinear Linear Programming

problem by appending additional (reformulation) constraints of the form Fi(x) ≤ γ to the constraints given

in equation, and then minimizing γ over x. fminimax uses a sequential quadratic programming (SQP)

method to solve the problem discussed. Thus at x1 = 2.5345 and x2 = 2.3614 the maximum values of the

functions are minimized. These values are shown in the output. Although the algorithm used by fminimax

is different from the traditional DP algorithm, it gives the same solution in much less time. Also with the

increasing number of functions and variables, the number of stages in recursive DP method increases, use

of software is warranted.

Problem with Multi-Objective Functions (Pareto Optima)

There are many cases where solution of a multi-objective function is required. For example knapsack

problems where the objective can be to minimize weight volume along with a new objective say

dissatisfaction level. This then becomes a multi-objective problem. It is known that multi-objective

maximization and minimization problems can be solved using dynamic programming methods. However

with increasing number of variables it becomes necessary to use software techniques to solve such

problems. Solver such as Cplex and Couenne cannot easily handle multi objective problems. However

there is an inbuilt function in MATLAB which enables users to solve multi objective functions easily. An

example is illustrated to show how to perform multi-objective optimization using multi-objective genetic

algorithm function gamultiobj in the global optimization toolbox.

minimize vmin (2 x1−3 x2 , 4 x1+x22 )subject ¿ ,x1+ x2 ≤12x1 , x2≥ 0

The first step is to define the functions as was done for the minimax problem. The code is given below:

function f = vmin(x)%objective functions are defined heref(:,1) = 2*x(1)-3*x(2);f(:,2) = 4*x(1)+x(2).^2;%notice some of the functions are nonlinear

It can be seen that function f2 is non-linear. The next step is to define the constraints. It is not necessary to

define a new function for the constraints since all the constraints are linear in nature. These are defined in

the main program. The main program is given below:

%Program for Pareto Optima%clc;clear all;close all;numberofvariables = 2;lb = [0;0];%define lower boundsub=[];%undefined upper boundsA = [1 1];%Defining inequality linear constraintsb = [12];Aeq = [];%no equality linear constraints are presentbeq = [];options = gaoptimset('PlotFcns',{@gaplotpareto,@gaplotscorediversity});%plot the objective functions on the axis[x,fval] = gamultiobj(@vmin,numberofvariables,A,b,Aeq,beq,lb,ub,options);%Finding Pareto optimafprintf('The number of points on the Pareto front was: %d\n',size(x,1));

Defining constraints is similar in this case as was discussed earlier. The solver gamultiobj attempts to

create a set of Pareto optima for a multi-objective minimization. The definition of the solver is the same

as it calls the function where the function values are defined, along with the number of variables in the

problem, the constraints and the options option which creates a plot of the two objective functions. This

plot is shown below:

-40 -35 -30 -25 -20 -15 -10 -5 00

50

100

150

Objective 1

Obj

ectiv

e 2

Pareto front

-40 -20 0 20 40 60 80 100 120 140 1600

5

10

15

20Score Histogram

Score (range)

Num

ber o

f ind

ivid

uals

fun1 [-36.0019 -0.00132518]fun2 [3.94134e-05 144.019]

Figure: MATLAB plot of the Pareto front

The first figure shows the pareto front which shows the pareto optimal points with the value of their

respective objective functions. The second plot shows the score diversity for each objective. The output is

shown in below: All the 27 pareto optimal points and their respective function values are shown.

Figure: MATLAB output

Note on the Algorithm: gamultiobj uses a controlled elitist genetic algorithm. An elitist GA always favors

individuals with better fitness value (rank). A controlled elitist GA also favors individuals that can help

increase the diversity of the population even if they have a lower fitness value. It is important to maintain

the diversity of population for convergence to an optimal Pareto front. Diversity is maintained by

controlling the elite members of the population as the algorithm progresses. Two options, ParetoFraction

and DistanceFcn, control the elitism. ParetoFraction limits the number of individuals on the Pareto front

(elite members). The distance function, selected by DistanceFcn, helps to maintain diversity on a front by

favoring individuals that are relatively far away on the front. The algorithm stops if the spread, a measure

of the movement of the Pareto front, is small.

AMPLAMPL (A Mathematical Programming Language) is an algebraic modeling language for describing and

solving high complexity problems for large scale mathematical computation. AMPL supports lots of

solvers, both open source and commercial. It also supports the entire optimization modeling lifecycle i.e.

formulation, testing, development and maintenance. Problems in AMPL are comparatively easy to

formulate and since it supports a large number of solvers it becomes an important part of this study.

AMPL creates models and data for optimization and the solvers compute them to obtain optimal

solutions. Solvers such as CPLEX, Gurobi, Xpress handle linear and convex quadratic optimization

problems where the variables can be both continuous and integer. On the other hand solvers such as

CONOPT, KNITRO, MINOS, SNOPT provide most effective way of finding locally optimal solution to

problems involving smooth non-linear functions. Alternative solver such as BARON and Couenne can

also be interfaced with AMPL to solve optimization problems. CPLEX is a popular solver for linear

programming problems and uses a branch and bound algorithm to solve LP problems whereas Couenne is

extensively used for solving smooth non-linear programming problems. However in this report, MINOS

is used to solve smooth non-linear programming problem with continuous constraints. Since linear

programming problems were discussed earlier, an example problem using CPLEX is solved for a smooth

quadratic objective function. A traditional dynamic programming approach is solves both linear and non-

linear problems however with increasing number of variables, the computational complexity also

increases and thus use of software technique is warranted.

CPLEX

CPLEX follows a branch and bound algorithm and is very effective in solving high dimensional linear

programming problems. A simple linear programming problem is solved with seconds however with the

increasing number of variables (say in thousands) the computational time increases. This is usually

tackled by setting a time bound on the solver. In doing so, under worst case, the solver runs as much

iteration as possible within the specified time frame to return the best possible feasible solution. With new

addition to the solver, CPLEX can also handle quadratic objective functions. An example is shown below

to illustrate this:

:Portfolio Selection: An investor has $5000 and two potential investments. Let x j for j = 1 and j = 2 denote his

allocation to investment j in thousands of dollars. From historical data, investments 1 and 2 have an expected

annual return of 20 and 16 percent, respectively. Also, the total risk involved with investments 1 and 2, as measured

by the variance of total return, is given by, 2 x12+x2

2+(x¿¿1+x2)2 ¿, so that risk increases with total investment and

with the amount of each individual investment. The investor would like to maximize his expected return and at the

same time minimize his risk. Clearly, both of these objectives cannot, in general, be satisfied simultaneously. There

are several possible approaches. For example, he can minimize risk subject to a constraint imposing a lower bound

on expected return. Alternatively, expected return and risk can be combined in an objective function, to give the

model:

Maximize f ( x )=20 x1+16 x2−[2 x12+x2

2+( x1+x2 )3 ]subject ¿ ,x1+ x2 ≤5x1 , x2≥ 0

Such a problem obtains a quadratic form and can be modeled in AMPL and solved using CPLEX.

The code for modeling the above problem is shown below:

#Define Variablesvar x1;var x2;

#Define Objective Functionmaximize z: 20*x1+16*x2-2*x1^2-x2^2-(x1+x2)^2;

#Define Constraintssubject to c1: x1+x2<=5;subject to c2: x1>=0;subject to c3: x2>=0;

The output is given below:

ampl: model nlp.mod;ampl: option solver cplex;ampl: solve;CPLEX 12.6.1.0: optimal solution; objective 46.3333332117 QP barrier iterationsNo basis.ampl: display x1;x1 = 2.33333

ampl: display x2;x2 = 2.66667

ampl:

The model file is stored as a .mod file which is called in the AMPL workspace. The solver is then

selected to run the model file. The solution obtained is the maximum value of the objective function for

the given constraints and can be seen from the output above.

Now if a cubic term is added in the objective function and the code is run using CPLEX solver, it can be

seen that CPLEX returns an error warning.

ampl: model nlp.mod;ampl: option solver cplex;ampl: solve;CPLEX 12.6.1.0: C:\Users\IE-FAC~1\AppData\Local\Temp\at3936.nl contains a nonlinear objective.

ampl:

This shows that CPLEX does not handle non-linear problems. MINOS is used to solve this problem. The

problem definition remains unchanged only difference is that while running the code in AMPL

workspace, MINOS is selected chosen instead of CPLEX.

ampl: model nlp.mod;ampl: option solver minos;ampl: solve;MINOS 5.51: optimal solution found.5 iterations, objective 25.59249598Nonlin evals: obj = 11, grad = 10.ampl: display x1;x1 = 1.39724

ampl: display x2;x2 = 0.794486

ampl:

The results can be seen from the output and are pretty straightforward to interpret. These solvers provide

a considerable amount of advantage when the number of variables is very large. Since a nonlinear

programming problem with a high number of variables increases computational complexity, if solved

using traditional dynamic programming method.

DP SolverDeveloped by Paul A. Jensen, a professors at Mechanical engineering department at UT Austin. The link

to his all OR software is https://www.me.utexas.edu/~jensen/ORMM/

This DP add-in provides a general structure with which most problems appropriate for DP can be

modeled and solved. The add-ins in this package are programmed using the Microsoft macro language,

Visual Basic for Applications (VBA). The source code is available for viewing and changing when the

add-in is installed in Excel. Some of the add-ins are not password protected. These include the Teach DP

and Stochastic Models add-ins. They include specific model classes that are provided through VBA

subroutines. They are left open so the user can add new problem classes.

Limitation: 2/20/2011: The MP Model Builder add-in, Jensen LP/IP solver add-in, and Math

Programming add-in were modified to allow more than 32000 nonzero coefficients in the LP constraint

matrix. The Jensen Network Solver add-in was modified to have more than 32000 arcs.

The Jensen Software Add-in covers a variety of optimization problem:

Linear Programming Nonlinear Programming

Figure 1: DP collection Add-in

Integer Programming Mixed Integer Programming Network Flow Programming Transportation Model Stochastic programming Dynamic programming Combinatorics Queueing Morkov Decision process Simulation

DP Collection Add-in:

The DP Data add-in

The DP Data add-in constructs tables that describe data for instances of specific application areas of

dynamic programming or Markov analysis. The DP Data add-in calls the DP Models add-in to construct

the model form to hold the decisions and stochastic features of the model. The DP Data add-in then fills

the model form with parameters linked to the data form. The resulting model will ultimately be used to

provide data for the DP Solver or Markov Analysis add-ins.

Although the model forms available in the DP Models add-in are very general, it is somewhat difficult for

the unskilled user of Excel to build a model. The DP Data add-in removes this difficulty by

automatically building the model for specific problem classes. At this time only five classes are

available: birth-death, finite queue, random walk, inventory and resource. For each class there are

several variations.

Addition of new problem classes requires VBA programming. The programs of this add-in are not

password protected so you can examine the programs for existing classes. As a user of this add-in, you

may find it interesting to add new classes.

The Start and Finish commands are used to respectively add and remove buttons from the worksheets.

This is important when moving models from one computer to another. Always delete the buttons with the

Finish command before attempting to open a model on another computer. Otherwise you will experience

an error message regarding links. Add buttons back with the Start command. Although the Finish

command deletes all buttons from the workbook, the Start command only adds buttons for the add-in

from which the command was called.

The picture shows the ORMM menu when the DP add-in collection is

installed. Each add-in has an individual role. The DP Data add-in accepts

data concerning general situations and calls the DP Models add-in to

construct models. The DP Models add-in calls the DP Solver add-in to

obtain numerical solutions.

The DP Models add-in also models Markov Chain problems. These are

described by states and events alone. No actions are involved. Models

constructed can either be analyzed with the DP Solver add-in or the

Markov Analysis add-in.

Markov Decision Processes are subsumed under this discussion as they

are similar to stochastic dynamic programming models.

Here, we provides a brief introductions to the three add-ins:

Although the add-ins do not restrict the size of the problem entered, some computational methods are

impractical for large problems. The size of the worksheet in terms of rows and columns limits some

models. Some computations in the DP Solver add-in use the Excel INVERSE function that is

limited to about 50 stages, although the limit seems to be greater for Excel 2007. The size of a model, as

measured by the number of stages, decisions, and transitions, is an important feature of a DP model

because for many problems these numbers may be very large. In the parlance of dynamic programming

this is the curse of dimensionality.

Dynamic Programming Models

The DP Models add-in provides a structure to abstractly define several kinds of models. The first step is

to choose the kind of model. Four model types are allowed.

add-ins consume part of the memory allotted to Excel. Add-ins also increase the time required for Excel

to launch. If you experience trouble, a possible remedy is to decrease the number of add-ins installed.

This section describes a general model form constructed on an Excel workbook that seems to be useful

for a variety of problems that are related to DP. It is too early to tell if the structure is sufficient for all

problems that involve stages, States and events, but it has been used for several problems that have

appeared in the literature.

Stages Stages Event

s

Stage 

Subsets

Decisions Transitions

Markov Chain X X Optional X

Deterministic DP X X Optional Optional X

Stochastic DP

or Markov Decision

Process

X X X Optional Optional X

States and transitions are required for every model type. Stages and decisions are added for optimization

problems. Events are added for problems involving probabilities. The optional elements are useful for

restricting combinations that are infeasible. For example the State Subsets element is used to identify

subsets of stages that have different characteristics. Most deterministic DP models start at some Initial

state and finish at one or more Final stages. The State Subset element is used to define the stages where

the process terminates.

The example of allocation problem With DP Solver

The DDP model of the resource problem has only states and actions. No events are required because there

is no uncertainty in this model. We use the four resource model as an illustration.

This is the linear version of the problem that for most instances is easily solved by math programming

solution techniques. The add-in creates a DP version of the problem. It can handle separable nonlinear as

well as linear models. Here we consider the Resource problem with a deterministic DP model. When this

page is written, only the DDP model is available.

•

•

• The facility has four machines of type 1, five of type 2, three of type 3 and seven of type 4. Each machine operates 40 hours per week. The problem is to determine the optimum weekly production quantities for the products. The goal is to maximize total profit.

• Maximize Z = 18X1+ 25X2 + 10X3 + 12X4 + 15X5

• Solution: X1 = 59, X2 = 62, X3 = 0, X4 = 10

Machine Quantity Product 1 Product 2 Product 3 Product Product 5M1 160 h 1.2 1.3 0.7 0.0 0.5M2 200 h 0.7 2.2 1.6 0.5 1.0M3 120 h 0.9 0.7 1.3 1.0 0.8M4 280 h 1.4 2.8 0.5 1.2 0.6Unit profit, $ —— 18 25 10 12 15

Fill the name field with a short name with no spaces or punctuation. The name, number of items, and

number of resources are fixed once the worksheet is created. The maximum of each item can be changed

on the data worksheet.

By checking the Make Random Problem box, the model parameters are assigned randomly. The

Maximize Objective button determines the direction of optimization.

• Clicking the Build Data Table button constructs tables for the revenue and resource use data. This allows the representation of nonlinear forms that calls the add-in to make tables for Revenue and resource use as a function of the number of items selected.

Clicking the Build Model button on the top of the data worksheet, calls the DP Models add-in to construct

the model worksheet. The Build/Modify Model button also constructs the model worksheet, but first

presents the Model dialog.

Z = 18X1+ 25X2 + 10X3 + 12X4 + 15X5

Suppose Z= 18 x12 then the row for item one should be

changed to 18, 72, 162, and so on.

The solver cannot solve a problem and return error message. Since The enumeration process finds all

combinations of five state variables. 6*161*201*121*281 = 6,601,838,166, and it is beyond the capacity

of excel to handle it, then the software cannot solve a problem.

DP2PN2Solver

The solver software DP2PN2Solver presented in this section is a general, flexible, and expandable

software tool that solves DP problems. It consists of modules on two levels. A level one module takes the

specification of a discrete DP problem instance as input and produces an intermediate Petri net (PN)

representation called Bellman net (Lew, 2002; Lew, Mauch, 2003, 2004) as output — a middle layer,

which concisely captures all the essential elements of a DP problem in a standardized and mathematically

precise fashion. The optimal solution for the problem instance is computed by an “executable” code (e.g.

Java, Spreadsheet, etc.) derived by a level two module from the Bellman net representation.

DP2PN2Solver’s unique potential lies in its Bellman net representation. In theory, a PN’s intrinsic

concurrency allows to distribute the computational load encountered when solving a single DP problem

instance to several computational units. It will tell you how and where to deploy the files to your

computer and how to invoke the two major modules DP2PN and PN2Solver.

A Petri net is a directed bipartite graph, in which the nodes represent transitions (i.e. events that may

occur, signified by bars) and places (i.e. conditions, signified by circles). The directed arcs describe which

places are pre- and/or postconditions for which transitions (signified by arrows).

Disadvantage of current software system reviewed by Mauch

Attempts to develop general-purpose DP computer codes have been made in the early 70’s by Hastings

(1974) and others. While Hasting’s “DYNACODE” is frequently referenced (e.g. in Hastings, 1973),

detailed information about this system is unavailable as of today. Sniedovich characterizes these codes as

problem specific and concludes that “no general-purpose dynamic programming computer codes seem to

be available commercially” (Sniedovich, 1992, p. 193). Other, non-general DP codes have been

developed in Hubert, Arabie, Meulman (2001), Lin and Bricker (1990), Sniedovich (1993, 1994). DP

software accompanying textbooks such as Hillier and Lieberman (1990) does not generalize well either.

The shortcoming of these attempts is that they are usually restricted to solve instances of one special

problem. Mailund’s DPROG system at www.daimi.au.dk/~mailund assumes that the user has a

substantial amount of expertise in software engineering practice. As of today it seems that major parts of

the system are still under construction.

The optimization package system LINGO / LINDO solves optimization problems across a wide problem

domain with a variety of solution methods. While its strength lies in the areas of linear programming, and

to some extent in integer programming and nonlinear programming, it is only of very limited use to tackle

DP problems. In particular, it is not obvious at all how to retrieve the series of optimal decisions that lead

to the optimal solution. A disadvantage of this and many other optimization software systems is that they

incorporate search mechanisms. This can in the best case be very efficient, but in the worst case not lead

to the optimal solution at all. The most common approach taken today for solving real-world DP

problems

is to start a specialized software development project for each and every problem. Even taking into

account that a module-oriented development philosophy allows considerable reuse of components, this is

still a rather expensive approach. A general solver solution like the DP2PN2Solver introduced here

should be able to save software development costs for many DP problems arising in practice.

gDPS Language

DP problems can can take on various forms and shapes. This is why it is hard to specify a strict and fixed

specification format like for linear programming (LP). But there are common themes across all DP

problems and those are captured in the general DP specification (gDPS) language. A special purpose

language should allow for both a convenient description of a DP problem instance and efficient parsing of

the specification. The gDPS language has been designed as a general source language that can describe a

variety of DP problems. It offers the flexibility needed for the various types of DP problems arise in

reality.

Advantage of using DP2PN2SOLVER

• Automatically solve many optimization problems of practical relevance with DP• The DP2PN2Solver user no longer needs deep skills in software development to solve a problem• It is no longer necessary to worry about the DP solution process itself• DP2PN2Solver guarantees that the optimal solution to a DP instance will be found, provided of

course that the DP functional equation is correct and its solution does exist.• The gDPS language is easy to learn.

Solving Traveling salesman with DP2PN2Solver

For instance let,

The procedure in getting the solution is the following:

1. Invocation of DP2PN2. Consequences of Invocation: Desired Bellman net3. Invocation of PN2Solver4. Consequences of Invocation:

1. Spreadsheet solver: update the cell and get new solution2. Java Solver3. Solution Tree in text

The solution tree of TSP:

• The optimal value is: 39.0

• The solution tree is:

State (0,{0}) has optimal value: 39.0 Decision d=1 State (1,{0,1}) has optimal value: 38.0 Decision d=3

State (3,{0,1,3}) has optimal value: 35.0 Decision d=4 State (4,{0,1,3,4}) has optimal value: 20.0

Decision d=2 Base state (2,{0,1,2,3,4}) has initial value: 7.0

ConclusionThis report surveyed softwares that could be used to solve dynamic programming problems. It can be

seen from the report that even though there does not exist a specific software to handle dynamic

programming problems, a variety of open source and commercially available softwares can solve a range

of DP problems. Some of the softwares highlighted in this report are, LINDO, TORA, MATLAB, APML,

Excel and DP2PN2 solver. Although not all of the solvers use a backward recursion approach, the

algorithms embedded in the solvers are a good fit to the problem definitions. A lot of research is going on

in this area where scripting languages are used to create toolboxes for such softwares that would handle

more complex problems.

Referenceshttp://home.ubalt.edu/ntsbarsh/Business-stat/opre/LindoCitrix.pdf

https://www.youtube.com/watch?v=qjM_5RP-eJw

http://taha.ineg.uark.edu/tora.html

http://www.lindo.com/index.php?option=com_content&view=article&id=1&Itemid=9

http://www.ampl.com/BOOKLETS/ampl-osl.pdf

http://www.ampl.com/BOOKLETS/ampl-cplex3.pdf

http://ampl.com/products/solvers/solvers-we-sell/minos/

http://www.math.clemson.edu/~mjs/courses/mthsc.440/integer

http://ampl.com/BOOK/CHAPTERS/21-nonlinear.pdf

http://ampl.com/products/solvers/open-source/

https://www.tu-chemnitz.de/mathematik/part_dgl/teaching/WS2009_Grundlagen_der_Optimierung/

amplguide.pdf

http://www.mathworks.com/help/gads/examples/performing-a-multiobjective-optimization-using-the-

genetic-algorithm.html

http://www.mathworks.com/help/gads/gamultiobj.html

http://www.mathworks.com/help/gads/using-gamultiobj.html

http://www.mathworks.com/help/optim/ug/constrained-nonlinear-optimization-algorithms.html#f26684

AppendixTravelling Salesman Problem

%=======Traveling Salesman Problem Dynamic Programming Function ===========%= This function solves the Traveling Salesman Problem (TSP) using Dynamic=%= programming (DP). =%= The function is based on the paper by Held and Karp from 1962. The DP =%= is guaranteed to provide the accurate (optimal) result to the TSP, but =%= the time complexity of this algorithm is O(2^n n^2), which limits the =%= the use of this algorithm to 15 cities or less. =%= NOTE: For reasonable runtime, please do not try to calculate a tour of =% more than 13 cities. DP is not for large sets of cities. =%= Inputs: =% Cities: an n-by-2 (or n-by-3) matrix of city locations. If this =% input is not given, a set of 10 cities will be randomly =% generated. =% Dmatrix: This is an n-by-n matrix of the distances between the =% cities. If not given, this matrix will be calculated at =% initialization. =%= Outputs: =% OptimalTour: a vector with indices of the cities, according to the=% the optimal order of cities to visit. It is a closed tour =% that will return to city 1, which is assumed to be the =% salesman's origin (depot). =%= mincost: the total length of the optimal route. =%= Developed by: Elad Kivelevitch =%= Version: 1.0 =%= Date: 15 May 2011 =%========================================================================== function [OptimalTour,mincost]=tsp_dp1(cities, Dmatrix)% Initialization Processif nargin==0 cities=random('unif',0,100,10,2);end[NumOfCities,dummy]=size(cities);Primes=primes(NumOfCities*10); if nargin<2 %No Dmatrix used D=diag(inf*ones(1,NumOfCities)); %Assign an inifinite cost to traveling from a city to itself for i=1:NumOfCities %Assign the distances between pairs of cities for j=i+1:NumOfCities D(i,j)=norm(cities(i,:)-cities(j,:)); D(j,i)=D(i,j); end endelse D=Dmatrix;endNumOfDataSets=1;for i=2:NumOfCities NumOfDataSets=NumOfDataSets+nchoosek(NumOfCities,i);

endData(NumOfDataSets).S=[];Data(NumOfDataSets).l=0;Data(NumOfDataSets).cost=inf;Data(NumOfDataSets).pre=[];Data(NumOfDataSets).m=[];LookUpTable(NumOfDataSets)=0;%Define a data structure that holds the following pieces of data we need%for later. This data structure uses the same notation used in the paper %by Held and Karp (1962):% S - the set of cities in the tour.% l - the last city visited in the set S. % cost - the cost of a tour, which includes all city in S and ends at l.%In addition, the following data items are used in the dataset for reducing%runtime:% Pre - the index of predecessor dataset, i.e. the one with Spre=S-{l}% m - the city in S-{l} that yielded the lowest cost C(Spre,m)+D(m,l).% This index will facilitate the generation of the optimal tour without% further calculations.Data(1).S=[1];Data(1).l=1;Data(1).cost=0;Data(1).Pre=[];Data(1).m=[];for s=2:NumOfCities Data(s).S=[Data(1).S,s]; Data(s).l=s; Data(s).cost=D(s,1); Data(s).Pre=1; Data(s).m=1; LUT=calcLUT(Data(s).S,s,Primes); LookUpTable(s)=LUT;endIndexStartPrevStep=2; %index into Data that marks the beginning of the previous stepIndexLastStep=NumOfCities; %index into Data that marks the end of the previous stepCurrentData=IndexLastStep; %index into Data that marks the current dataset %This is the main Dynamic Programming loopfor s=3:NumOfCities %generate possible sets with s-1 cities out of the possible N-1 cities %(not including city 1) TempSets=nchoosek(2:NumOfCities,s-1); NumOfSets=size(TempSets); for j=1:NumOfSets(1) %iterate through all the sets for k=1:NumOfSets(2) %iterate through all the elements in each set SminuskSet=[1,TempSets(j,1:k-1),TempSets(j,k+1:NumOfSets(2))]; %this is the set S-{k} candidatecost(2:length(SminuskSet))=inf; indices=[]; for mm=2:length(SminuskSet) %choose a city in S-{k} that will be last LUV=calcLUT(SminuskSet,SminuskSet(mm),Primes); index=find(LUV==LookUpTable(IndexStartPrevStep:IndexLastStep));

index=index+IndexStartPrevStep-1; if index==0 candidatecost(mm)=inf; else candidatecost(mm)=Data(index).cost+D(SminuskSet(mm),TempSets(j,k)); indices(mm)=index; end end [mincost,indexcost]=min(candidatecost(2:end)); CurrentData=CurrentData+1; Data(CurrentData).S=[1,TempSets(j,:)]; Data(CurrentData).l=TempSets(j,k); Data(CurrentData).cost=mincost; Data(CurrentData).Pre=indices(indexcost+1); Data(CurrentData).m=SminuskSet(indexcost+1); LookUpTable(CurrentData)=calcLUT(Data(CurrentData).S,TempSets(j,k),Primes); end end IndexStartPrevStep=IndexLastStep+1; IndexLastStep=CurrentData; endmm=0;%Now add the distance back from the last city to city 1for i=IndexStartPrevStep:IndexLastStep mm=mm+1; candidatecost(mm)=Data(i).cost+D(Data(i).l,1);end%Find the one that minimizes the total distance[mincost,indexcost]=min(candidatecost);Temp=Data(IndexStartPrevStep+indexcost-1);%Generate the optimal tour by traversing back from the last city to its%predecessorsOptimalTour=1;while ~isempty(Temp.Pre) OptimalTour=[OptimalTour,Temp.l]; Temp=Data(Temp.Pre);endOptimalTour=[OptimalTour,1]; function LUT=calcLUT(vec,last,Primes)j=length(vec);LUT=Primes(last);for i=2:j LUT=LUT*Primes(vec(i));end

Output