Practical Guidelines for Solving Difficult Mixed Integer Programs

© 2012 IBM Corporation

®

IBM ILOG Optimization Virtual Users Group SessionAugust 28, 2013

Practical Guidelines for Solving Difficult Mixed Integer Programs

2

IBM Software Group


Welcome to our August 2013 Virtual Users’ Group

� Welcome back to Past Attendees and Welcome to New Clients!

– Please continue to check out our blog for new information regarding these sessions as well as other interesting articles

https://www.ibm.com/developerworks/mydeveloperworks/blogs/sca/?lang=en_us

� Please write your questions to the Q&A Group via SmartCloud Meeting

– As we are expecting a lot of participants in today’s session, please send your questions via SmartCloud Meeting Chat:

• This is the conversation balloon tab in the right hand panel of the meeting center

• Address the question to the Q&A Group option in the drop down box

– If you have further questions after the meeting please contact Kitte Knight at [email protected]

� Our goal is to present topics of common interest in Optimization and create an open forum for our clients to exchange information between us and other clients

� If you should encounter any problems during the web conference please contact LOTUS LIVE HELP NUMBER: 1-888-376-0105

2

3

IBM Software Group


Host: Aimee-Emery Ortiz

Product Marketing Manager –

Optimization Solutions

IBM Software Group

Speaker: Ed Klotz

Mathematical Programming

Specialist, IBM ILOG CPLEX, IBM

Software Group

@ibmoptimization


®

Practical Guidelines for Solving Difficult Mixed Integer Programs

Ed Klotz, Ph.D.

IBM

[email protected]

5

IBM Software Group


Objective

� CPLEX solves most MIPs to optimality (including many

considered unsolvable only a few years ago) effectively

with default settings

– With recent advances in MIP heuristics, CPLEX finds good solutions to many of the MIPs it cannot solve to optimality

� For the remaining difficult MIPs, parameter tuning may

suffice to solve them

6

IBM Software Group


Objective

� If not, finding good or optimal solutions may be a time

consuming process

–– Finding good cuts that strengthen the formulationFinding good cuts that strengthen the formulation

• Simple, straightforward guidelines help derive these cuts or determine which other approaches are suitable

– Column Generation or other decomposition methods

– Basis Reduction

– Other Reformulations

– Constraint Programming

7

IBM Software Group


Overview

� Objective

� Brief Review of Branch and Bound

� Identifying Weak Formulations

� Tuning CPLEX Parameters

� Guidelines for Tightening Difficult MIPs

� Example

� Methods for Adding Cuts to a Model

� Sanity checking/debugging

� Summary

8

IBM Software Group


A mixed integer (quadratic) program (MIP/MIQP) is an optimization problem of the form

integer allor some jx

uxl

bAxtoSubject

QxxxcMinimizeTT

≤≤

=

+

Review of Branch and Bound

9

IBM Software Group


Root;

v=3.5

x=2.3

Integer y=0.6

z=0.3

v ≤≤≤≤ 3v > 4

x ≤≤≤≤

2 x >3

y ≤≤≤≤

0 y >1

z ≤≤≤≤

0 z >1

Lower Bound

Integer

Upper Bound

Infeas

z=0.1

G

A

P


Fathomed

Branch and Bound for MIP

z ≤≤≤≤

0z >

1

10

IBM Software Group



� Key aspects of Branch and Bound

– Branching divides the feasible region in a manner that all integer

feasible solutions belong in one of the branches

• Standard B&B uses up and down branch on an integer variable

• But any number of branches on any object is possible� Special ordered sets, indicator constraints, hyperplane branching

– Child node objective (of node MIP or LP) can be no better than the

parent node objective

• Enables pruning

– At any point in the tree, a better solution obtained via branching (as

opposed to node heuristics) can only come from children of the

active node list

• Explains why the lower bound is the active node with minimal relaxation objective value

11

IBM Software Group



� Progress of the algorithm depends on:

– Ability to find integer feasible solutions

• # of integer infeasibilities at each node

– Ability to prune nodes

• Objective value of best integer feasible solution

– Ability to move lower bound

• # of other node relaxations with same objective value

• # of active nodes remaining

– Strength of the model formulation

– Node throughput

• Node relaxation solve times

• Cut, heuristic computation times

12

IBM Software Group


Nodes Cuts/

Node Left Objective IInf Best Integer Best Node ItCnt Gap

...

300 229 22.6667 40 31.0000 22.0000 4433 29.03%

400 309 cutoff 31.0000 22.3333 5196 27.96%

500 387 26.5000 31 31.0000 22.6667 6164 26.88%

...

7800 5260 28.5000 23 31.0000 25.6667 55739 17.20%

7900 5324 28.2500 26 31.0000 25.6667 56424 17.20%

8000 5385 27.3750 30 31.0000 25.7778 57267 16.85%


� Optimizer Node Log shows algorithm progress

– Here we have progress in best node but not best integer

Node pruning Feasible solns

Strength /

lower boundNode throughput

13

IBM Software Group



� Optimizer Node Log shows algorithm progress

– Progress in best integer but not best node:

Nodes Cuts/

Node Left Objective IInf Best Integer Best Node ItCnt Gap

…

300 296 2018.0000 27 3780.0000 560.0000 3703 85.19%

* 300+ 296 0 2626.0000 560.0000 3703 78.67%

* 393 368 0 2590.0000 560.0000 4405 78.38%

400 372 560.0000 291 2590.0000 560.0000 4553 78.38%

500 472 810.0000 175 2590.0000 560.0000 5747 78.38%

...

* 7740+ 5183 0 1710.0000 560.0000 66026 67.25%

7800 5240 1544.0000 110 1710.0000 560.0000 66279 67.25%

7900 5325 944.0000 176 1710.0000 560.0000 66801 67.25%

8000 5424 1468.0000 93 1710.0000 560.0000 67732 67.25%

Node pruning Feasible solns

Strength /

lower boundNode throughput

14

IBM Software Group


Identifying Weak Formulations � Convex hull of the integer feasible solutions provides the strongest

formulation

� Add valid cuts to strengthen formulation

P1

P2

}:{P2P3

}0,:{P2

})0,:({P1

dCxRx

xbAxRx

xbAxZxconv

n

n

n

≤∈∩=

≥≤∈=

≥≤∈=

P3

22 dxcT ≤

33 dxcT ≤

n)(separatio :2 2)

(validity) 1 1)

satisfymust Cuts

i

T

i

i

T

i

dxcPx

Pxdxc

>∈∃

∈∀≤

11 dxcT ≤

15

IBM Software Group


Identifying Weak Formulations (example)

� A difficult knapsack problem

Minimize

COST:

Subject To

c1: 12228 x1 + 36679 x2 + 36682 x3 + 48908 x4

+ 61139 x5 + 73365 x6 = 89716837

Bounds

x1 >= 0

…

x6 >= 0

Generals

x1 x2 x3 x4 x5 x6

End

16

IBM Software Group


Identifying Weak Formulations (example)

� LP solutions are trivial to find (even after many branches on the

variables), but MIP solutions may not be.

c1: 12228 x1 + 36679 x2 + 36682 x3 + 48908 x4 +

61139 x5 + 73365 x6 = 89716837

Domain of x1: [0, 89716837/12228]

Domain of x2: [0, (89716837 – 12228 x1)/36679]

…

Domain of x6: [0, (89716837 – (12228 x1 + 36679 x2

+ 36682 x3 + 48908 x4

+ 61139 x5)/73365]

17

IBM Software Group


Identifying Weak Formulations (example) c1:12228 x1 + 36679 x2 + 36682 x3 +

48908 x4 + 61139 x5 + 73365 x6 = 89716837

MIP:

89716837

73365

61139

48908

36682

36679

12228

36682

12228

48908

61139

73365

36679

18

IBM Software Group


Identifying Weak Formulations (example) c1:12228 x1 + 36679 x2 + 36682 x3 +

48908 x4 + 61139 x5 + 73365 x6 = 89716837

LP:

89716837

73365

61139

48908

36682

36679

12228

36682

12228

48908

61139

73365

36679

19

IBM Software Group


Tuning CPLEX parameters

� Try the automatic Tuning Tool (starting with CPLEX 11)

� Use node log to identify whether lack of progress involves best integer, best node, lack of node throughput, or some combination of all three.

– Too many parameters for arbitrary experimentation

– Set parameters based on lack of progress

• MIP Emphasis parameter sets multiple other parameters at once� Settings 1 and 4 well suited for lack of progress in the best integer

� Settings 2 and 3 well suited for lack of progress in the best node

• In addition, particular parameters are well suited to address different

performance issues� More frequent heuristics, best estimate node selection, less backtracking well

suited for more progress in best integer

� More aggressive probing and cuts, strong branching, aggressive symmetry reductions well suited for more progress in best node

� LP performance tuning, turning off heuristics and cuts can help with slow node throughput

20

IBM Software Group

© 2012 IBM CorporationCopyright © ILOG 2008

Tuning CPLEX Parameters

�How do I know if parameter settings won’t get

me the results I need? Should I move on, or is

success just around the corner?

– Use extremely aggressive settings to address lack

of progress. If that doesn’t show progress, time to

look elsewhere for performance improvement

• For lack of progress in the best node, use MIP emphasis 3,

strong branching (variableselect 3), presolve symmetry 5.

Disable heuristics to speed up nodes. Use a MIP start from

a previous run if available. If that doesn’t show progress in

the best node, unlikely that other parameter settings will.

• For lack of progress in the best integer, apply heuristics at

every node, turn on feasibility pump heuristic, increase sub

MIP node limit for heuristics, set backtrack tolerance to 1.

21

IBM Software Group


Guidelines for Tightening Difficult MIPs

� Identify the parts of the model that make it difficult

– Simplify the model if necessary

• Filtering: remove any constraints and integrality restrictions not

involved in the performance trouble� If you remove a set of constraints or restrictions and the MIP remains

hard, that set is not involved in the performance issue

• Try to reproduce the trouble in a smaller data instance

� Determine how fractional solutions in the node relaxations

allow the objective to improve

� Use fractional solutions to identify the constraints and

variables that will motivate additional cuts

22

IBM Software Group


Guidelines for Tightening Difficult MIPs

� Common tactics for deriving additional cuts

– Linear or logical combinations of constraints

• Associate a graph with the model structure

– Disjunctions

– Solve one or more related models

– Use infeasibility

• CPLEX Conflict Refiner

– Use solution objective value

• Feasibility on model with constraint on objective value added

• Especially true for models with soft constraints

23

IBM Software Group


Guidelines: Linear, Logical Combinations

� Linear combinations of constraints

}1xi0 ,132

,131

,121 :{ 3

≤≤≤+

≤+

≤+∈= +

xx

xx

xxZxP

.5) .5, (.5, like solutions fractional offCut

1321 :down round y tointegralit Use

2/3321 :2by divide s,constraint theSum

≤++

≤++

xxx

xxx

24

IBM Software Group


Guidelines: Linear, Logical Combinations

� But how do we find the constraints to combine in a practical

model?

><

≤+

><

≤+

><

≤+

><

sconstraint 8000

132

sconstraint 4000

131

sconstraint 20000

121

sconstraint 10000

xx

xx

xx

25

IBM Software Group


Guidelines: Linear, Logical Combinations� Conflict graph

– Associate a node with each binary variable

– Create an edge between the nodes of any pair of binary variables that

cannot both be 1

– At most one binary in a clique can be 1

– Conflict graph identifies a useful linear combination of constraints

– CPLEX creates the conflict graph and by default finds clique cuts

}1xi0 ,132

,131

,121 :{ 3

≤≤≤+

≤+

≤+∈= +

xx

xx

xxZxP

x1

x3x21321 :Cut Clique ≤++ xxx

26

IBM Software Group


Guidelines: Disjunctions

Gomory/MIR cut:

),,2( off cuts ;232 Combining,

22 323

23322

3

}232321:{

}321:{

}10352314:{

51

31

21

41

41

21

41

41

21

21

41

413

25

45

433

3

≥+

≥⇒−≤+−⇒≥

≥⇒≥+−⇒≤

−=

+=+−++∈=

=++∈=

=++∈=

+

+

+

xx

xxxt

xxxt

t

xxxxxZxP

xxxZxP

xxxZxP

27

IBM Software Group



dxa

dxa

dxa

SS

SSxda

j

SSj

j

j

Sj

j

j

Sj

j

jj

≥

≥

≥

=∩

≥

∑

∑

∑

∪∈

∈

∈

21

2

1

Then

or

constraint edisjunctiv heConsider t

Suppose

,0,,Given

21

2,1

φ

28

IBM Software Group



)min()max(

Then

or

constraint edisjunctiv heConsider t

,, ,0Given

2,1

2

1

,21

,

,

ddx

dx

dx

Sddx

j

Sj

jj

j

Sj

j

j

Sj

j

jjj

≥

≥

≥

≥

∑

∑

∑

∈

∈

∈

βα

β

α

βα

29

IBM Software Group



� More general disjunctions involve solving LP subproblem to

generate cuts

– CPLEX does disjunctions on x <= k and x >= k+1 for fractional integer variables at the root node LP

– Can also do disjunctions on arbitrary constraints

• Write a program to solve the LP, calculate the cut

• See “Integer Programming” by Wolsey for the details

30

IBM Software Group


Guidelines: When to derive cuts

� CPLEX generates generic cuts, including

– Clique cuts

– Disjunctive cuts based on individual fractional variables

– Gomory cuts

– Zero half cuts

– See manual for complete list

– You don’t need to generate these yourself

� CPLEX is unlikely to generate cuts more specific to particular models

– Concepts used for Clique, Disjunctive and Gomory cuts can often be

customized to particular models with more special structure

31

IBM Software Group


Statistics: 559 constraints, 1066 variables (516 binary, 516 general integer)

Node log:

Nodes Cuts/

Node Left Objective IInf Best Int Best Node ItCnt Gap

0 0 101984.7744 28 101984.7744 35

*0+ 0 0 4.10026e+08 101984.7744 35 99.98%

153036.9306 35 4.10026e+08 Cuts: 41 151 99.96%

*0+ 0 0 4.00022e+08 153036.9306 151 99.96%

…

*55950+ 0 1.02822e+07 202475.0432 98.03%

56000 infeasible 1.02822e+07 202518.1842 98.03%

Elapsed time = 186.20 sec. (tree size = 13.36 MB).

Example: Penalty variables

32

IBM Software Group


Node log (ctd):

Nodes Cuts/

Node Left Objective IInf Best Int Best Node Gap

7149e4 7726073 infeas 1.02822e+07 307724.1416 97.01%

7150e4 7727024 309418.1 33 1.02822e+07 307728.0479 97.01%

Elapsed time = 161631.76 sec. (tree size = 9072.93 MB).

Nodefile size = 8945.58 MB (3124.04 MB after compression)

7151e4 7727720 357983.4 22 1.02822e+07 307731.4823 97.01%


…

33

IBM Software Group


Example: Penalty Variables

� Review the guidelines and tactics

– Simplify the model if necessary

–– Determine how fractional solutions affect the objectiveDetermine how fractional solutions affect the objective

– Use fractional solutions to motivate additional cuts

– Linear or logical combinations of constraints

– Disjunctions

– Solve one or more related models

– Use infeasibility

– Use solution objective value

34

IBM Software Group


Determine how fractional solutions affect the objective:Determine how fractional solutions affect the objective:

Min obj: 10000000 id134 + 10000000 id135 + ...

+ 10000000 id161 + 10000 id168 + 10000 id169

+ 1000 id170 + 1000 id171 + 34.299999237 id200

+ … + 10000 id2309

id78: id134 - id135 + 3 id200 + 3 id204 + 3 id220

+ 3 id228 + 3 id248 + ... + 3 id2096 + 3 id2144

+ 2 id2148 = 4


(Implied integer by integrality of other variables in the constraint)

35

IBM Software Group


Determine how fractional solutions affect the Determine how fractional solutions affect the objective(ctdobjective(ctd))::

Nodes Cuts/

Node Left Objective IInf Best Int Best Node Itcnt Gap

…

*55950+ 11356 0 1.02822e+07 202475.0432 861218 98.03%

56000 11367 infeasible 1.02822e+07 202518.1842 862287 98.03%

Elapsed time = 186.20 sec.

Comparing the best integer and best node values, we see that

removing integrality enables solutions with the sum of the

expensive penalty variables << 1. But, we don't know yet whether

an integer solution exists with all such penalty variables set to 0.

Can we answer that question?


36

IBM Software Group


Example: Penalty variablesYes we can, by solving a related problemsolving a related problem:

Add a constraint that sets all the expensive penalty

variables to 0:

conobj: id134 + id135 + id136 + id137 + … + id161 = 0

Results:

MIP - Integer infeasible or unbounded.

Current MIP best bound is infinite.

Solution time = 18.80 sec. Iterations = 409663

Nodes = 38384

37

IBM Software Group


Solve another related problemSolve another related problem, using solution objective valueusing solution objective value:

Nodes Cuts/


…

*55950+ 11356 0 1.02822e+7 202475.0432 861218 98.03%

56000 11367 infeasible 1.02822e+7 202518.1842 862287 98.03%

Elapsed time = 186.20 sec.

conobj: id134 + id135 + id136 + id137 + id138 + … + id161 >= 1


conobj: id134 + id135 + id136 + id137 + id138 + … + id161 = 1

38

IBM Software Group


0 0 1.01576e+07 16 1.01576e+07 54

1.01851e+07 20 Cuts: 42 82

…

*205 160 0 1.02922e+07 1.02098e+07 1233 0.80%

…

58200 319 infeasible 1.02822e+07 1.02806e+07 440316 0.02%

58300 226 cutoff 1.02822e+07 1.02811e+07 440441 0.01%

MIP - Integer optimal, tolerance (0.0001/1e-06):

Objective =1.0282191250e+07

Current MIP best bound = 1.0281164221e+07 (gap = 1027.03, 0.01%)

Solution time = 38.51 sec. Iterations = 440476 Nodes = 58326

(201)


Solve another related problemSolve another related problem, using solution objective valueusing solution objective value:

Nodes Cuts/


39

IBM Software Group



� Summary

–– Determine how fractional solutions affect the objectiveDetermine how fractional solutions affect the objective

• Involved both integer feasible solutions and fractional solutions to relaxations

–– Solve one or more related modelsSolve one or more related models

–– Use infeasibilityUse infeasibility

–– Use solution objective valueUse solution objective value

– More generally, don’t try to combine multiple objectives into a

single objective

• Solve separate problems with each individual objective

40

IBM Software Group


Methods for Adding Cuts to a Model

� Explicit Cuts

– More MIP preprocessing to tighten the model

– Larger problem may increase memory, run time

� User Cuts at start of optimization

– Added to cut pool rather than problem

– Less memory, node relaxation solve time

– MIP preprocessing not applied to user cuts

� Cut Callback

– Use when need to add cuts at child nodes

• Child node information needed to derive cuts

• Too many cuts to add at start of optimization

– More work to implement

41

IBM Software Group


Debugging/Sanity checking� Sufficient (but not necessary) conditions that the cut(s) tightened the MIP

too much

– Adding the cuts makes a feasible MIP infeasible

– Or adding the cuts contradicts the upper or lower bound of the original

MIP

• Use CPLEX’s conflict refiner to obtain minimal set of infeasible constraints

Lower Bound = 50

Upper Bound = 100

G

A

P

cuts Infeasible

Original MIP Tightened MIP

42

IBM Software Group


Debugging/Sanity checking

� Sufficient (but not necessary) conditions that the cut(s) tightened the MIP too

much

– Reversing the direction of the (globally valid) cut and optimizing finds an

integer feasible solution

• Fix variables to solution for the tightened model with the bad cut, run the conflict refiner on the resulting infeasible model

bad cut

43

IBM Software Group


Debugging/Sanity checking

� Sufficient (but not necessary) conditions that the cut(s) didn’t tighten the MIP enough

– Fractional solutions that we used to identify the cuts are not cut off

in the node LPs of the tightened MIP

44

IBM Software Group


Conclusions

� Most MIPs easy to solve (defaults or parameter tuning)

� The remaining MIPs can be very time consuming for the practitioner

� I hope the presentation didn’t just illustrate how to improve performance on the particular example

� Use the guidelines to help quickly identify the good cuts on your own difficult MIPs

� Sometimes a sharp cut found in an hour’s work is better than the sharpest cut found in a week’s (or more) work

45

IBM Software Group


Additional Reading� Wolsey, L. Integer Programming.

� Polya, How to Solve It.

� Bixby, Fenelon, Gu, Rothberg, Wunderling. Mixed Integer Programming: A

Progress Report. Available in Grotschel, M.(editor), The Sharpest Cut.

� Klotz, Newman. Practical Guidelines for Solving Difficult Mixed Integer Programs

http://www.sciencedirect.com/science/article/pii/S1876735413000020 or

http://inside.mines.edu/~anewman/MIP_practice120212.pdf

� (LP Performance Tuning): Klotz, Newman. Practical Guidelines for Solving

Difficult Linear Programs

http://www.sciencedirect.com/science/article/pii/S1876735412000189

� CPLEX MIP parameter settings:

http://www-01.ibm.com/support/docview.wss?uid=swg21400023

46

IBM Software Group


Q&A Reminder

� Please write your questions to the Q&A Group via SmartCloud Meeting

– As we are expecting a lot of participants in today’s session, please send your

questions via SmartCloud Meeting Chat:

• This is the conversation balloon tab in the right hand panel of the meeting

center

• Address the question to the Q&A Group option in the drop down box

– If you have further questions or sales enquiries after the meeting please

contact Kitte Knight at [email protected]

46

Science

Practical Guidelines for Solving Difficult Mixed Integer Programs