Optimization on the Computational Gridegon.cheme.cmu.edu/ewo/docs/LinderothSEminaroptimization_grid.pdf · Optimization on the Computational Grid Jeff Linderoth ISE Department COR@L

Optimization on the Computational Grid

Jeff Linderoth

ISE DepartmentCOR@L Lab

Lehigh [email protected]

Enterprise-Wide Optimization (EWO) Tele-SeminarAllentown, PAMay 9, 2006

Jeff Linderoth (Lehigh University) Optimization on the Computational Grid EWO Tele-Seminar 1 / 67

An Abject Apology

Much of this work was done long ago

There is no original material!I even have reused many of the same hackneyed jokes

I have a good excuse


My Good Excuse—Jacob Linderoth


Outline

What I lack in quality, I make up for in quantity: ≥ 70 slides

What is The Computational Grid?

Tools for using the Computational Grid

Condor and MW

Solving large optimization problem instances on the ComputationalGrid

Two-stage stochastic linear programmingQuadratic Assignment Problem

Theme of the Talk

Fit your algorithm to your computational platform!


The Grid Richard Dawson Rules!

Come on Let’s Play the Feud

‘‘100 People Surveyed. Top5 answers are on the board.Here’s the question...’’

Name one common use of the Internet



The Big Board

1 email

2 Looking up answers to homeworkproblems

3 Looking up stock quotes

4 Downloading MP3s

5 Looking at pictures of Anna Kournikova



Strike!

DoingComputations


The Grid Building a Grid

People envision a “Computational Grid” much like the national powergrid

Users can seamlessly draw computational power whenever they need itMany resources can be brought together to solve very large problemsGives application experts the ability to solve problems of unprecedentedscope and complexity, or to study problems which they otherwise wouldnot.



This ain’t easy!

User access and security

Who should be allowed to tap in?

Interfaces

How should they tap in?

Heterogeneity

Different hardware, operating systems, and software

Dynamic

Participating Grid resources may come and goFault-Tolerance is very important!

Communicationally challenged

Machines may be very far apart ⇒ slow communication.



Building a Grid

There have been lots of software tools that accomplish these goals tovarying degrees.

One problem remains: GREED!Most people don’t want to contribute “their” machine!

How to induce people to contribute their machine to the Grid?

Screensaver – seti@homeSocial Welfare – fightaids@homeOffer frequent flyer miles – company went bankruptLet the people keep control over their machineGive donaters a chance to use the Grid


The Grid Condor

Tool I – Condor

Peter KellerMiron LivnyErik PaulsenRajesh Raman

Marvin SolomonTodd Tannenbaum

Doug ThainDerek Wright

http://www.cs.wisc.edu/condor


The Grid Condor

What is Condor?

Manages collections of “distributively owned” workstations

User need not have an account or access to the machineWorkstation owner specifies conditions under which jobs are allowed torunAll jobs are scheduled and “fairly” allocated among the pool

How does it do this?

Scheduling/MatchmakingJobs can be checkpointed and migratedRemote system calls provide the originating machines environment


The Grid Condor

Matchmaking

MyType = JobTargetType = MachineOwner = ralphsCmd = cplexArgs = seymour.d10.mpsHasCplex = TRUEMemory ≥ 64Rank = KFlopsArch = SUN4uOpSys = SOLARIS27 ||

SOLARIS28

MyType = MachineTargetType = JobName = nova9HasCplex = TRUEArch = SUN4uOpSys = SOLARIS27Memory = 256KFlops = 53997RebootedDaily = TRUE


The Grid Condor

Checkpointing/Migration


The Grid Condor

Other Condor Features

Pecking Order

Users are assigned priorities based on the number of CPU cycles theyhave recently usedIf someone with higher priority wants a machine, your job will bebooted off

Flocking

Condor jobs can negotiate to run in other Condor pools.

Glide-in

Globus (Grid computing toolkit from Argonne) provides a “front-end”to many traditional supercomputing sites.Submit a Globus job which creates a temporary Condor pool on thesupercomputer, on which users jobs may run.


The Grid Condor

Condor is Really Taking Off

Companies Presenting at Recent Condor Week 2006

JP Morgan

Altera Corporation

Hartford Life

Micron

UBS

Yahoo

http://www.cs.wisc.edu/condor/CondorWeek2006/

http://www.cs.wusc.edu/condor


http://www.cs.wisc.edu/condor/CondorWeek2006/

http://www.cs.wusc.edu/condor

The Grid MW

Grid computing applications

What Condor is good at...

Naturally/Pleasantly/Embarassingly Parallel Applications

The computation can a priori be broken into tasks

Monte Carlo Methods, Code Crackingseti@home – the search for intelligent life

What Condor is (not so) good at...

Parallel Applications with non-trivial control structures

Like Optimization Algorithms for EWO!

To make parallel algorithms dynamically adjustable and fault-tolerant,we could use the master-worker paradigm

What is the master-worker paradigm, you ask?


The Grid MW

Master-Worker!Fee

dM

e!OK!

Tutor

Me!

OK!

Master assigns tasks to theworkers

Workers perform tasks, andreport results back to master

Workers do not communicate(except through the master)

Simple!

Fault-tolerant

Dynamic


The Grid MW

Tool II – MW

Wen-Han Goh

Sanjeev KulkarniGreg ThainMike Yoder{

Jean-Pierre GouxJeff Linderoth

http://www.cs.wisc.edu/condor/mw


The Grid MW

MW

There are three abstraction is the master-worker paradigm: Master,Worker, and Task.

MW is a software package that encapsulates these abstractions

C++ abstract classesUser writes 10 functionsThe MWized code will transparently adapt to the dynamic andheterogeneous environment

MW also has abstract layer to resource management andcommunications packages

Condor/PVM, Condor/FilesCondor/Unix SocketsSingle processor


The Grid MW

MW Classes

MWMaster

get userinfo()setup initial tasks()pack worker init data()act on completed task()

MWTask

(un)pack work(un)pack result

MWWorker

unpack worker init data()execute task()


The Grid MW

MWApplications

MWFATCOP (Chen, Ferris, Linderoth) – A branch and cut code for linearinteger programming

MWMINLP (Goux, Leyffer, Nocedal) – A branch and bound code fornonlinear integer programming

MWSVM (Ferris, Voelker) – A column generation code for solving supportvector machine problems

MWQPBB (Linderoth) – A branch-and-bound code for nonconvex quadraticprogramming

MWAND (Linderoth, Shen) – A (preliminary) nested-decomposition code formultistage stochastic linear programming

MWLShaped/MWATR (Linderoth, Shapiro, Wright) – A (trust-region)cutting plane code for linear stochastic programming and verification ofsolution quality

MWQAP (Anstreicher, Brixius, Goux, Linderoth) – A branch and boundcode for solving the quadratic assignment problem


Stochastic Programming Background

Stochastic Programming Collaborators

Alex Shapiro

ISyEGeorgia Tech

Steve Wright

Computer Science DepartmentUniversity of Wisconsin-Madison


Stochastic Programming Background

Stochastic Programming

A Stochastic Program

minx∈X

{f(x)def= EP F (x, ω)}

F (x, ω) is a real-valued functionof two vector variables.

x ∈ Rn and ω ∈ Rd

ω is a random vector havingprobability distribution P

X ⊆ <n

A Two-Stage Stochastic LP

minx≥0,Ax=b

cT x +Q(x)

Q(x)def= EP [Q(x, ω)]

The recourse problem:

Q(x, ω)def= min qT y

Wy = h(ω)− T (ω)x

y ≥ 0


Stochastic Programming The LShaped Method

Best-Known Solution Procedure

METH O D



Two-Stage Stochastic Linear Programming

We assume that the P has finite support, so ω has a finite number ofpossible realizations (scenarios):

Q(x) =N∑

i=1

piQ(x, ωi)

For a partition of the N scenarios into chunks N1,N2, . . .Nt, letQ[j](x) be the contribution of the jth chunk to Q(x):

Q[j](x)def=

∑i∈Nj

piQ(x, ωi)

so then Q(x) =∑t

j=1Q[j]



Important (and well-known) Facts

Q(x, ωi), Q[·](x), and Q(x) are piecewise linear convex functions of x.

If πi is an optimal dual solution to the linear program correspondingto Q(x, ωi), then −T T

i πi ∈ ∂Q(x, ωi)

gj(x)def=

∑i∈Nj

−piTTi πi ∈ ∂Q[j](x).

Represent Q[j](x) by an artificial variable θj and find supportingplanes for θj

θj ≥ gj(xk)T x + (Q[j](x

k)− gTj xk) (∗)

Evaluation of Q(x) is separable

We can solve linear programs corresponding to each Q(x, ωi)independently – in parallel!



Worth 1000 Words

x

Q(x)



Worth 1000 Words

x

Q(x)

xk



Worth 1000 Words

x

Q(x)

x1x2



Multicut L-shaped method

M

s1 s2 s3 s4 s3

M

s1 s2 s3 s4 s5

1 Solve the masterproblem M with thecurrent θj-approximationsto Q[j](x) for xk.

2 Solve the subproblems,(sj) evaluating Q[j](x

k)and obtaining asubgradient gj(x

k). Addinequalities (*) to themaster problem

3 k = k+1. Goto 1.



A Trust Region Method

The LShaped method suffers from “stability” problems,

Especially in early iterations when a “good enough” model of Q(x) isnot knownEspecially bad if started from a good guess at the solution

Borrow the trust region concept from NLP

At iteration k, impose constraints ‖x− xk‖∞ ≤ ∆k

Accept new iterate if it improves the objective by a “sufficient”amount. Potentially increase ∆k.Otherwise, improve the estimation of Q(xk), resolve master problem,and potentially reduce ∆k.



Stop Thief!

“Lesser artists borrow, great artists steal.”

– Igor Stravinsky

Stabilizing Bender’s decomposition is hardly a new idea

Marsten, Hogan, Blankenship (’75)Lemarechal (’75, . . . )Kiwiel (’83, . . . )Ruszczynski (’86)Neame, Boland, and Ralph (’98)



minor Differences

We like ‖ · ‖∞ trust region

The master problem is just an LP with modified boundsDegeneracy has not reared its ugly head

(Perhaps due to better LP solvers)

Convergence analysis is somewhat different

Convergence for arbitrary subgradients – not necessarily extreme pointdual solutions π(ωi).

Liberal cut deletion from master is possible

This is especially important in a grid computing implementation


Stochastic Programming Gridifying the LShaped Method

Preparing for the Grid

Work

One or more scenario chunks Nj1 , . . .NjCand point (x)

Result

A subgradient of each of the Q[jk](x).

act on completed taskAdd subgradient inequalities to master problemSolve master problem if all workers have reported their results for theiteration



Headaches!

Solving the master problem is a“synchronization point” of thealgorithm

Amdahl’s Law: Parallel efficiency islimited by the amount ofsynchronization.

This synchronization problem is MUCH worse in Computational Gridcomputing environments!



Worker Usage—Number of Idle Workers

0

10

20

30

40

50

60

70

80

90

100

0 500 1000 1500 2000 2500 3000 3500 4000

Num

ber

of Id

le W

orke

rs

t (sec.)Jeff Linderoth (Lehigh University) Optimization on the Computational Grid EWO Tele-Seminar 37 / 67


Stamp Out Synchronicity!

On a Grid, different processors act at different speeds,

Many may wait idle for the “slowpoke”

Even worse, grid computing tools can fail to inform the user that theirworker has failed!

Asynchronicity is key!

Asynchronous methods are preferred for traditional parallel computingenvironments. They are nearly required for Grid Computing environ-ments!



ATR – An Asynchronous Trust Region Method

Keep a “basket” B of trial points for which we are evaluating theobjective function

Make decision on whether or accept new iterate xk+1 after entireQ(xk) is computed

Populate the basket quickly by initially solving the master problemafter only α% of the scenario LPs have been solved

Greatly reduces the synchronicity requirements

Might be doing some “unnecessary” work – the candidate pointsmight be better if you waited for complete information from thepreceding iterations


Stochastic Programming Solving Big Instances

The World’s Largest LP

Storm – A stochastic cargo-flight scheduling problem (Mulvey andRuszczynski)

We aim to solve an instance with 10,000,000 scenarios

x ∈ <121, yk ∈ <1259

The deterministic equivalent LP is of size

A ∈ <985,032,889×12,590,000,121



The Super Storm Computer

Number Type Location

184 Intel/Linux Argonne

254 Intel/Linux New Mexico

36 Intel/Linux NCSA

265 Intel/Linux Wisconsin88 Intel/Solaris Wisconsin239 Sun/Solaris Wisconsin

124 Intel/Linux Georgia Tech90 Intel/Solaris Georgia Tech13 Sun/Solaris Georgia Tech

9 Intel/Linux Columbia U.10 Sun/Solaris Columbia U.

33 Intel/Linux Italy (INFN)

1345



TA-DA!!!!!

Statistics from Storm Computation

Wall clock time 31:53:37CPU time 1.03 Years

Avg. # machines 433Max # machines 556Parallel Efficiency 67%

Master iterations 199CPU Time solving the master problem 1:54:37

Maximum number of rows in master problem 39647



Number of Workers

0

100

200

300

400

500

600

0 20000 40000 60000 80000 100000 120000 140000

#wor

kers

Sec.Jeff Linderoth (Lehigh University) Optimization on the Computational Grid EWO Tele-Seminar 43 / 67

Quadratic Assignment Problem Background

The Quadratic Assignment Problem

Mathematical Formulation

minπ∈Π

n∑i=1

n∑j=1

aijbπ(i)π(j) +n∑

i=1

ciπ(i)

Assign facilities to locations

QAP is NP-”Super”-Hard

Branch and Bound is themethod of choice, but very fewtight, computable, bounds exist.


Quadratic Assignment Problem Background

QAP Collaborators

{Kurt Anstreicher

University of Iowa

{Nate Brixius

Micro$oft

{Jean-Pierre Goux

Argonne, Northwestern, and Artelys


Quadratic Assignment Problem Practical Branch-and-Bound

How to solve BIG problems with Branch and Bound

1 Start with a state of the art bounding technique2 Do intelligent search and branching

We employ/extend “strong branching” in this non-linear contextUse different branching strategies depending on the relative difficulty ofa node

3 Pick a very powerful computing platform

The Computational Grid!

4 Fit the algorithm to the platform



Branch and Bound

Kurt Anstreicher and Nate Brixius showed that the solution to thefollowing problem provides a lower bound to the solution of QAP:

min f(X) ≡ vec(X)T Q vecX + C •X

such that Xe = XT e = e, X ≥ 0.

Q ≡ (B ⊗A)− (I ⊗ S)− (T ⊗ I)

S and T are obtained from the spectral decompositions of A and B

There are more details that I don’t understand

This is a convex quadratic programming problem relaxation



An Oldie but a Goodie

The Frank-Wolfe algorithm is used to solve the quadraticprogramming subproblems

1. X∗k = arg minX∈Π{∇f(Xk) •X}. Linear Assignment Problem

2. Xk+1 = Xk + α(X∗k −Xk), where 0 ≤ α ≤ 1 minimizes f(Xk+1).

Closed form solution

3. Go to 1.

Each X∗k comes with a matrix of “reduced costs”, from which a valid

lower bound can be deduced: zOPT ≥ f(Xk)− Uk •Xk.











Branch and Bound

We do not do “normal” IP branching

Instead do n-ary branching

Fix each assignment in arow or column

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � ��

� � � � � � � � � ��

� � � � � � � � � ��

� � � � ��

� � � � ��

� � � � ��

� � � � ��

� � � � � � � � � ��

� � � � � � � � � ��

� � ��

� � ��

! !! !! !! !! !! !! !! !! !! !! !! !! !

" " " " "" " " " "" " " " "" " " " "" " " " "" " " " "" " " " "" " " " "" " " " "" " " " "" " " " "" " " " "" " " " "" " " " "" " " " "

# # # # ## # # # ## # # # ## # # # ## # # # ## # # # ## # # # ## # # # ## # # # ## # # # ## # # # ## # # # ## # # # ## # # # ## # # # #

$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $$ $ $ $ $ $ $ $ $

% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % %% % % % % % % % % Facility 3

Location 2

Also adapted strong-branching to this nonlinear context.



Strong Branching

There are many “unsatisfied entities”, how do we decide on which tobranch?

We would like to select an entity that will increase the bound as muchas possible

MIP – Tentatively select a fractional variable, and perform a smallnumber of dual simplex pivots

QAP – Tentatively select a row or column and perform a smallnumber of Frank-Wolfe iterations

Branch on entity where the lower bounds improve “the most”











Our Computational Grid

Number Type Location414 Intel/Linux Argonne96 SGI/Irix Argonne

1024 SGI/Irix NCSA16 Intel/Linux NCSA45 SGI/Irix NCSA246 Intel/Linux Wisconsin146 Intel/Solaris Wisconsin133 Sun/Solaris Wisconsin190 Intel/Linux Georgia Tech94 Intel/Solaris Georgia Tech54 Intel/Linux Italy (INFN)25 Intel/Linux New Mexico5 Intel/Linux Columbia U.10 Sun/Solaris Columbia U.12 Sun/Solaris Northwestern

2510











MW Implementation

Fitting the B & B algorithm into the master-worker paradigm is notground-breaking research

We must avoid “contention” at the master



All The Queueing Theory I Know

We can reduce contention in two ways

1. Increase the service rate2. Reduce the arrival rate

A parallel depth-first oriented strategy achieves these goals.

Available worker is given “deepest” node by masterWorker examines the subtree rooted at this node in a depth-firstfashion for t seconds.



Parallel Depth-First Search

��

��

Master Task Pool

Worker A

sent back to masterUnexplored nodes

Worker B

Worker C

Other “standard” searchstrategies fail completely!

Too much memoryrequired at masterToo many nodes passedback to master

Don’t try this at home!

If you don’t have a goodupper bound with which tofathom, this can failmiserably!



Truth in Advertising

The parallel depth-first search strategy is awful too!

Define the parallel efficiency:

η =Σ(Time workers spend executing tasks)

Σ(Time workers are available)

In our initial implementation, η = 0.41

Since there is very little synchronization required in the algorithm,this number is shockingly low!



Deducing the Problem

We may want the workers toexamine a subtree for tseconds, but that doesn’tmean that there are tseconds of work!

A histogram of task times:

0

500

1000

1500

2000

2500

3000

0.1 1

� �

10

� �

100

� �

�

�

�

� �



Elementary, Dear Watson

Make sure that workers onlypass back nodes that willhave enough “meat”

Order children so that“easy” ones are first in theDFS stackAllow additional time forworkers to pop up the DFSstack, finishing offremaining easy nodes.

η improved to 0.9

0

150

300

450

600

0.1 1��

10�

100�

300

� �

500

� �

��



The Holy Grail

(NUG30) (n = 30) has been the “holy-grail” of computational QAPresearch for > 30 years

In 2000, Anstreicher, Brixius, Goux, & Linderoth set out to solve thisproblem

Using an old idea of Knuth, we estimated the CPU time required to solveNUG30 to be 5-10 years on a fast workstation

We’d better get a pretty power computing platform, like the ComputationalGrid.



NUG30 is solved!

14, 5, 28, 24, 1, 3, 16, 15, 10, 9, 21, 2, 4, 29, 25, 22, 13, 26, 17, 30, 6, 20, 19, 8, 18, 7, 27, 12, 11, 23

“My father used 3.46× 108 CPU seconds, and all I got wasthis lousy permutation”

Wall Clock Time: 6:22:04:31Avg. # Machines: 653

CPU Time: ≈ 11 yearsNodes: 11,892,208,412LAPs: 574,254,156,532

Parallel Efficiency: 92%



Workers



KLAPS



Even More Wasted CPU Time

KRA30B KRA32 THO30

Wall Clock Time (Days) 3.79 12.3 17.2Avg. # Machines 462 576 661Max. # Machines 780 1079 1307CPU Time (Years) 4.32 15.2 24.7

Nodes 5.14× 109 16.7× 109 34.3× 109

LAPs 188× 109 681× 109 1.13× 1012

Parallel Efficiency: 92% 87% 89%



Conclusions

The Computational Grid is a powerful computing platform

With the Master-Worker paradigm, it is relatively easy to implementand run on the Grid.

With a little tuning, many algorithms can be “shoehorned” to runeffectively with the Master-Worker paradigm, and hence on the Grid.

You have to fit your algorithm to the computational platform!

The boost in computing power that you get by making yourcode/algorithm Grid-enabled is worth the effort!



The End!

We want YOU to use the Computational Grid and MW.

Papers at http://www.optimization-online.org

http://www.cs.wisc.edu/condor/mw

mailto:[email protected]


Documents

Optimization on the Computational Gridegon.cheme.cmu.edu/ewo/docs/LinderothSEminaroptimization_grid.pdf · Optimization on the Computational Grid Jeff Linderoth ISE Department COR@L