Chapter 19

ASYNCHRONOUS TEAMS

Sarosh TalukdarCarnegie Mellon University

Sesh Murthy and Rama AkkirajuT. J. Watson Labs, IBM

1 INTRODUCTION

Obtaining a good solution to a complex problem, such as the operation of an electric gridor the scheduling of a job-shop, can require a large and diverse set of skills. (An electricgrid has millions of devices that must be controlled and coordinated. In job-shop-scheduling, much of the problem can often be solved by a single optimization procedure,such as linear programming. But the remainder invariably requires a variety of situation-specific heuristics and insights.) The questions for the designer of a problem-solvingsystem are which skills to include and how to package them.

In an Asynchronous Team (A-Team), skills are packaged as agents. More specif-ically, an A-Team is a multi-population, multi-agent system for solving optimizationproblems. The calculation process is as follows: The problem-to-be-solved is decom-posed into sub-problems. A population of candidate-solutions is maintained for eachsub-problem. These populations are iteratively changed by a set of agents. Also, solu-tions are made to circulate among the populations. Under conditions to be discussed,some of the solutions in each population will improve. The calculation process isterminated when the improvements cease.

The agents in an A-Team are identical in two respects. First, they are all com-pletely autonomous (they work without any supervision). Second, they all have thesame work-cycle (in every iteration, every agent performs the same three steps: selecta solution from a population; modify the selected solution; then insert the modifiedsolution in a population). In all but these two respects, the agents can, and usuallyshould, be diverse. For instance, some may be computer programs while others arehuman. Some may iterate quickly while others are slow. Some may make smallimprovements while others make radical or even innovative changes to the candidate-solutions.

S. Talukdar et al.

Design and assembly: How much effort will be needed to produce and upgradethe system?

Solution-quality: How close will the solutions obtained by the system be to thebest possible solutions?

Solution-speed: How quickly will the system complete its calculations?

Robustness: How will solution-quality and speed be affected by failures anddisturbances?

2 OPTIMIZATION TERMINOLOGY

Optimization is not the best language for expressing every problem, but it is rela-tively simple and quite general (all problems can, at least in principle, be expressed asoptimization problems). It is used here for these reasons.

An optimisation problem is a search specified in terms of three components—objectives, constraints and decision variables. The objectives specify the goals of thesearch, the constraints specify the limits of the search, and the decision variables specifythe space over which the search is to be conducted. In other words, an optimisationproblem has the form: where is a set of objectives, C is a set of constraints,

In designing a problem-solving system there are four conflicting issues of concern:

A qualitative assessment of the principal relationships between the structuralfeatures of A-Teams and these issues-of-concern is given in Figure 19.1.

The remainder of the chapter is organized as follows. Section 2 lists the terminologythat will be used in describing problems and their solutions. Sections 3–6 elaborate onthe cause–effect-relationships indicated in Figure 19.1. Section 7 describes the structureof an A-Team. Section 8 describes how the quality of the solutions produced by anA-Team can, when necessary, be improved. Section 9 contains some guidelines fordesigning A-Teams. And, Section 10 describes an A-Team that is used for solvingjob-shop-scheduling problems from paper mills.

538

Asynchronous Teams 539

and X is a set of decision variables. The values that the decision variables can takeconstitute a space, S, that is called decision- or solution-space. As an example, considerthe problem: simultaneously solve the equations, and x = y. One of manyways of expressing this problem in optimisation terms, is:

Here, “x” and “y” are decision variables, is an objective, “x = y” isa constraint, and the solution-space is the plane whose axes are the decision-variables,x and y.

The solution space can be divided into three subsets: infeasible solutions (thoseviolating one or more constraints), feasible solutions (those meeting all the constraints),and optimal solutions (the best of the feasible solutions).

Real problems often contain either no objectives or multiple conflicting objectives.In the former case, all the feasible solutions are optimal, in the latter case, the Paretosolutions are optimal. (Two objectives conflict—as opposed to being commensurate—when one objective cannot be improved without degrading the other. A Pareto solutionis a feasible solution that is not dominated by any other feasible solution. Solution-Xdominates solution-Y, if X is at least as good as Y with respect to all the objectives,and X is better than Y with respect to at least one objective.)

Two optimization problems are coupled if they share decision variables, constraintsor objectives.

Consider a set of coupled optimization problems, The pointX is a Pareto solution of this set, if X is feasible (with respect to the constraints of all theproblems) and X is not dominated (with respect to the objectives of all the problems)by any other feasible point.

Suppose that the problem-to-be-solved, is decomposed into B, a set of coupledsub-problems. Suppose that a set of agents, V, is assigned the task of iteratively solvingthe sub-problems. The result is a dynamic system: where is theassignment of agents to sub-problems, is the schedule of communications amongthe agents, and is the schedule of the iterations by the agents. If the results of thisdynamic system converge, they will do so, not to optimal solutions of but rather,to equilibria and other attractors of the dynamic system. (These attractors are features,such as points and limit cycles, in the solution space of the ) Note: changing any ofthe five components of the dynamic system will change its attractors.

If: is one-to-one (sub-problem-m is assigned to agent-m),

prescribes sequential iterations (the agents take turns working, each completingone iteration per turn),

V contains only optimizing agents (each agent solves its sub-problem tooptimality in each iteration),

prescribes information broadcasts (each agent sends the results of each of itsiterations to all the other agents), and

the iteration-results converge to a single point in the solution space of

x, y

subject to x = y

540 S. Talukdar et al.

then: this point, is called a Nash equilibrium. Note: a Nash equilibrium represents astalemate (once all the agents get there, no agent can leave without suffering adegradation of its objective or a violation of its constraints). Also, the Paretosolutions of the Pareto solutions of B, and the Nash equilibria may all bedistinct and quite different.

3 POPULATION METHODS

Consider the problem with solution space S. A population method, when appliedto this problem, generates a sequence, where is a population (set) ofcandidate solutions, and is calculated from with k < n. The calculations ceasewhen at least one member of the latest population is found to have come acceptablyclose to an optimal solution.

The original population method, called the Simplex method, was proposed bySpendley et al. [1]. The name comes from the method’s use of a simplex to searchthe solution-space. A simplex in a space of dimension m, is a set of m + 1 pointsthat can be thought of as the vertices of a body with planar faces. The method obtains

from by replacing the worst vertex in by its reflection in the centroidof the other vertices. Many variations of this method have been proposed, includingmethods called “swarms” that claim insect societies as their inspiration, but have morein common with simplex methods than insects. On the plus side, simplex-like methodshave wide applicability and easy implementation; on the minus side, simplex-likemethods can be very slow and tend not to handle constraints well.

Genetic algorithms are better known examples of population methods than theSimplex method. In the basic genetic algorithm, solutions are represented by binarystrings. is calculated from First, a “fitness function” is used to select thefittest members of Next, “crossover” and “mutation,” two operators that mimicprocesses in natural reproduction, are repeatedly applied to the fittest members toproduce the new population. On the plus side, genetic algorithms tend to produce goodsolutions and are fairly robust. On the minus side, they are painfully slow and are oftendifficult to apply—a consequence of using only string-representations and synchronouscalculations ( must be calculated from and the faster operators to have to waitfor the slowest to finish.) A-Teams attempt to eliminate the disadvantages of Simplex-like methods and Genetic algorithms by packaging skills in autonomous agents ratherthan in mathematical operators, such as “crossover”and “mutation.”

4 AUTONOMOUS AGENTS

Agents are the modules from which problem-solving systems can be built. Structurally,an agent can be thought of as a bundle of sensors, decision-makers and actuators.Behaviorally, an agent can be thought of as a mapping from an in-space (all the thingsthe agent can sense) to an out-space (all the things the agent can affect). These spacesmay overlap. (Note that by these definitions, a great many, very different things qualifyas agents, including thermostats, computer programs, mobile robots, insects, people,societies and corporations.)

An agent is autonomous to the extent that it is unsupervised. A non-autonomousagent does only what its supervisors tell it to do; it makes no decisions for itself. In

Asynchronous Teams 541

No supervisory structure need be constructed.

The mistakes and failures of an agent do not affect its supervisees (there are none).

The speed of an agent’s reactions are limited only by its capabilities, not by thedelays in a chain-of-command.

The agents can work asynchronously (in parallel, each at its own speed).

The agents can improve themselves through learning (converting experiences intocompetence).

where and are mappings representing the selection and modification processesof the j-th agent, and and are sequences of the populations in this agent’sinput and output memories.

The agents can be divided into two categories, creators and destroyers, by differ-ences in the way they perform the second step. Creators add solutions to the populationsin their output memories ( is non-empty); destroyers remove solutions from thepopulations in their input memories ( is empty). The creators can be furtherdivided into constructors and improvers. The constructors produce new solutions fromscratch ( is empty); the improvers modify existing solutions ( is non-empty).

Note that the output memory of an agent may be the same as its input memory, andseveral agents may share memories, resulting in configurations of the sort shown inFigure 19.2.

5 ASYNCHRONOUS WORK

Synchronous iteration schedules dictate when agents must work and when they mustremain idle. They do this by specifying the order in which iterations are to occur, for

contrast, a completely autonomous agent has no supervisors, rather, it decides entirelyfor itself what to do and when.

The exclusive use of completely autonomous agents has some advantages:

However, the work of completely autonomous agents cannot be externallycoordinated. Unless such agents are self-coordinating, they will tend to work atcross-purposes, creating pandemonium rather than progress.

In an A-Team, self-coordination results from the agents’ selection mechanisms.(These mechanisms are part of the first step of the work-cycle of each agent. This work-cycle is outlined below. The selection mechanisms and other details are described inlater sections.)

The agents of an A-Team work on populations of candidate-solutions. These pop-ulations are stored in computer memories. Consider the j-th agent. In each of itsiterations, this agent performs three steps: (a) it reads the incumbent population inits input-memory, selects some candidate-solutions, and removes the selected solutionsfrom (b) it modifies the selected solutions; and (c) it inserts the modified solutionsinto the incumbent population in its output-memory. Symbolically:

542 S. Talukdar et al.

instance, “no agent will begin iteration m +1 till all the agents have completed iterationm.” Asynchronous schedules do not constrain the order of iterations. Rather, they allowall the agents to work in parallel all the time, each at its own speed. In other words,asynchronous schedules allow completely autonomous agents, if they so choose, toreduce their idle times to zero.

The relationship between idle times and overall performance is neither obvious normonotone. But in certain circumstances, reducing idle times does increase performance.A qualitative explanation follows.

Consider a synchronous schedule in which the iterations of all the agents occurin lock-step, and there is just one memory that serves as both the input and the out-put memory for all the agents. Suppose that in its n-th iteration, each agent selectsand modifies some members from the latest population, in this memory, and theaggregate of these modifications produces the next population, In other words:

where

Asynchronous Teams 543

and J is the set of agents. In this synchronous schedule, all the agents must wait tillthe slowest one has finished.

This synchronous schedule can be converted to an asynchronous schedule by relax-ing the constraint that the agents work in lock-step and allowing each to iterate as fastas it can. As a result, one agent may be performing its 100-th iteration while another isjust on its 10-th. Symbolically, the m-th iteration for the j-th agent is:

where is the latest population when agent-j starts its m-th iteration, and is thelatest population when agent-j completes its m-th iteration.

It happens that sufficient conditions for (4)–(7) to converge to a unique solution areonly slightly less restrictive than sufficient conditions for (8)–(10) to converge to thesame solution [2]. Therefore, one may expect that the reductions in idle time obtainedthrough the use of asynchronous schedules will often leave solution-quality unaffected.What about solution-speed? It seems that populations are able to transmute reductions-in-idle-time into increases in solution-speed when many agents are involved. We donot understand the mechanism, but populations seem to be especially helpful when theagents differ considerably in speed.

Fast agents, such as those using hill-climbing techniques, iterate quickly and tend tobe good at refining solutions through a succession of many incremental improvements.Slower agents, such as humans, are often better at making radical changes. These twotypes of changes can be combined by constraining the fast and slow agents to workin series, so a few iterations of a slow agent are always followed by many iterationsof a fast agent. Populations provide a better way to combine fast and slow agents.Specifically, they free agents to work in parallel, each as fast as it can. In other words,populations seem to provide for the automatic blending of incremental and radicalchanges. (For experiments with mixes of very different software agents, see [3,4], formixes of software and human agents, see [4,16].)

6 COMPETITION AND COOPERATION

In 1776, Adam Smith made a case for markets and competition [5]. His insight was thatgood solutions to certain large and difficult problems could be obtained by dividing theproblem among a number of autonomous agents, and making these agents compete.This insight has become an article of faith for some designers of autonomous-agent-systems, particularly, designers of markets. But autonomy and competition do not,by themselves, guarantee optimality. Rather, a number of other, quite impracticalconditions must also be met, such as the availability of perfect information and theabsence of externalities [6]. These other conditions are invariably violated by realsystems. Consequently, competitive arrangements often produce very poor solutions,such as the shortages and high prices of the California energy market of 2001.

Of course, agents do not have to compete. They can, instead, cooperate.

544 S. Talukdar et al.

Competition requires conflicting goals. In optimization terms, two agents cancompete if they are assigned distinct but coupled problems with conflicting objec-tives. Cooperation requires commensurate goals. In optimization terms, two agentscan cooperate if they are assigned coupled problems with commensurate or almostcommensurate objectives.

In other words, whether agents cooperate or compete depends, at least in part, onthe problems they are assigned.

Let be the problem-to-be-solved; and let be a decomposi-tion of into sub-problems, such that and B have the same Pareto solutions. If Bis a competitive decomposition then, on rare occasions, its Nash equilibria will be thesame as the Pareto solutions of More often, the Nash equilibria of B will be quiteinferior to the Pareto solutions of In contrast, if B is a cooperative decomposition,its Nash equilibria will invariably be the same as the Pareto solutions of

In other words, cooperative decompositions produce better Nash equilibria thancompetitive decompositions. The generalization of this observation is: cooperationmakes possible solutions of higher quality than can be obtained from competition. Wesuspect that this generalization is valid.

The structure of an A-Team allows for cooperation at two levels. First, the problemto be solved is decomposed into sub-problems. The team’s designer can choose tomake this a cooperative decomposition. Second, multiple agents are assigned to eachsub-problem. These agents select and modify solutions from a population of candidate-solutions that is maintained for the sub-problem. The agents work asynchronously, eachusing its own methods for deciding when to work, which solutions to select, and how tomodify them. The agents do not explicitly coordinate their activities with one another,nor is there a central controller. Rather, the agents cooperate through the products oftheir work—by modifying solutions that have been worked on by other agents. Thefinal solution is invariably the result of contributions from many agents. (This form ofcooperation was inspired by the work-styles of social insects—bees, ants and certainwasps. Although these insects live in close-knit colonies, they have no supervisors.Certainly there is a queen, but her function is strictly reproductive; she does not leadthe other colony members, nor does she issue orders to the workers. Although differentcastes exist within the colony—drones, soldiers and workers, for instance—there isno hierarchical relationship among them. Rather, they act as autonomous agents andcooperate through the products of their work. For instance, the construction of the nestproceeds without centralized control or the benefit of a blueprint to show what thefinished result should be; instead, “it is the product of work previously accomplished,rather than direct communication among nestmates, that induces the insects to performfurther labor. Even if the work force is constantly renewed, the nest structure alreadycompleted determines, by its location, its height, its shape and probably also its odor,what further work will be done” [19].)

7 ORGANIZATIONS, SUPER-AGENTS ANDORGANIZATION SPACE

The previous sections have covered some aspects of the structure of A-Teams. Thissection specifies their structure more precisely, and lists the decisions that the designersof A-Teams must make.

Asynchronous Teams 545

Lesser agents can be organized into greater (super) agents, which can be organizedinto still greater agents, and so on, just as cells are organized into organs, which areorganized into humans, which are organized into societies and nations. The capabilitiesof a super-agent depend on its organization, and can range from much less than thesum of the capabilities of its constituent-agents, to very much more. In other words,the design of the organization is at least as important as the choice of agents.

An organization is the “glue” that binds the constituent–agents together. Its pur-pose is two-fold: to divide the labor among the agents, and to coordinate their labor.Structurally, an organization can be thought of as a stack of five networks (Figure 19.3):

1. Control Flow: a tree-like network that divides the agents into layers, showinga) supervisory relationships (who reports to whom), and b) how much auton-omy each agent has. Nodes in this network denote agents. Directed arcs denotesupervisory relationships. A number from zero to one is appended to each arc todenote the degree of control the supervisor exercises over the “supervisee” (thelarger this number, the greater the degree of control).

2. Problem Decomposition: a network that shows the results of decomposingthe problem-to-be-solved, into B, a set of sub-problems. Nodes represent thesub-problems. Arcs represent the couplings among the sub-problems.

3. Sub-Problem Assignment: a bi-partite network that shows which agent isassigned to which sub-problem. Nodes are of two types; one type representsagents, the other, sub-problems. Arcs connect agents to the sub-problems theyhave been assigned.

4. Data Flow: a directed, bipartite network that shows who can “talk” to whom andhow. There are two types of nodes; one type represents agents, the other, datastores. Arcs represent directed communication channels over which the agentscan send messages to one another, post messages in the data stores, or obtainmessages from the data stores. (These stores serve as bulletin boards for theagents connected to them.)

5. Work Schedule: a network that shows the order in which the agents’ tasks—iterations and communications—are to be performed. Nodes in this networkrepresent the tasks. Arcs represent the precedence constraints among the tasks.

546 S. Talukdar et al.

The problem of designing an organization has a very large solution space, namely,the set of all the possible five-network-stacks of the sort shown in Figure 19.3. Of course,other problems, such as aircraft and computer design, also have large solution spaces.But many of these other problems benefit from extensive simulation and verificationfacilities. They can use “generate and test strategies,” i.e., strategies that rely on thequick and accurate evaluation of many candidate-solutions. However, such simulationand verification facilities are not available for organizations. Therefore, it is necessaryto prune their space, leaving a smaller and more easily searched sub-space. The set ofA-Teams is one such sub-space. It is obtained from organization space by:

Setting the control flow network to “null” (the agents in an A-Team are completelyautonomous).

Assigning multiple agents to each sub-problem.

Eliminating all the arcs from the data flow that connect pairs of agents (the agentsin an A-Team communicate only with computer memories, not other agents, andcooperate only by modifying one another’s work).

Making the data flow strongly cyclic, i.e., establishing paths, through agents, bywhich solutions can circulate among the memories.

Setting the work schedule network to “null” (the agents in an A-Team workasynchronously).

Note that each memory in an A-Team contains a population of solutions. All thesolutions in a population are expressed in the same representational form. This rep-resentation can vary from one memory to another. A memory used by humans mayexpress solutions in diagrams, while a memory used by optimization software mayuse vectors. One way to allow for agents that require different representations but mustwork on the same population, is to maintain copies of the population, each in a differentrepresentation and memory.

8 SOLUTION-QUALITY AND CONVERGENCE

The following thought experiment helps explain how A-Teams work and how solution-quality can be improved.

Consider an A-Team that contains only one memory which is dedicated to storinga population of solutions to the problem, This memory is shared by C, a set ofconstruction agents, I, a set of improvement agents, and D, a set of destruction agents.Suppose that the improvement agents may select solutions randomly, but make onlydeterministic modifications to the selected solutions. Suppose that the constructionagents create an initial population of solutions, from which the agents in I and D,working asynchronously, produce a sequence of populations, Supposethat the work is stopped when a population is obtained, such that no agent in I canimprove the quality of the best solution in . What is the quality of And, is Nfinite?

To address these questions, we define a distance metric in terms of iterations byimprovement agents. Consider: a trajectory in S, the solution spaceof Each step, in this trajectory is produced by one iteration of anagent drawn from I. Thus, we can think of the trajectory as being M iterations long.

Asynchronous Teams 547

Of course, there may be other trajectories from to We define the distance fromto as the length, in iterations, of the shortest trajectory from to Note that

the distance from to is: (a) infinite, if there is no trajectory from to and(b) dependent on I.

Let:

be a calculable, scalar measure of the quality of every possiblesolution, s, with q(s) = 1 if s is an optimal solution, and q(s) = 0 if s is verydifferent from an optimal solution.

be the quality of population

G(Q) be the set of all the solutions of quality Q or better, that is,

f(s,g, I) be the distance, in iterations, from to Note: f(s, g, I)is infinite if there is no trajectory from s to g; and if

be the distance, in iterations,from to G(Q). Note: is infinite when I does not contain theskills necessary to transform any point in into a solution of quality Q orbetter.

is finite;

the improvers select solutions randomly, with a bias for solutions of higher quality(the biasing details are given in [13]);

the destroyers select solutions randomly, with a bias towards solutions of lowerquality (the biasing details are given in [13]);

is made non-decreasing with n by saving a copy of the best solution in

then, G(Q) is reachable (the expected value of N is finite, and the expected value ofis Q or better.). The proof can be found in [13,18].

The problem of calculating the distance, is intractable. Therefore, theabove result cannot be used to predict the quality of the solutions that will be producedby an A-Team. But it does tell us that G(Q) will be reachable, if the agents use relativelysimple and random strategies for selection, and if there is at least one trajectory from apoint in to a point in G( Q). Furthermore, the lack of such a trajectory can be remediedby adding construction agents (thereby, changing ) or adding improvement agents(thereby, increasing the number of trajectories emanating from points in and thechances that one of them will pass through a point in G(Q). For instance, if I allows fortrajectories: and and if we happen to augment I with an agent that canbridge the gap from a to b, then a trajectory will be obtained). In other words,solution-quality tends to increase as the number and diversity of the construction andimprovement agents increases.

Solution-quality can also be increased by using better destroyers. (We suspect thatcreation—construction together with improvement—and destruction are duals, andthat adept destruction can compensate for inept creation, and vice-versa.)

If:

S. Talukdar et al.

Of course, increasing the number of agents could increase the total computingtime. But the agents work asynchronously. Therefore, providing more computers, sothe additional agents can work in parallel with all the other agents, is likely to makeany increases in total computing time slight, if not negligible.

In other words, one might expect, from the above analysis, that A-Teams are scale-effective: adding agents to an A-Team tends to improve its solution-quality, and addingcomputers tends to improve its solution-speed. There is some empirical evidence insupport of this conclusion [3], but not enough has to be completely convincing.

9 DESIGN GUIDELINES

The design of A-Teams, like the design of many artifacts, is a craft whose proceduralknowledge does not extend beyond very general guidelines. In such situations, thedesigner has little choice but to build, test and modify prototypes, till one with accept-able behavior is found. Since A-Teams tend to be scale effective, it makes sense to startwith a small team, adding agents when solution-quality is to be improved, and addingcomputers when solution-speed is to be improved.

Some guidelines and observations for designers to keep in mind are:

Problem decomposition: The decomposition of the problem-to-be-solved is crit-ical; only the number and diversity of the agents assigned to each sub-problemaffects overall performance to a greater extent. Let be the problem to be solved,and be a sub-problem. Then, can be of three types:

1.

2.

3.

(Notice that the A-Team of Figure 19.2 contains sub-problems of all three types,while the team of Figure 19.4 contains only types 1 and 3.) All A-Teams shouldcontain at least one sub-problem of type-1 so that solutions to are automaticallyavailable. The quality of the final solutions to this sub-problem can usually beimproved by adding sub-problems of types 2 and 3.

Agent assignment: The greater the variety of skills that are brought to bear ona sub-problem, the greater the quality of the solutions that will be obtained forit. In other words, the solution-quality of an A-Team can be improved by addingagents with new and relevant skills. In making these additions, one should keepin mind that creation and destruction appear to be duals: adept destruction canprobably compensate for inept creation, and vice-versa. In other words, addingadept destroyers is as good as adding adept creators.

Data flow: Empirical evidence suggests that the circulation of solutions amongthe populations has a beneficial effect on both solution-quality and speed.

Population size: In our experience, solution-quality increases with populationsize, but at a rapidly diminishing rate.

is related to just as the 1-Tree problem is related to the travelingsalesman problem (Figure 19.2).

is a component of that is, whereis a set of objectives, C is a set

of constraints and X is a set of decision variables.

548

Asynchronous Teams 549

550 S. Talukdar et al.

Selection strategies: We have experimented with only two selection strategies forimprovement agents: (a) randomly select solutions with a bias that makes thebetter solutions more likely to be selected, and (b) randomly select solutions witha bias towards solutions the agent is more likely to be able to improve. Both seemto work well. For destroyers, we have also tried two strategies: (a) randomly selectsolutions with a bias towards the poorer solutions, and (b) select duplicates. Bothseem to work.

10 A CASE STUDY

This section describes an A-Team, developed at IBM, for scheduling the productionand distribution of paper products.

Paper production is a complex task involving multiple objectives: maximize profit,maximize customer satisfaction, and maximize production efficiency. These objectivesare often in conflict and their relative importance varies with the state of the produc-tion environment and with market conditions. Process interactions further increase thedifficulty of the problem [9]. For instance, a scheduling improvement in one stage ofthe production process may negatively impact downstream processes.

In paper production and distribution, the key decisions are:

(a) Order allocation: Allocating orders to paper machines across multiple paper millsin different geographical locations

(b) Run formation and sequencing: Forming and sequencing batches of similar typesof paper on each paper machine

(c) Trimming: Cutting large reels of paper produced by paper machines into smallerrolls of customer-specified widths and diameters

(d) Load planning: Loading paper rolls onto vehicles for shipment

The traditional approach to paper mill scheduling is to schedule each stage in the processindependently. Typically, paper manufacturers allocate orders to paper machines andsequence them manually. Then they use one software-package for trim scheduling andanother package for outbound logistics scheduling, and so on. Each of these packagesfocuses on a single process-step and attempts to create an optimized schedule basedon local objectives. Since there is no interaction between applications, the completeschedule obtained by combining the sub-schedules is usually of very low quality. Forexample, a trim schedule that minimizes trim-loss may cause vehicles to be loadedinefficiently, unacceptably increasing shipping costs. This piecemeal approach presentsschedulers1 with a single take-it-or-leave-it choice and does not illustrate the tradeoffsbetween competing objectives that are needed to make well informed decisions.

Realizing the shortcomings of the existing approaches, we at IBM Research, havebuilt a new scheduling system that considers all stages of paper production and distribu-tion simultaneously, and generates multiple enterprise-wide schedules. These schedulesare created by algorithms that take into account the interactions between the processstages and focus on enterprise-wide objectives. The algorithms that we have developeduse approaches such as linear programming, integer programming with and without

1 Schedulers are the people who perform the task of scheduling in an organization. The software in ourmodel assists these schedulers in performing their tasks.

Asynchronous Teams 551

randomized rounding, network flow and various heuristic methods. We combine thesemultiple problem-solving approaches in an A-Team to achieve iterative improvementsin the solutions.

For scheduling as well as other manufacturing applications, we have found fourcategories of attributes to be important: Timeliness, Product-Quality, Profitability, andDisruptions. Not coincidentally, these categories reflect the concerns of the peopleaffected by the schedules: customer service representatives, quality engineers, accoun-tants and manufacturing supervisors. As the iterations by the software agents proceed,the schedules with the best tradeoffs among the categories are displayed to the sched-uler. By examining these schedules, the human scheduler gains an understanding ofthe tradeoffs. She can select schedules, drastically modify them, and return them to thepopulations being worked on by software agents. Thereby, she can dramatically alterthe course of the software agents’ calculations. She can also negotiate compromiseswith other interested parties. When she sees a solution she feels is acceptable, sheterminates the calculations.

We cannot overemphasize the importance of intimately involving human schedulesin the calculations. The asynchronous mode of work and a number of filters makesuch involvements possible and practical. Specifically, asynchronous work allows fastsoftware agents to work in parallel with much slower humans. The filters allow onlythe solutions with the best tradeoffs to be viewed by humans, thereby, keeping thehumans from being overloaded with information. The filter most often used allowsonly non-dominated solutions to pass through it.

10.1 An Application

This section describes the construction of an A-Team for solving an instance of apaper-manufacturing problem. This problem consists of more than 5000 orders (whichconstitute about 8 weeks of production for 15 product types) to be scheduled on 9machines that are located in 4 different mills.2 Once orders are allocated to machinesand grouped according to their grades, these groupings (known as runs) have to betrimmed to fit the roll size requirements of the orders. The efficiency of trimming isdependent on the order composition in the groups. However, orders may not be groupedsolely to increase trim efficiency, since such groupings may incur unacceptable delaysof orders with tight deadlines.

A partial set of evaluation metrics used in our implementation, and some of theirvalues, are presented in columns 2–10 of Table 19.1. These evaluation metrics arecustomer dependent and are configurable. We obtained these metrics during the initialdesign study and incorporated them into the system during the benchmarking process.

The manufacturing process contains two distinct stages: (1) run formation andsequencing, and (2) trimming.3 These stages are coupled: the quality of trim optimiza-tion depends on how the runs are formed and sequenced. Therefore, it is important toconsider the overall global optimization problem while generating schedules.

The A-Team we use is depicted in Figure 19.4. First, we create a run formationteam with its own set of constructors, improvers and destroyers for creating runs and

2The data were provided by one of the largest paper manufacturers in the U.S.3A third optimization problem, namely transportation scheduling or load planning, is eliminated from this

analysis for simplicity. A more detailed description of the problem and our solution approach can be foundin [16].

552 S. Talukdar et al.

Asynchronous Teams 553

sequencing the orders within those runs. This team generates a set of non-dominatedschedules4 that serve as starting points for trim optimization. While these solutions canbe evaluated at a high level based on transportation cost, due dates, and order-machinerestrictions, the goodness of these solutions cannot be determined until each run ineach schedule is trimmed and the amount of waste is compared. However, trimmingin itself is a multi-objective optimization. Therefore, we next construct a trim team fortrim optimization with suitable constructors, improvers and destroyers. This trim teamcreates near optimal trim solutions, given a single run and a sequence of orders withinthat run. This trim team can be invoked multiple times to trim each run in a givenschedule. However, there are many such schedules, generated by the run formationteam, that need to be explored for overall trim efficiency. Therefore, in order to explorethe best possible solutions, we employ a third team, the global optimization team,that changes the run formation and sequencing in an effort to achieve better overallsolutions (including better trim) as defined by the evaluation metrics. In essence, therun formation and sequencing team and trim team are super-agents within the globaloptimization team.5 Trimming is computationally intensive. Therefore, it is importantto be judicious in selecting the schedules for trimming. Below, we briefly describethe algorithms that we used in the run formation and sequencing team and the trimteam. The global optimization team uses a combination of constructors, improvers anddestroyers from the run formation team and the trim team.

10.2 Run Formation and Initial Sequencing Stage

In our system, orders are allocated to machines based on considerations such astransportation cost, due dates, trim preferences and order-machine restrictions. Theconstructors, improvers and destroyers used in run formation team are:

Constructors: Many approaches such network flow, dispatch algorithms, lin-ear programming models, and greedy approaches can be used to create initialpopulation of solutions for order sequencing in an A-Team. In solving NP hardproblems such as these, the A-Team framework encourages the use of multipleapproaches for generating initial solutions in the population. Multiple approachescould potentially cover more search space than one approach. In our implemen-tation, we use dispatch algorithms for order allocation. The idea is to select oneorder at a time from the sorted list of remaining orders (several sorting heuris-tics could be used) and schedule it as the next order on a given machine. Thesemethods create partial solutions that have order allocation information, and some-times an initial sequencing of the orders (and hence a sequence of runs) on eachmachine.

Improvers: Improvement algorithms take an existing schedule and try to improveit in several different dimensions. For example, an improver may move ordersbetween runs to reduce tardiness and improve trim efficiency, may merge runsin order to decrease the number of small runs, may resequence runs by movingsubsets of orders in a run to improve the solution of a downstream problem (e.g.,

4A schedule at this point is a sequence of runs in which each run is a grouping of similar orders in aspecific sequence.

5The same approach can be extended to include additional down-stream processes, such as sheeting andtransportation planning.

554 S. Talukdar et al.

moving a set of orders to a different run to improve trim efficiency) etc. Sinceimprovers have inherent knowledge about what aspects of a solution they intendto improve, they are programmed to pick those solutions that exhibit weaknessin those specific aspects. For example, an improver that intends to improve thetardiness of a solution would pick solutions that have many late orders.

Destroyers: In our A-Team we used a simple “delete duplicates” destructionapproach. More intelligent destruction agents could be created as well.

Further details on the algorithms can be found in [8].

10.3 Trim Stage

A paper machine produces large reels of paper. The process of cutting the reels intorolls of paper (based on customer specifications) is called trimming. The main objectivein trimming is to minimize the trim loss (the unused portion of the reel that cannot beused to fill an order) while considering other manufacturing objectives such as on-time delivery and customer satisfaction. This again, is a multi-objective optimizationproblem. The constructors, improvers and destroyers used in the trim team are:

Constructors: Trimming paper rolls can be cast as a one-dimensional cuttingstock problem (CSP). This problem has been studied by Gilmore and Gomory in1961 in their seminal work [11]. Past work in this area [10,12,13] indicates thatlinear and integer-programming models work fairly well for generating initialtrim patterns. Therefore, we use linear programming and integer-programmingapproaches to generate initial trim solutions. However, these solutions can beimproved further by using iterative heuristic techniques.

Improvers: Trim efficiency can be improved by modifying the sequence of orderswithin a run or by exchanging orders with other runs in the schedule. This can bedone either randomly or based on some heuristics that have specific informationabout the required widths that could improve trim. For example, if a paper millknows that there is a constant demand for certain standard widths such as 25" and30", it may not mind making rolls of that size and stocking them, if it improvesthe trim efficiency, and even if there are no immediate orders for those rolls(these are sometimes called “help rolls”). Trim efficiency improvement heuristicscan embody these types of domain details to improve overall trim efficiency.In deciding which solutions to improve, we use simple selection mechanismssuch as random selection with a bias. For example, improvers that specialize inimproving trim efficiency are programmed to randomly pick those solutions fromthe populations that do not have perfect trim.

Destroyers: In our trim team we used a simple “delete duplicates” destructionapproach. More intelligent destruction agents could be created as well.

More details on the algorithms can be found in [14].The global A-Team in this application consisted of 15 constructors and 5 improvers.

Each agent (constructor or an improver) is an embodiment of the algorithms describedabove (run with various parameter settings). For our sample problem, an A-Teaminvocation of of CPU time on a single processor IBM RS/6000 Model 591 (256MBof memory generated approximately 100 solutions; of these solutions, 10 solutions werenon-dominated. Table 19.1 shows the evaluations for these 10 solutions illustrating

Asynchronous Teams 555

the tradeoffs among the objectives. For example, solutions 3 and 4 have the sametransportation cost, which suggests that they have almost the same allocation of ordersto mills. However, they differ significantly in their tardiness and trim efficiencies.Solution 3 sacrifices order tardiness for better trim while solution 4 sacrifices trim forbetter on-time order delivery.

Comparison of solution 10 with the schedule generated by our customer by theirtraditional methods showed that our system could provide significant reductions incosts (6% savings in transportation costs and improvements in customer satisfactionthrough reduced tardiness were reported). In this company, as in many other papercompanies, an experienced team of schedulers worked on generating the traditionalschedules. They allocated and sequenced orders manually or by using a stand-alonescheduling program, and used another computer program for trimming. For fine-tuningthe schedules, they used numerous heuristics they had developed over the years.

10.4 Business Impact

Our paper mill scheduling system has been fielded at several paper mills in the UnitedStates and is being used in their day-to-day operations. The system significantlyimproves the scheduling and decision making process for our customers, giving themsubstantial monetary returns. Improvements come both from the higher quality of thesolutions that our system generates and from positive changes in the business processesthat our approach to decision support fosters. In terms of solution quality, one of ourcustomers, Madison Paper Industries, reports a reduction in trim loss by 6 tons per dayand a 10% reduction in freight costs [17]. Each of these savings amounts to millionsof dollars per year.

Adam Stearns of Madison Paper Industries, our pilot customer, reports “We woulduse our old trim package, throw orders into it and let it trim them the best it could.Then we would let the IBM module take the same orders and manipulate them tocome up with a trim. We saw that the IBM package was consistently saving over twoinches [of trim loss]—just an incredible amount.” [17, page 74] “Testing shows thatwe are getting about 10% savings annually on distribution costs from the load planningpiece alone, which amounts to millions of dollars... .We expected the system’s GUI[graphical user interface] to make load planning easier, but we didn’t expect to gainthese efficiencies.” [17].

By 1999, 31 mills were either using or were in the process of implementing the IBMmill scheduling system. The users are primarily roll manufacturers in the corrugatedand publishing paper markets.

In summary, we have developed a system for enterprise-wide scheduling of papermanufacturing and distribution using the iterative improvement framework facilitatedby Asynchronous Teams. The A-Team architecture facilitated cooperation between thevarious computer algorithms and the human scheduler, resulting in better solutionsthan any one of the implemented algorithms could have achieved by working alone.

REFERENCES

[1] P.E. Gill and W. Murray (eds.) (1974) Numerical Methods for ConstrainedOptimization. Academic Press.

556 S. Talukdar et al.

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

S.S. Pyo (1985) Asynchronous Procedures for Distributed Processing. Ph.D.dissertation, CMU.

P.S. deSouza (1993) Asynchronous Organizations for Multi-Algorithm Problems.Ph.D. dissertation, CMU.

S. Sachdev (1998) An exploration of A-teams. Ph.D. dissertation, CMU.

A. Smith (1776) The Wealth of Nations.

G. Debreu (1959) The Theory of Value. Wiley, New York.

S.N. Talukdar (1999) Collaboration rules for autonomous software agents. TheInternational Journal of Decision Support Systems, 24, 269–278.R. Akkiraju, P. Keskinocak, S. Murthy and F. Wu (1998) An agent-based approachfor multi-machine scheduling. Proceedings of the Tenth Annual Conference onInnovative Applications of Artificial Intelligence, Menlo Park, CA.

C. Biermann (1993) Essentials of Pulping and Papermaking. Academic Press,San Diego.H. Dyckhoff (1990) A typology of cutting and packing problems. EuropeanJournal of Operational Research, 44, 145–159.

P.C. Gilmore and R.E. Gomory (1961) A linear programming approach to thecutting stock problem. Operations Research, 9, 849–859.

R.W. Haessler (1980) A note on the computational modifications to the Gilmore-Gomory cutting stock algorithm. Operations Research, 28, 1001–1005.

R.W. Haessler and P.E. Sweeney (1991) Cutting stock problems and solutionprocedures. European Journal of Operational Research, 54, 141–150.P. Keskinocak, F. Wu, R. Goodwin, S. Murthy, R. Akkiraju, S. Kumaran andA. Derebail (2002) Scheduling solutions for the paper industry. OperationsResearch, 50(2), 249–259.

S. Murthy (1992) Synergy in cooperating agents: designing manipulatorsfrom task specifications. Ph.D. thesis. Carnegie Mellon University, Pittsburgh,Pennsylvania.

S. Murthy, R. Akkiraju, R. Goodwin, P. Keskinocak, J. Rachlin, F. Wu,S. Kumaran, J. Yeh, R. Fuhrer, A. Agarwal, M. Sturzenbecker, R. Jayaramanand R. Daigle (1999) Cooperative multi-objective decision-support for the paperindustry. Interfaces, 29, 5–30.M. Shaw (1998) Madison streamlines business processes with integrated infor-mation system. Pulp & Paper, 72(5), 73–81.

S. Talukdar, L. Baerentzen, A. Gove and P. deSouza (1998) Asynchronous teams:cooperation schemes for autonomous agents. Journal of Heuristics, 4, 295–321.

P.P. Grassé (1967) Nouvelle expériences sur le termite de Muller (macrotermesmülleri) et considérations sur la théorie de la stigmergie. Insectes Sociaux, 14(1),73–102.