THE PERTURBATION ANALYSIS STORY: 1979 - 2017 · Christos G. Cassandras CODES Lab. - Boston University THE PERTURBATION ANALYSIS ... WHENEVER bIS OBSERVED: If aactivated with new lifetime

C. G. CassandrasDivision of Systems Engineering

Dept. of Electrical and Computer Engineering

Center for Information and Systems Engineering

Boston University

https://christosgcassandras.org

Christos G. Cassandras CODES Lab. - Boston University

THE

PERTURBATION ANALYSIS

STORY:

1979 - 2017

OUTLINE


▪ 1979: How a “real problem” gives birth to

… an academic discipline

… a research field

… a toolbox for practical problem solutions

▪ 1979-1999: Development chronology in

Perturbation Analysis (PA)

▪ 2002: Re-inventing Infinitesimal Perturbation Analysis (IPA)

for Hybrid Systems

▪ 2017: The IPA Calculus, Event-Driven control theory,

Data-driven methods come of age


THE PROBLEM (circa 1978)

What is the best way to distribute BUFFER capacity

in a manufacturing transfer line?

PARTS

INPRODUCTS

OUT

BUFFERS

SERIAL OPERATIONS

Ho, Eyler, Chien,

Cassandras,

Cao,…



PARTS

INPRODUCTS

0UT

BUFFERS

SLOW… PRETTY FAST…

What is the best way to distribute BUFFER capacity

in a manufacturing transfer line?



Complexity of this buffer allocation process

(K buffers, N stages)

K

NK 1

Example: K = 24, N = 6 → 118,755 possible allocations

• “Brute Force” trial-and-error: test each allocation for about a week to

get statistically meaningful results

(that’s if the manager allows you to mess with the system…)

→ about 2300 years…

• Suppose you can reduce to only 1000 “promising” allocations:

→ about 19 years…

• Using a simulated transfer line, about 3 minutes per trial (1980s

computing technology…)

→ about 250 days…

THROUGHPUT

Manufacturing system with N sequential operations:

… … ……

DELAY

l(t)

THROUGHPUT increases(GOOD)

INCREASE l(t)

average DELAY increases

(BAD)THROUGHPUT

AV. DELAY

SY

ST

EM

CA

PA

CIT

Y

WHY IS THIS PROBLEM IMPORTANT ?


SWEET SPOT

CONTROLLED BY

BUFFER ALLOCATION


TWO KEY OBSERVATIONS

1. This is a dynamic system.

But it’s not like the usual TIME-DRIVEN ones, i.e., described by

differential equations

),,( tuxfdt

dx

Need a NEW modeling framework for these EVENT-DRIVEN systems

→ DISCRETE EVENT DYNAMIC SYSTEMS

2. You don’t need brute-force trial-and-error for each allocation…

Once the system dynamics are understood, you can predict

what happens by changing allocations

(adding, removing, moving buffers)

→ PERTURBATION ANALYSIS THEORY


ARRIVAL 1

ARRIVAL 2

ARRIVAL 3

A B C

SYSTEM DYNAMICS:

HOW ONE COMPONENT OF THE SYSTEM

AFFECTS OTHER COMPONENTS

THE PROBLEM: SYSTEM DYNAMICS


DEPARTURE 1 FROM A



DEPARTURE 1 FROM B

ARRIVAL 4IDLING



DEPARTURE 2 FROM A



DEPARTURE 2 FROM B

DEPARTURE 3 FROM A



DEPARTURE 3 FROM B

BLOCKINGARRIVAL 5

WHAT IF THIS

HAD BEEN ADDED?

RECORD BLOCK START TIME: DB(3)

PERTURBATION

ANALYSIS



DEPARTURE 4 FROM A



DEPARTURE 1 FROM C

BLOCKING ENDS

RECORD BLOCK END TIME: Dc(1)

PART SYSTEM DELAY WOULD HAVE

BEEN REDUCED BY: Dc(1) - DB(3)

PERTURBATION

ANALYSIS


SYSTEMDesigns, Options,

Policies, ParametersPerformance

Measures

CONVENTIONAL TRIAL-AND-ERROR ANALYSIS

• Repeatedly change parameters/operating policies

• Test different conditions

• Answer multiple WHAT IF questions

LEARNING BY TRIAL AND ERROR


N “What-If” questions N+1 trials !

ANSWERS TO MULTIPLE “WHAT IF” QUESTIONS AUTOMATICALLY PROVIDED FROM A SINGLE TRIAL

SYSTEMPerformance

Measures


LEARNING WITH PERTURBATION ANALYSIS

Designs, Options,

Policies, Parameters

WHAT IF…• Parameter p1 = a were replaced by p1 = b

• Design option 1 were replaced by option 2••

Performance Measures

under all WHAT IF Questions

PERTURBATIONANALYSIS

• Automatica (reject)

“Automata have no place in control engineering”

• Math. Systems Theory (reject)

“FSM and regular languages are nothing new at best and trivial at

worst”

• SIAM J. Control and Optimization (accept)

“If it’s optimal control we’ll take it”

Anonymous referee comments for 1983 papers on

Supervisory Control: (courtesy W.M. Wonham)

W.M. Wonham’s conclusion:

“Crossing cultural divides can be a chilly business”


DES CONTROVERSY IN THE 1980s…

1

2Observed

with

K = 2 buffers

1

2 Perturbed

with

K = 1 buffers

[THOUGHT EXPERIMENT]

Dx = 0 Dx = -1 Dx = 0

x(t+) = 0

Dx = 0

BUFFER MACHINE

PartArrivals

PartDepartures

x(t): SYSTEM CONTENT

x(t)

LEARNING THROUGH PERTURBATION ANALYSIS


1 2 4

1 2 3 4

x(t+) = 2

Dx = -1

DERIVATIVE ESTIMATION:INFINITESIMAL PERTURBATION ANALYSIS (IPA)

Christos G. Cassandras CISE - CODES Lab. - Boston University

“Brute Force” Derivative Estimation:

q )(ˆ qJSYSTEM

SYSTEMqDq J q q D]

dJ

d

J J

estq

q q q

q

( ) ( )D

D

DRAWBACKS: • Intrusive: actively introduce perturbation Dq

• Computational cost: 2 observation processes

[ (N+1) for N-dim q ]

• Inherently inaccurate: Dq large poor derivative approx.

Dq small numerical instability

Infinitesimal Perturbation Analysis

(IPA):q J qSYSTEM

IPAestd

dJ

q


Ho, Y.C., M.A. Eyler, and D.T. Chien, “A Gradient Technique for General Buffer Storage Design in a Serial Production Line,” Intl. Journal

of Production Research, Vol. 17, pp. 557-580, 1979.

Ho, Y.C., and C.G. Cassandras, “A New Approach to the Analysis of Discrete Event Dynamic Systems,” Automatica, Vol. 19, No. 2, pp.

149-167, 1983.

Ho, Y.C., X. Cao, and C.G. Cassandras, “Infinitesimal and Finite Perturbation Analysis for Queueing Networks,” Automatica, Vol. 19, pp.

439-445, 1983.

Cao, X., “Convergence of Parameter Sensitivity Estimates in a Stochastic Experiment,” IEEE Transactions on Automatic Control, Vol. 30,

pp. 834-843, 1985.

Cassandras, C.G., Wardi, Y., Panayotou, C.G., and Yao, C., “Perturbation Analysis and Optimization of Stochastic Hybrid Systems”,

European Journal of Control, Vol. 16, No. 6, pp. 642-664, 2010

• Ho, Y.C., and X. Cao, Perturbation Analysis of Discrete Event Dynamic Systems, Kluwer Academic Publisher, 1991.

• Glasserman, P., Gradient Estimation via Perturbation Analysis, Kluwer Academic Publishers, Boston, 1991

• Cassandras, C.G., Discrete Event Systems: Modeling and Performance Analysis, Irwin Publ. 1993.

• Cao, X., Realization Probabilities: The Dynamics of Queueing Systems, Springer-Verlag, 1994.

• Glasserman, P., and D.D. Yao, Monotone Structure in Discrete Event Systems, Wiley, 1994.

• Fu, M.C., and J.Q. Hu, Conditional Monte Carlo: Gradient Estimation and Optimization Applications, Kluwer Academic Publ., 1997.

• Cao, X., Stochastic Learning and Optimization – A Sensitivity-Based Approach, Springer, 2007.

• Cassandras, C.G, and S. Lafortune, Introduction to Discrete Event Systems, 2nd Edition, Springer, 2008.

Ramadge, P.J., and W.M. Wonham, "The control of discrete event systems," Proceedings of the IEEE, Vol. 77, No. 1, pp. 81--98,1989.

David, R., and H. Alla, Petri Nets & Grafcet: Tools for Modelling Discrete Event Systems, Prentice-Hall, New York, 1992.

Moody, J.O., and P.J. Antsaklis, Supervisory Control of Discrete Event Systems Using Petri Nets, Kluwer Academic Publishers, 1998.

SAMPLE BIBLIOGRAPHY

SELECTED BOOKS

INFINITESIMAL

PERTURBATION

ANALYSIS

(IPA)


INFINITESIMAL PERTURBATION ANALYSIS (IPA)


1. INITIALIZATION: If a feasible at x0: q

a

ad

dV 1,:D

Else for all other a: 0:Da

2. WHENEVER b IS OBSERVED:

If a activated with new lifetime Va :

2.1. Compute dVa/dq

2.2. q

aba

d

dVDD :

INFINITESIMAL PERTURBATION ANALYSIS (IPA)


QUESTION: Are IPA sensitivity estimators “good” ?

▪ Unbiasedness

▪ Consistency

ANSWER: Yes, for a large class of DES that includes

▪ G/G/1 queueing systems

▪ Jackson-like queueing networks (e.g., no blocking allowed)

NOTE: IPA applies to REAL-VALUED parameters of some event process distribution (e.g., mean interarrival and service times)

IPA UNBIASEDNESS


…provided interchange of E and d/dq is possible !

)()]([ q

qq

qqTT

Ld

dELE

d

d

d

dJ

IPA ESTIMATOR

1979

Ho, Y.C., M.A. Eyler, and D.T. Chien, “A Gradient Technique for General Buffer

Storage Design in a Serial Production Line,” Intl. Journal of Production Research,

Vol. 17, pp. 557-580, 1979.

M.A. Eyler

1982

C.G. Cassandras

Ho, Y.C., and C.G. Cassandras, “A New Approach to the Analysis of Discrete

Event Dynamic Systems,” Automatica, Vol. 19, No. 2, pp. 149-167, 1983.

1984

Ho, Y.C., X. Cao, and C.G. Cassandras, “Infinitesimal and Finite Perturbation

Analysis for Queueing Networks,” Automatica, Vol. 19, pp. 439-445, 1983.

X. Cao

Cao, X., “Convergence of Parameter Sensitivity Estimates in a Stochastic Experiment,”

IEEE Transactions on Automatic Control, Vol. 30, pp. 834-843, 1985.

1986

M. Zazanis

Suri, R., and M. Zazanis, “Perturbation Analysis Gives Strongly Consistent Sensitivity

Estimates for the M/G/1 Queue,” Management Science, Vol. 34, pp. 39-64, 1988.

Heidelberger, P., X. Cao, M. Zazanis, and R. Suri, “Convergence Properties of

Infinitesimal Perturbation Analysis Estimates,” Management Science, Vol. 34,

No. 11, pp. 1281-1302, 1988.

1988 1989

Glasserman, P., and W.B. Gong, “Smoothed Perturbation Analysis for a Class of Discrete Event

Systems,” IEEE Transactions on Automatic Control, Vol. AC-35, No. 11, pp. 1218-1230, 1990.

P. Glasserman

Glasserman, P., Gradient Estimation via Perturbation Analysis, Kluwer Academic

Publishers, Boston, 1991.

Ho, Y.C., and S. Li, “Extensions of Perturbation Analysis of

Discrete Event Dynamic Systems,” IEEE Transactions on

Automatic Control, Vol. AC-33, pp. 427-438, 1988.

S. Li

Vakili, P., and Y.C. Ho, “Infinitesimal Perturbation Analysis of a Multiclass Routing Problem,” Proceedings of

25th Allerton Conference on Communication, Control, and Computing, pp. 279-286, 1987.

P. Vakili

1987

W. Gong

Gong, W.B., and Y.C. Ho, “Smoothed Perturbation Analysis of Discrete

Event Systems,” IEEE Transactions on Automatic Control, Vol. AC-32,

No. 10, pp. 858-866, 1987.

M. Fu

Fu, M.C., “Convergence of a Stochastic Approximation Algorithm for the GI/G/1

Queue Using Infinitesimal Perturbation Analysis,” Journal of Optimization Theory

and Applications, Vol. 65, pp. 149-160, 1990.

Cassandras, C.G., and S.G. Strickland, “Observable Augmented Systems for

Sensitivity Analysis of Markov and Semi-Markov Processes,” IEEE Transactions

on Automatic Control, Vol. AC-34, No. 10, pp. 1026-1037, 1989

19911990

J.Q. Hu

Ho, Y.C. and Hu, J.Q., "An Infinitesimal Perturbation Analysis Algorithm for a

Multiclass G/G/1 Queue," Operations Research Letters, 9, pp.35-44, 1990.

Hu, J.Q. and Strickland, S.G., "Strong Consistency of Sample

Path Derivative Estimates," Applied Mathematics Letters, Vol. 3,

No. 4, pp. 55-58, 1990.

Wardi, Y., and J.Q. Hu, “Strong Consistency of Infinitesimal

Perturbation Analysis for Tandem Queueing Networks,” Journal of

Discrete Event Dynamic Systems, Vol. 1, No. 1, pp. 37-60, 1991.

Ho, Y.C., and X. Cao, Perturbation Analysis of Discrete Event Dynamic Systems, Kluwer Academic Publisher, 1991.

Vakili, P., “A Standard Clock Technique for Efficient Simulation,” Operations Research Letters, Vol. 10, pp. 445-452, 1991.

1992

Chong, E.K.P., and P.J. Ramadge,, “Convergence of Recursive Optimization Algorithms

Using Infinitesimal Perturbation Analysis Estimates,” Journal of Discrete Event Dynamic

Systems, Vol. 1, No. 4, pp. 339-372, 1992.

Bremaud, P., and F.J. Vazquez-Abad, “On the Pathwise Computation of Derivatives

with Respect to the Rate of a Point Process: The Phantom RPA Method,”

Queueing Systems, Vol. 10, pp. 249-270, 1992.

L. Shi

Ho, Y-C, Shi, L., Dai, L., and Gong, W-B. “A New

Method for Optimizing Complex Networks,” 30th

IEEE conference on Decision and Control, 1991,

pp. 105-109.

Shi, L. “Variance Property of Discontinuous

Perturbation Analysis,” Winter Simulation

Conference, Dec. 1996, pp. 412-417.

1993

L. Dai

Dai, L., and Y.C. Ho, “Structural Infinitesimal Perturbation Analysis (SIPA) for

Derivative Estimation of Discrete Event Dynamic Systems,” IEEE Transactions

on Automatic Control, Vol. 40, pp. 1154-1166, 1995.

Cassandras, C.G., Discrete Event Systems: Modeling and

Performance Analysis, Irwin Publ. 1993.

1994

Cao, X., Realization Probabilities: The Dynamics of Queueing Systems,

Springer-Verlag, 1994.

1997

Fu, M.C., and J.Q. Hu, Conditional Monte Carlo: Gradient Estimation and

Optimization Applications, Kluwer Academic Publishers, Boston, 1997.

1999

Cassandras, C.G., and C.G. Panayiotou, “Concurrent Sample Path Analysis of

Discrete Event Systems,” Journal of Discrete Event Dynamic Systems, 1999.

2002

Cassandras, C.G., Wardi, Y., Melamed, B., Sun,

G., and Panayiotou, C.G., "Perturbation Analysis

for On-Line Control and Optimization of

Stochastic Fluid Models", IEEE Trans. on

Automatic Control, AC-47, 8, pp. 1234-1248, 2002

2010

Cassandras, C.G., Wardi, Y., Panayotou, C.G., and

Yao, C., “Perturbation Analysis and Optimization of

Stochastic Hybrid Systems”, European Journal of

Control, Vol. 16, No. 6, pp. 642-664, 2010.


IPA RE-BORN: HYBRID SYSTEMS

2017

Yao, C, and Cassandras, C.G., “A Solution to the Optimal Lot Sizing

Problem as a Stochastic Resource Contention Game”, IEEE Trans. on

Automation Science and Engineering, Vol. 9, No. 2, pp. 250-264, 2012.

Geng, Y., and Cassandras, C.G., “Multi-intersection Traffic

Light Control with Blocking”, J. of Discrete Event Dynamic

Systems, Vol. 25, 1, pp. 7-30, 2015.

Fleck, J.L., and Cassandras, C.G., “Optimal Design of

Personalized Prostate Cancer Therapy using Infinitesimal

Perturbation Analysis”, Nonlinear Analysis: Hybrid Systems,

Vol. 25, pp. 246-262, 2017.

Cassandras, C.G., Lin, X., and Ding X.C. “An

Optimal Control Approach to the Multi-agent

Persistent Monitoring Problem”, IEEE Trans. on

Automatic Control, AC-58, 4, pp. 947-961, 2013.

THE IPA CALCULUS

q

q

q

q

k

k

txtx ,

,NOTATION:

HYBRID AUTOMATA


HYBRID AUTOMATA

Q: set of discrete states (modes)

X: set of continuous states (normally Rn)

E: set of events

U: set of admissible controls

),,,,,,,,,,( 00 xqguardInvfUEXQGh

f : vector field,

: discrete state transition function,

XUXQf :

QEXQ :

Inv: set defining an invariant condition (domain),

guard: set defining a guard condition,

: reset function,

q0: initial discrete state

x0: initial continuous state

XQInv

XQQguard

XEXQQ :

HYBRID AUTOMATASTOCHASTIC HYBRID AUTOMATA

), ,( txfx k

Event at time k(q) Event at time k+1(q)

kth discrete state (mode)

q : control parameter, (system design parameter,

parameter of an input process,

or parameter that characterizes a control policy)

q

q


IPA: THREE FUNDAMENTAL EQUATIONS

1. Continuity at events:

Take d/dq :

)()( kk xx

kkkkkkk ffxx ')]()([)(')(' 1

If no continuity, use reset condition q

d

xqqdx k

),,,,()('


System dynamics over (k(q), k+1(q)]: ),,( txfx k q

q

q

q

q

k

k

txtx ,

,NOTATION:


Solve over (k(q), k+1(q)]:q

)()('

)()(' tftx

x

tf

dt

tdx kk

t

k

dux

uf

kdu

x

uf

k

v

k

kt

k

k

xdvevf

etx

q

)()(

)(

)()(

initial condition from 1 above


2. Take d/dq of system dynamics over (k(q), k+1(q)]:),,( txfx k q

q

)()('

)()(' tftx

x

tf

dt

tdx kk

NOTE: If there are no events (pure time-driven system),

IPA reduces to this equation

3. Get depending on the event type:


k

- Exogenous event: By definition, 0k

0)),,(( qq kk xg- Endogenous event: occurs when

)()(

1

kkkk xx

ggf

x

g

q

- Induced events:

)()(

1

kkkk

k yt

y


Ignoring resets and induced events:


kkkkkkk ffxx ')]()([)(')(' 1

t

k

dux

uf

kdu

x

uf

k

v

k

kt

k

k

xdvevf

etx

q

)(')(

)(

)()(

0k

)()(

1

kkkk xx

ggf

x

g

qor

1.

2.

3.


21

3

)('

kx

q

q

q

q

kk

txtx

,

Recall:

Cassandras et al, Europ. J. Control, 2010

IPA PROPERTIES


1. ROBUSTNESS TO NOISE:

Performance sensitivities can often be obtained from information

limited to event times, which is easily observed

No need to track system in between events !

2. DECOMPOSABILITY:

Performance sensitivities are often reset to 0

sample path can be conveniently decomposed

3. SCALABILITY:

IPA estimators are EVENT-DRIVEN

IPA scales with the EVENT SET, not the STATE SPACE !

TIME-DRIVEN v EVENT-DRIVEN CONTROL


REFERENCE

PLANTCONTROLLERINPUT

-

+

SENSOR

MEASURED

OUTPUT

OUTPUTERROR

REFERENCE

PLANTCONTROLLERINPUT

-

+

SENSOR

MEASURED

OUTPUT

OUTPUTERROR

EVENT:

g(STATE) ≤ 0

EVENT-DRIVEN CONTROL: Act only when needed (or on TIMEOUT) - not based on a clock

SELECTED REFERENCES - EVENT-DRIVEN CONTROL


- Astrom, K.J., and B. M. Bernhardsson, “Comparison of Riemann and Lebesgue sampling for

first order stochastic systems,” Proc. 41st Conf. Decision and Control, pp. 2011–2016, 2002.

- T. Shima, S. Rasmussen, and P. Chandler, “UAV Team Decision and Control using Efficient

Collaborative Estimation,” ASME J. of Dynamic Systems, Measurement, and Control, vol. 129,

no. 5, pp. 609–619, 2007.

- Heemels, W. P. M. H., J. H. Sandee, and P. P. J. van den Bosch, “Analysis of event-driven

controllers for linear systems,” Intl. J. Control, 81, pp. 571–590, 2008.

- P. Tabuada, “Event-triggered real-time scheduling of stabilizing control tasks,” IEEE Trans.

Autom. Control, vol. 52, pp. 1680–1685, 2007.

- J. H. Sandee, W. P. M. H. Heemels, S. B. F. Hulsenboom, and P. P. J. van den Bosch, “Analysis

and experimental validation of a sensor-based event-driven controller,” Proc. American Control

Conf., pp. 2867–2874, 2007.

- J. Lunze and D. Lehmann, “A state-feedback approach to event-based control,” Automatica, 46,

pp. 211–215, 2010.

- P. Wan and M. D. Lemmon, “Event triggered distributed optimization in sensor networks,”

Proc. of 8th ACM/IEEE Intl. Conf. on Information Processing in Sensor Networks, 2009.

- Zhong, M., and Cassandras, C.G., “Asynchronous Distributed Optimization with Event-Driven

Communication”, IEEE Trans. on Automatic Control, AC-55, 12, pp. 2735-2750, 2010.

CONCLUSIONS

1979: Buffer allocation problem Basic PA technique

UNBIASEDNESS ???

IPA Calculus,

Stochastic Hybrid Systems

Complexity management in optimization and control

Continuous-parameter optimization IPA

Alternative PA techniques:

SPA, RPA, cut-and-paste, etc

NO

Large-scale system optimization

YES

Realization Factors

COMPLEXITY

Documents

THE PERTURBATION ANALYSIS STORY: 1979 - 2017 · Christos G. Cassandras CODES Lab. - Boston University THE PERTURBATION ANALYSIS ... WHENEVER bIS OBSERVED: If aactivated with new lifetime