BOOKSHELF The Book - IEEE Control Systems Society

96

BOOKSHELF

December 20051066-033X/05/$20.00©2005IEEE

IEEE Control Systems Magazine

In this issue we bring you reviewsof four books covering diverseaspects and applications of

advanced control. The review byJankovic of Guzzella and Onders’Introduction to Modeling and Control ofInternal Combustion Engine Systemsdiscusses the numerous issues thatarise in controlling modern enginesystems. The next review, by Jad-babaie, provides an overview of con-vex optimization and its applications,the subject of a recent book by Boydand Vandenberghe. The third review,by Sarangapani, of Neural Engineeringby Eliasmith and Anderson, discussesthe control engineering applicationsof concepts from this biologicallyinspired area of research. Finally, thereview by Borkar of Yin and Zhang’sDiscrete Markov Chains explains theintricacies of two-time-scale models.

—Kirsten Morris

Introduction to Modeling and Con-trol of Internal Combustion EngineSystems by L. Guzzella and C.H.Onder, Springer-Verlag, 2004, ISBN:354022274X, US$79.95. Reviewed byMrdjan J. Jankovic.

The BookThe topic of this book is modelingand control of internal combustionengines for automotive applications.With global automotive annual salesof around 60 million vehicles, internalcombustion engines impact theworld’s oil consumption and local airquality. In developed markets suchas the United States, government reg-ulations on tailpipe emissions andfuel consumption have been in exis-tence for more than 30 years. Regula-tions and customer preferenceshave induced significant develop-ments in engine hardware andexhaust gas after-treatment systems.These advances have been matchedby progress in the theory and prac-tice of engine control. For example,achieving tailpipe emission reduction

of two orders of magnitude is madepossible by catalytic converters thattreat the engine exhaust gas. Efficientoperation of these after-treatmentsystems critically depends on accu-rate management of the in-cylinderair-fuel ratio, which is accomplishedby increasingly sophisticated controlsystems. To facilitate the productionimplementation of evermore complexcontrol systems, the computingpower and memory size of power-train electronic control units isincreasing by about an order of mag-nitude per decade [4].

On the fuel economy side, various“tuning” devices, such as the exhaustgas recirculation (EGR) system, vari-able valve timing and lift, and variabledisplacement (also called displace-ment-on-demand), have been addedto reduce fuel consumption. In itsbase configuration, a spark ignition(gasoline) engine needs to controlonly air, fuel, and spark timing torespond to the driver’s accelerationdemand and meet fairly strict emis-sion standards. In the process of baseengine hardware design, parameterssuch as valve timing—the relativephase between the crankshaft, themain shaft that rotates powered bythe engine pistons, and thecamshafts, which control openingand closing of the intake and exhaustvalves—are fixed at values that pro-vide a reasonable compromisebetween operating conditions. Byallowing these parameters to varywith the operating condition, addi-tional fuel economy and performance(torque and power) improvementscan be achieved. Similarly, in a dieselengine, the optimized parametersmay include the air-fuel ratio, percentof recirculated exhaust gas, fractionsof fuel in multiple fuel-injection puls-es, and dwell times between pulses.

The set points for the optimizedparameters are not known a priori.These values are obtained after anelaborate and time-consuming map-ping, optimization, and calibration(fine-tuning) process. The task of the

control system is to regulate the opti-mized parameters to set points thatvary with operating conditions and toremove undesirable transient sideeffects. Online determinations of theset points and “decoupling” of theeffects rely substantially on feedfor-ward control (see [1] for an example).In addition, many set-point regulationcontrol loops in an engine are affectedby transport delays, which reduce theeffectiveness of the feedback andnecessitate the use of a feedforwardcomponent. For feedforward compen-sation, the availability of accuratecontrol-oriented models is essential.This fact provides a solid justificationfor the material selection by theauthors. More than half of Introductionto Modeling and Control of InternalCombustion Engine Systems is devotedto engine subsystem modeling.

Despite the rich journal and con-ference proceedings literature onengine controls, there are few textsthat provide a broad, introductoryexposition to the subject. The onlyother textbook that this reviewer isaware of is [2], which presents abroader set of automotive controlproblems, including not only engine

97December 2005 IEEE Control Systems Magazine

controls but also the driveline andvehicle dynamics. In fact, most of thematerial in [2] is devoted to the lasttwo topics. Thus, the book by Guzzel-la and Onder can and does provide amore detailed account of enginemodeling and control. However, per-sons interested in this subject canbenefit from reading both textsbecause there is less overlap thanone would expect.

Contents of the Book The book is divided into three parts.The first part provides a brief intro-duction to the engine control prob-lem, while the two remaining partsfocus on the modeling of engine andafter-treatment subsystems as well ascontrol design.

Models of EngineSubsystemsThe modeling part of the book isdivided into two chapters accordingto the nature of the engine modelsconsidered. Chapter 2 considersmean value models (MVM), whichare control-oriented models thatneglect the discrete nature of theengine cycle and consider evolutionof variables (states) to be continu-ous in an average sense over thecycle. In contrast, a discrete-eventmodel (DEM) of an engine systemexplicitly takes into account the dis-crete engine events. DEMs are con-sidered in chapter 3.

MVMs are the staple of control-ori-ented engine modeling. Chapter 2provides an in-depth account of suchmodels for diesel and gasolineengines. Models of air and fuel deliv-ery, torque generation, thermal sys-tems, pollution formation, andafter-treatment are discussed indetail. When the discrete nature ofthe engine cycle plays a crucial rolein understanding a phenomenon oran effect, DEMs are more appropriate.Chapter 3 presents DEMs of mean-torque production, air and fuel flows,in-cylinder residual gas, and in-cylin-der pressure.

The air system MVMs, which arefairly standard, include the manifoldfilling dynamics derived from theideal gas law PV = mRT , the throttle(sharp edge orifice) flow model, andthe engine volumetric efficiency.Although a formula for the volumetricefficiency is provided by (2.22), inthis reviewer’s experience, it is morelikely that a characterization isobtained based on experimentalengine mapping data. The bonus ofthis section (section 2.3) is a detailedturbocharger model that contains theMoore-Greitzer compressor surgephenomenon. The correspondingDEM models of chapter 3 include astandard DEM for mass airflow intothe cylinders (section 3.2.2). For com-pressed natural gas engines, theeffects of a stratified (that is, nonho-mogeneous) air-fuel mixture and theback flow of gas into the port causedby late intake valve closing are alsoincluded (section 3.2.4).

The fuel system model is focusedon the wall-wetting effect, that is, thepropensity of injected fuel to stick tothe intake port walls (and intake valvesurface if the engine is cold) and formpuddles that evaporate at a certainrate. Because of puddle formation andevaporation, the mass of fuel that endsup in the cylinder is different from themass of injected fuel. For accurate air-fuel ratio control, it is important tomodel these effects and provide com-pensation. The first principles modelin section 2.4 is based on new resultsby the second author [3]. From thecontrol point of view, it is important totrack wall wetting cylinder by cylinder.A DEM of the fuel flow dynamics,based on multiplexing the continuous-time dynamics, is presented in sec-tion 3.2.3.

The torque generation section(section 2.5), which provides ampleillustration of the problem of engineoptimization, shows that the torquedepends on most engine variables(such as speed, manifold pressure,spark timing, air-fuel ratio, and EGRfraction). Thermal system models

(section 2.6) are relevant because ofthe desire to achieve quick light-off ofcatalytic converters after a cold startand maintain their temperature in thedesired range.

I found the pollutant formationand abatement sections (sections 2.7and 2.8) to be particularly informa-tive. Formation of the three mainregulated gases, namely, CO, non-methane organic gases (NMOG) (alsocalled hydrocarbons), and nitrogenoxide (NOx), is discussed in detail.The three-way catalysts, combinedwith the control system that regu-lates air-fuel ratio at stoichiometry(defined as the ratio of air and fuelwith just enough oxygen for completefuel combustion—equal to about 14.6for gasoline), achieve emission reduc-tion for all three gases by a factorranging from 20 to over 100. Dieselengines cannot run with a stoichio-metric (or rich) air-fuel ratio becausesuch operation results in high partic-ulate matter emissions and visiblesmoke. Thus, with conventionalthree-way catalysts, the reduction ofNOx is not achieved due to excessoxygen in the exhaust gas. Instead, aselective catalytic reduction (SCR)system that uses urea to break downNOx can be employed. The book pre-sents a model of an SCR system.

Finally, DEMs of in-cylinder phe-nomena, the residual gas dynamics,and in-cylinder pressure, can be foundin section 3.3. This section shows howto estimate the burned-mass fraction(BMF), as a function of crank-angledegree, from the in-cylinder pressuresignal. The BMF is relevant for com-bustion development since the loca-tion of 50% BMF correlates well withthe maximum brake torque (MBT)spark timing, the timing that achievesthe best fuel consumption.

Engine SubsystemControlThe second part of the book, chapter 4and appendices A and B, is devoted tocontrol design. There are several dif-ferent modes of operation for IC

98 December 2005IEEE Control Systems Magazine

engines during the typical drive. Thesemodes include cranking, cold-startemission reduction (CSER), idle, nor-mal drive, and speed control. Eachmode is defined by a different objec-tive or set of objectives. For example,in the CSER mode, the objective is toachieve catalyst light-off in a shorttime, possibly at the expense of fueleconomy. In the drive mode, theengine produces torque to respond tothe driver’s demand (and concurrentlyoptimizes fuel consumption and emis-sions), whereas, in the idle mode, thegoal is to regulate the engine speed toa desired set point. In each mode,many engine subsystems, whichrequire local or coordinated control toachieve the objective, are active.

In addition, an engine control sys-tem has to provide monitoring, diag-nostics, and failure mode managementfunctionality that are either requiredby regulation or needed to ensurecapability and availability. Present-ing a description of each of thesesubsystems in a book of this formatis not desirable or even feasible.Instead, in Chapter 4 the authorspresent four engine control designcase studies: air-fuel ratio control,multivariable engine speed and air-fuel ratio control, SCR system con-trol, and thermal management. In allfour cases, the controller perfor-mance is evaluated experimentally.

The air-fuel ratio regulation (sec-tion 4.2) is a typical example of anengine control system design. Theobjective is to maintain the air-fuelratio at stoichiometry to ensure highefficiency of the catalytic converter.The controller contains feedforwardcomponents, specifically, an air-charge estimator, and a compensatorfor fuel wall-wetting dynamics.Improved emission performance isachieved if the model of catalyst oxy-gen storage is included in the controlsystem design. The feedback signal isprovided by either a switching or awide-range (continuous) exhaustoxygen sensor. The feedback compo-nent is either a conventional PI con-

troller or a more complex controllerobtained using H∞ design.

The second example is H∞ designfor a multivariable control systemthat uses the throttle, fuel injectors,and spark timing to regulate enginespeed in the idle mode as well as theair-fuel ratio. To use spark timing asan actuator, the base timing must becentered away from the MBT point toprovide two-sided actuation.Although this setup entails a fueleconomy penalty, this practice isstandard in the industry because ofthe importance of high-quality idlespeed control. The same caveatapplies to fuel if it is used for speedcontrol; with the nominal air-fuelratio set point at stoichiometry (forbest emissions), the fuel cannotincrease the torque, and thus enginespeed. This point illustrates the non-linear nature of an internal combus-tion engine and difficulties that arisein applying linear control techniques.

The last two examples of Chapter 4are control of the selective reductionsystem in a diesel engine and a ther-mal management system. In bothcases, the controller has the familiarfeedback-feedforward structure.

Two appendices conclude thebook. Appendix A provides a broadbut not very detailed review of mod-eling and control of dynamical sys-tems. The material touches onvarious issues that an engine controldesigner must take into account.Appendix B presents a control designcase study of an idle speed con-troller, this time as a single-input, sin-gle-output system. The subtopicspresented are the synthesis of themathematical model, system identifi-cation, model linearization, approxi-mation of the delay with Padéapproximation, and design of the con-trol system for the air-path usingobserver-based pole placement.

ConclusionsThis book presents a selection oftopics of high relevance to enginecontrol design. The text investigates

modeling of physical and chemicalprocesses and discusses selection ofthe control architecture and design ofthe feedback gains for several impor-tant control loops. The diversity of thematerial clearly illustrates the multi-disciplinary nature of the subject andthe steep learning curve faced by anewcomer to the field.

An internal combustion engine is adifficult system to control. In manycases, the behavior is nonlinear (see,for example, the throttle flow functionin figure 2.9). Some of the control vari-ables, such as spark timing, are actu-ally optimized parameters, whichmeans that a deviation in either direc-tion from its optimal set pointchanges the system output in thesame direction. In such a situation,linearization around the set point maynot produce a useful model. In addi-tion, in most control loops, there is asignificant delay between the controlinput and the measured output. Thesefacts make the feedforward compo-nent a major factor in the controldesign. In my experience, most of theeffort for a typical engine subsystemcontrol design is devoted to the mod-eling required for the feedforwardcomponent. Indeed, a major portionof this book is devoted to engine sub-system modeling, and I believe thisselection is right on target.

Overall, the examples are repre-sentative of a typical approach tointernal combustion engine controldesign. If I were to pick a weakness inthe presentation, I’d point to the H∞design, which is not described in much detail and is not included inthe control design overview in appen-dix A. I suspect that only a fraction ofpeople interested in this subject mayhave had exposure to such advancedcontrol material.

In summary, this book is an essen-tial text for anyone interested inengine control design. It seems appro-priate for a graduate-level course, inparticular, for students with somecontrol background. According to theauthors, the book is intended for stu-

99December 20051066-033X/05/$20.00©2005IEEE


dents interested in classical and novelengine control systems. I would like toadd that engine control practitionerscan also learn a lot from this book,especially those practitioners whohave expertise in only a subset ofengine control problems.

References[1] M. Jankovic, S.W. Magner, S. Hsieh, and J. Koncsol, “Transient effects and torque con-trol of engines with variable CAM timing,” inProc. American Control Conf., Chicago, IL,June 2000, pp. 50–54.

[2] U. Kiencke and L. Nielsen, AutomotiveControl Systems for Engine, Driveline, andVehicle. Berlin-Heidelberg: Springer-Verlag,2000.

[3] K. Locatelli, C.H. Onder, and H. Geering,“A rapidly tunable wall-wetting model for SIengine control,” in Proc. SAE World Congressand Exhibition, 2004, paper 2004-01-1469.

[4] T. Ueda and A. Ohata, “Trends of futurepowertrain development and evolution ofpowertrain control systems,” in Proc. SAEWorld Congress and Exhibition, 2004, paper2004-21-0063.

Mrdjan Jankovic received his B.S.from Belgrade University (1986) andmaster’s and doctoral degrees fromWashington University in St. Louis(1989 and 1992). He held postdoctoralpositions with Washington Universityand the University of California, SantaBarbara. He joined Ford Research Labo-ratory in 1995, where he is a technicalleader in the Powertrain Controls R&ADepartment. His responsibilities includeproject management, mentoring andsupervision of technical staff, and directtechnical contribution to develop-ment of engine control systems. Hisresearch interests include automotiveengine optimization and control, non-linear control, and time-delay sys-tems. He is a coauthor of ConstructiveNonlinear Control (Springer-Verlag,1997) and holds more than 20 U.S.patents. He has served on the editori-al board of IEEE Transactions on Con-trol Systems Technology since 1997. Hereceived the Ford Research TechnicalAchievement Award (2001) and IEEETCST Outstanding Paper Award(2002). He is a Fellow of the IEEE.

Convex Optimization by StephenBoyd and Lieven Vandenberghe, Cam-bridge University Press, 2004, 716 pp.,ISBN: 0-521-83378-7, US$65.00.Reviewed by Ali Jadbabaie.

Development of ConvexOptimizationOne of the most interesting develop-ments in systems and control theoryin the early 1990s was the observa-tion that various problems related toanalysis and synthesis of robust andnonlinear control systems could beformulated as optimization problemsinvolving a linear function over a setof matrix inequalities. A second,related development was the realiza-tion that such problems could besolved efficiently. Even now, afterabout 15 years, one can hardly openup a control theory journal and notsee a reference to a linear matrixinequality (LMI). As an example of theubiquity of LMIs in control theory,consider the most basic property of alinear time-invariant (LTI) system,that is, its stability. In 1890, A.M. Lya-punov showed, in his doctoral thesis,that the system of differential equa-tions x = Ax is stable (all trajectoriesconverge to zero), if and only if thereexists a positive-definite matrix Psuch that AT P + PA � 0. This condi-tion was perhaps the first LMI [1].

In the 1960s, V.A. Yakubovichwrote extensively about the role ofmatrix inequalities in control theory.Later, Jan Willems, who actuallycoined the term LMI in his seminal1971 paper [5], speculated abouttheir potential role in control theory.Specifically, he mentions that LMIsmight be exploited in computationalalgorithms.

LMIs belong to a special class ofoptimization problems known as con-vex optimization. In a convex opti-mization problem, a linear objectivefunction is minimized (or maximized)subject to a set of linear equality con-straints and convex inequality con-straints. While the inequalities in an

LMI problem are linear functions ofmatrix variables (hence the nameLMI), such problems are inherentlynonlinear. This nonlinearity is due tothe fact that the condition for a matrixto be negative is a nonlinear functionof the entries of the matrix, namely,that the eigenvalues are all negativeor, equivalently, the nested principalminors are all negative. Obviously,both of these conditions are messynonlinear functions of the matrixentries. So how could such a compli-cated nonlinear problem be solved?

The conventional wisdom in the1960s and 1970s suggested that anoptimization problem is easy if it islinear [linear objective with linearconstraints, for example, a linear pro-gram (LP)] and hard if it is nonlinear.This view was reinforced by the factthat large scale LPs were beingsolved easily. Furthermore, it wasknown how to exploit sparsity andstructure to solve very large prob-lems, thanks to Dantzig’s simplexalgorithm (which had favorable prac-tical complexity) as well as the workof Leonid Kantorovich in the SovietUnion (for which he won the 1975Nobel Prize in economics). Dantzig’ssimplex method performed quite wellin practice, although certain rareinstances took a long time to run.

Until recently, the conventional wis-dom in the optimization community

suggested that an optimization prob-lem is “easy” if it is linear (linear objec-tive with linear constraints, that is,linear program, or an LP) and “hard” ifit is nonlinear. This view was rein-forced due to the fact that LPs withhundreds and thousands of variablescan be solved very efficiently, since weknow how to exploit sparsity andstructure to solve large problems. Thiswas thanks to Dantzig’s simplex algo-rithm that had very favorable practicalcomplexity as well as the work ofLeonid Kantorovich in the Soviet Union(for which he won the 1975 Nobel Prizein economics). The simplex methodperformed quite well in practice,except in rare instances. It took sometime to realize that perhaps the easy-hard division is not along linearity ornonlinearity. As Rockefellar stated in1993, “the great watershed in optimiza-tion isn't between linearity and nonlin-earity, but between convexity andnon-convexity” [4].

A key turning point in changingthe conventional wisdom was the1975 discovery by N.Z. Shor in theSoviet Union of the ellipsoid algo-rithm and the 1979 discovery byLeonid Khachian that the ellipsoidalgorithm solves LPs in polynomialtime. In contrast, Dantzig’s simplexalgorithm had a worst-case exponen-tial complexity. This developmentwas such big news that it became apage 1 story in the New York Times.Although the ellipsoid algorithm pro-vides a polynomial time complexityguarantee, its practical performancein terms of running times was worsethan the simplex algorithm.

The 1984 discovery of a polynomial-time interior-point algorithm by Karmarkar changed the balance, sinceKarmarkar’s algorithm is efficient inpractice as well as in theory. His interi-or-point algorithm was named for thefact that it moved through the interior ofthe feasible region to reach the optimalsolution, even though the optimal solu-tion is known to be on the boundary.

Simply put, Karmarkar’s algorithmconverts a constrained optimization

problem into an equivalent uncon-strained one, in which the cost functionis augmented by adding a barrier func-tion whose value goes to infinity whenapproaching the boundary of the feasi-ble set. As a result, the effect of con-straints appears in the objectivefunction, and the problem is effectivelyrendered unconstrained. All thatremains to be done is to set the deriva-tive to zero using, for example, New-ton’s algorithm to find the minimum ofthe modified objective function. Howev-er, things are not as easy as they seem:to solve this problem efficiently with atheoretical polynomial time complexi-ty, one needs to have an a priori boundon the number of Newton steps neededto get arbitrarily close to the solution.The difficulty is that the classical com-plexity analysis of Newton’s methoddepends on constants that are func-tions of the third derivative; these con-stants are often coordinate dependentand thus are difficult to estimate.

Four years later, a major advancewas achieved by Nesterov andNemirovsky, who realized that Kar-markar’s analysis can be modifiedand extended to a much larger classof convex optimization problems,including but not limited to what wenow call LMI problems or semidefi-nite programs (SDPs) [2], [3].

The barrier function method ofconverting constrained optimizationproblems to unconstrained optimiza-tion problems was not new on itsown (in fact the idea goes back to the1960s), but here is where Nesterovand Nemirovsky’s brilliant ideacomes into play: they show that cer-tain logarithmic barrier functions forconvex constraint sets (such as theone used by Karmarkar) possess anice property called self-concordance.Roughly speaking, self-concordancemeans that the the third derivative ofthe barrier function can be boundedby a function of its curvature or sec-ond derivative in a coordinate-inde-pendent fashion. As a result,Nesterov and Nemirovsky were ableto provide a complexity estimate for

the barrier method that was indepen-dent of the chosen coordinates, andthe number of Newton steps could bebounded a priori. In their seminalpapers [2], [3], Nesterov and Nem-rovsky show that it is possible toconstruct a self-concordant barrierfor any convex optimization problem;however, not all such barriers arecomputable. There is, nevertheless, alarge set of convex optimizationproblems for which such barriersexist. These problems can be solvedin polynomial time with any givendesired degree of accuracy.

This background is only half ofthe story; the other half is identify-ing solvable convex optimizationproblems, which are not alwaysobvious. Sometimes the problem isnot convex but, with a change ofvariables, can be transformed into aconvex problem. This point is pre-cisely where the book at handbecomes an invaluable resource.

In Convex Optimization, theauthors introduce several classes ofoptimization problems that areamenable to efficient numerical solu-tion with interior point algorithms.One such class is that of conic prob-lems in which a linear objective func-tion is minimized subject to a set ofinequality constraints that representthe intersection of an affine space(such as a plane) and a cone (such asthe positive orthant).

Examples of conic problemsinclude linear programs (where the cone is the positive orthant), SDPs(where the cone is the set of positive-definite matrices; LMI feasibility prob-lems are of this category), andsecond-order cone programs (SOCPs)(where the cone is the Lorentz or icecream cone). All of these problems,which involve optimization problemsover self-dual cones, can be treatedmore or less in the same fashion. As aresult, it is possible to develop a duali-ty theory that closely mirrors that oflinear programs.

While SDPs and LMIs are familiarin the controls community, SOCP


problems are not as well known. Agood example of an SOCP, which alsodemonstrates the difficulty in recog-nizing convex problems, is a linearprogram with random constraints.This problem appears to be a linearprogram in which each row of theconstraint matrix is a Gaussian ran-dom vector, and each inequality con-straint needs to be satisfied withsome probability. To an untrainedeye, the difficulty of the problemwould be the same, irrespective ofthe value of probabilities. It turns out,however, that when the constraintsare required to be satisfied withprobability above 0.5, the problem isconvex, and an SOCP. If, however, theprobability of a constraint being satis-fied is below 0.5, the problem is nolonger convex and becomes very diffi-cult to solve. Another interestingexample of an SOCP is robust linearprogramming, in which each row ofthe constraint matrix is an uncertainvector in an ellipsoid and the goal isto minimize the worst-case cost.

Finally, geometric programs (GPs)are additional optimization problemsthat can be solved efficiently. In a GP,the constraints and objective areposynomials or multivariate polyno-mials, with positive coefficientswhose domains are positive real num-bers. In constrast to the convex pro-grams discussed earlier, GPs are notconvex in their natural form. Howev-er, a simple change of variable can beused to transform a GP into a convexproblem. The study of simple geomet-ric programs goes back to the 1960sin the chemical engineering literature,while new applications range frominformation theory to transistor sizingin RF circuits. The aforementionedexamples are just a few samples fromthe interesting examples of convexoptimization covered in the book.

The BookConvex Optimization by Boyd and Van-denberghe can be used as a graduate-level textbook for students acrossvarious disciplines of engineering and

applied sciences as well as practition-ers in industry. The book is a result ofover 15 years of research by theauthors and their students in formulat-ing, finding, and solving convex opti-mization problems in diverse areas ofengineering and applied sciences. Theauthors have used various drafts ofthis book in graduate-level courses onoptimization theory since 1995. As aresult, the current manuscript is wellwritten and easy to follow. While famil-iarity with classical optimization ishelpful, it is by no means a require-ment for reading the book, since themanuscript is self-contained with auseful set of appendices. That beingsaid, a good working knowledge of lin-ear algebra, advanced calculus, basicprobability theory, analysis, and basictopology (norms, open sets, and con-vergence) is required.

There are three major sections inthe book dedicated to theory, applica-tions, and algorithms for convex opti-mization problems. A semester-longgraduate level course can easily coverall three sections. The first five to sixweeks or the first half of the coursecan be used to cover the theory, withthe second half divided equallybetween applications and algorithms.While a diverse set of applications,ranging from statistics, geometry,approximation, and estimation prob-lems, are presented as individual chap-ters, additional applications appear asexercises. One notable absence on theapplication side are applications ofconvex optimization in control theory.The third and last part of the book pre-sents algorithms for solving convexoptimization problems along with com-plexity analysis of the algorithms.

Contents of the BookThe book consists of three sectionsincluding 11 chapters, as well asthree appendices and an extensivebibliography. Section I, which coversthe theory portion of the book, con-tains five chapters.

● Chapter 1 provides an intro-duction to the topic, sets up

the notation, and provides abrief history of the topic in itsbibliography.

● Chapters 2 and 3 provide amodern overview of convexanalysis. Convex sets aredefined in chapter 2. Convexfunctions and operations thatpreserve convexity are dis-cussed in chapter 3.

● Chapter 4 is the core of thetext, where LPs, SDPs, SOCPs,and GPs are introduced.

● Chapter 5 presents a unifiedtheory of duality for conic opti-mization problems. Severalinterpretations of duality, rang-ing from economics to mechan-ics, are also presented.

The second section, whichincludes chapters 6–8, covers appli-cations. Application areas such asconvex geometry, statistics, estima-tion, and approximation problemsare presented.

● Chapter 6 includes severalinteresting approximation andfitting problems, ranging fromnorm approximations, leastnorm problems, reconstructionand smoothing, function fittingand interpolation as well asapproximation. This chaptershould be of interest to the sig-nal processing community.

● Chapter 7 covers statisticalestimation, experiment design,maximum likelihood problems,robust detection, and hypothe-sis testing.

● Chapter 8 contains a variety ofgeometric problems, such asEuclidean distance and angleproblems, classification prob-lems, and placement and facili-ty location problems.

The third section of the bookdeals with algorithms for solving con-vex optimization problems.

● Chapter 9 contains algorithmsfor unconstrained optimiza-tion problems. Topics cov-ered include classical analysisof gradient algorithms for


unconstrained problems as wellas classical and modern analysisof Newton’s algorithm for suchproblems. The key idea is thedemonstration that the quality ofthe gradient algorithm dependson the chosen coordinates,whereas Newton’s method iscoordinate independent. Self-concordance turns out to be thecrucial property that allows acoordinate-independent analysisof Newton’s algorithm. Thisproperty is the main reason forthe success of barrier-based inte-rior point algorithms.

● Chapter 10 extends the analysisof chapter 9 to the case of equali-ty constrained convex optimiza-tion problems.

● The final chapter consists of adetailed analysis of interiorpoint algorithms for convexoptimization problems with lin-ear equality constraints andconvex inequality constraintsusing barrier and primal-dualmethods.

ConclusionsConvex Optimization is a great seg-way into the interesting world oftheory, applications, and algorithmsfor convex programs that can besolved efficiently. The text is suit-able for a graduate-level course inany engineering department. I haveused an earlier draft of the book in agraduate level optimization courseat the University of Pennsylvania.The book is also useful for practic-ing engineers since it presents thestate of the art in theory, applica-tions, and algorithms.

The key goal of the book is to helpthe reader identify and solve convexoptimization problems in diverse dis-ciplines of engineering and appliedsciences. In my opinion, the authorshave achieved this goal. Another fea-ture of this text is that the authorshave posted a copy of the book ontheir Web site, which also includesslides for teaching a course based on

the text. While the book does notfocus on LMIs for control applica-tions, the text provides the readerwith an understanding of how LMIalgorithms work.

Finally, I believe the authorshave done a great job in providing arigorous but comprehensible expla-nation of the success and efficiencyof convex optimization problems.Even though convex optimization ismore or less a technology, there isstill a long way to go for the tech-nology to mature. While linear pro-grams and least squares problemscan be solved for thousands of vari-ables, other forms of convex opti-mization such as SDPs and SOCPsare solvable for hundreds of vari-ables but not thousands. Exploitingstructure and sparsity as well asparallelization are subjects of ongo-ing research.

References[1] S. Boyd, L. El Ghaoui, E. Feron, and V. Bal-akrishnan, Linear Matrix Inequalities in Sys-tems and Control Theory (Studies in AppliedMath., vol. 15). Philadelphia, PA: SIAM, 1994.

[2] Yu. Nesterov and A. Nemirovsky, “A gen-eral approach to polynomial-time algorithmdesign for convex programing,” Centr. Econ.and Math. Inst., USSR Academy of Sciences,Moscow, USSR, Tech. rep., 1988.

[3] Yu. Nesterov and A. Nemirovsky, InteriorPoint Polynomial Methods in Convex Program-ing (Studies in Applied Math., vol. 13).Philadelphia, PA: SIAM, 1994.

[4] R.T. Rockafellar, “Lagrange multipliersand optimality,” SIAM Rev., vol. 35, no. 2, pp.183–238, 1993.

[5] J. Willems, “Least squares stationary opti-mal control and the algebraic Riccati equa-tio,” IEEE Trans. Automat. Contr., vol. 18, pp.184–186, 1971.

[6] S. Boyd and L. Vandenberghe, Convex Opti-mization. Cambridge Univ. Press, 2004[Online]. Available: http://www.stanford.edu/~boyd/cvxbook.html

[7] V.A. Yakubovich, “The solution of certainmatrix inequalities in automatic control theo-ry,” Soviet Math. Dokl., vol. 3. pp. 620–623,1962.

Ali Jadbabaie received his Ph.D.from the Control and Dynamical

Systems Department in 2001 fromCaltech. After spending a year as apostdoctoral associate at Yale, hejoined the Department of Electricaland Systems Engineering at theUniversity of Pennsylvania, wherehe is an assistant professor. Hisresearch interests are in coopera-tive control of networked systems,optimization theory, and optimiza-tion-based control.

Neural Engineering: Computation,Representation, and Dynamics inNeurobiological Systems by ChrisEliasmith and Charles H. Anderson,MIT Press, Cambridge, 2003, ISBN: 0-262-05071-4, US$49.95. Reviewed byJagannathan Sarangapani.

ComputationalNeuroscienceComputational neuroscience has onlyrecently been established as a scien-tific discipline in its own right. Sinceits inception, computational neuro-science has been dedicated to themodeling and simulation of biologicalneural systems, while neuroscienceper se focuses on improving ourunderstanding of how the brain andspinal cord work.

Computational neuroscience andengineering scientists are interestedin how to model a single neuron andits information processing capability,characterize neural networks as time-varying control structures, and applytechniques to generate large-scalerealistic simulations of networks ofneurons so that a specific biologicalbehavior of the brain can be emulated.Novel models of biological systemssuch as the locomotor, vestibular,and working memory systems areessential components for understand-ing the brain.

Due to the increasing importance ofcomputational neuroscience in engi-neering, a need has developed for atextbook for senior undergraduate andbeginning graduate students. NeuralEngineering introduces the theoretical

102 December 20051066-033X/05/$20.00©2005IEEE


foundations of neuroscience with afocus on the nature of information pro-cessing in the brain. In particular, thisbook encapsulates recent advances incomputational neuroscience andextends ideas about neural coding,neural computation, physiology, com-munication theory, control theory,representation, dynamics, and statisti-cal learning concepts.

With the advent of novel compu-tational neuroscience concepts, con-trol system engineers involved in thedesign and implementation of intelli-gent control systems for industrialprocesses can now use the neuralengineering concepts discussed inthis book. Control system engineerstake pride in modern feedbackcontrol concepts that have beenresponsible for major successes inaerospace, automotive, defense, andindustrial systems. However, thecomplexity of today’s systems, suchas formation flying of aircraft andmulti-vehicle cooperative control,has placed severe demands on exist-ing control design techniques. Morestringent performance requirementsin both speed and accuracy in theface of system uncertainties andunknown environments have chal-lenged the limits of modern feedbackcontrol. Operating a complex systemin multiple regimes requires that thecontroller be intelligent with adap-tive and learning capabilities in thepresence of unknown disturbances,unmodeled dynamics, and unstruc-tured uncertainties [1].

The application of neural net-works to closed-loop control sys-tems has only recently beenrigorously studied. Of particularimportance to intelligent-controlengineers is the neural network uni-versal approximation property.Approximation-based intelligentcontrollers are based on approxi-mating the unknown nonlineardynamics of the complex system.These controllers include adaptiveand robust control techniques aswell as learning control, neural net-

work, and fuzzy logic techniques.Intelligent controllers based on arti-ficial neural networks can approxi-mate the nonlinear dynamics ofindustrial systems without assuminglinearity in the unknown parameters(LIP), which is often required inadaptive control. Some forms of fric-tion, for instance, are not linear inthe parameters, while many nonlin-ear industrial processes do not sat-isfy this assumption [2]. Moreover,the LIP assumption requires aregression matrix for the nonlinearsystem that usually involves tediouscomputations.

Whereas robust control tech-niques require a bounding functionfor the unknown system dynamics,neural network controllers can learnthe unknown dynamics online. Thepassivity property of neural net-works provides robustness to distur-bances and unmodeled dynamics. Todesign a neural network controller,however, users must select a suitablearchitecture, network size, weight-tuning schemes for learning, and neu-ron-activation functions, whichrequire a good understanding of theneuron information-processing capa-bilities, neuron transformations, andassociated neural network modelsfrom a neurobiological perspective[1], [2].

Neural EngineeringNeural Engineering introduces neuronmodels that are suitable for exploringinformation processing in large brain-like networks. The book also intro-duces several neural networkarchitectures and discusses theirrelevance for information processingin the brain along with models ofhigher-order cognitive functions. Anadditional feature is the availability ofMATLAB programs for exploring themodels described in the book. Anaccompanying webpage includes pro-grams for download.

Neural Engineering addresses thegap between the low-level spikedneuron models and a population ofneurons represented as high-levelcognitive models. The book coversthe principles and methods of mod-eling and understanding diverseneural systems, while presenting acoherent picture of neural functionfrom single cells to complex net-works. A unified framework is presented in terms of models,assumptions, and techniques forunderstanding neural systems. Theauthors review this emerging field in historical and philosophicaloverviews and in stimulating sum-maries of recent results.

Chapter 1 introduces the singleneuron, the neural system, and neuraltransformations. Since the centralgoal of Neural Engineering is to providea general framework for constructingneurobiological simulations, theauthors describe the methodology inthree steps, specifically, systemdescription, design specifications,and implementation.

Biomedical and pattern recognitionengineers will find chapter 2 useful.This chapter is devoted to represent-ing scalar magnitudes of signals andsystems in the nervous system byusing engineering and biological rep-resentations. To represent neurobio-logical systems and the effect of noiseon the transmission of a signal from aneuron to its neighbor, a noise term isadded to the transmitted firing rate to


introduce random variation into theneuron’s activity. The proposedframework and empirical results areshown on examples involving horizon-tal eye position, arm movements, andthe semicircular canal of the vestibu-lar system.

Chapter 3 represents a populationof neurons using a nonlinear map-ping of input signals since visualimages, auditory streams, patternsof movements, and tactile sensationsare a nonlinear function of multiplevariables such as light intensity andspatial location. For instrumentationand biomedical engineers, such rep-resentations are of interest sinceadvanced instrumentation can bedesigned and developed.

The goal of chapter 4 is to repre-sent time-varying signals by spikingneurons using a spiking version of theleaky integrate-and-fire (LIF) model.The LIF neuron is best understood asa passive RC circuit coupled with anactive spike (delta function) genera-tor. This passive RC circuit is repre-sented mathematically as a first-orderdifferential equation. For a neural net-work engineering researcher, the LIFmodel incorporates physiologicalparameters, including membranecapacitance, membrane resistance,and absolute refractory period. TheLIF model, which is a good approxima-tion of most neurons operating over anormal range, incorporates the impor-tant nonlinearity of neural spike.

Next, the most interesting andimportant discussion included in thischapter is the information transmis-sion capacity per frequency channel,which is measured in bits per spikeor bits per second. This issue may beof interest to engineers in selectingan activation function. Next, thecanonical model, which incorporatesa family of models that effectivelycapture the behaviors of an entireclass of neurons, can be studied byusing the θ -neuron. This θ -neuronmodel captures the qualitative fea-tures of spiking neurons and com-pares favorably with the LIF neurons.

Control researchers have developed-neuron based neural network con-trollers to approximate nonlinearfunctions over compact sets withfewer hidden layer neurons than feed-forward neural network architectures.

Chapter 5 extends neural represen-tation by means of encoding anddecoding. Nonlinear encoding isdefined as the process of representinga neurobiological process mathemati-cally as a function mapping from onespace to the other. By contrast, decod-ing is defined as the process of obtain-ing an inverse of the functionmapping. Population and temporalrepresentations of neural populationsare also united in this chapter by view-ing them as time-varying vectors. Theaddition of noise is included, and theeffect of distortion is analyzed. Thischapter is relevant to engineers work-ing in the pattern recognition area.

While the first few chapters focuson representations of neurobiologicalsystems, neurobiological transforma-tions are characterized in chapter 6.The neurobiological communicationchannel between two neuron popula-tions is used to explain the concept ofa neurobiological transformation. Acommunication channel exampleillustrates how one can model a sim-ple biological process by using a one-layer artificial neural networkconsisting of weights, biases, neuron-activation functions, and inputs. Inaddition, the chapter explains thatthe effect of noise should be takeninto account at a very early stage ofmodeling since many biologicalprocesses are sensitive to noise.Once a suitable model is developed,the book presents how to analyticallydetermine neural network weights toaccurately represent a biologicalprocess as a nonlinear mapping ofinputs, weights, biases, and outputs.For an engineer, this neurobiologicalcharacterization explains how tomodel physical systems using artifi-cial neural networks and ensures thatartificial neural networks can approxi-mate a nonlinear mapping with a suit-

able set of weights. In addition, thisframework indicates that findingweights by analytical means is gener-ally far less computationally intensivethan running large training regimes.

By contrast, computational intelli-gent control engineers typically use anoffline training phase to identify suit-able neural network weights, and,once these weights are selected, theweights are not tuned online. It isoften difficult to identify training setsof input and output signals for con-structing an unknown nonlinear map-ping. On the other hand, to overcomethe need for training sets, it is typicallyassumed that a linear representationof the unknown nonlinear mapping isknown beforehand, and the linear rep-resentation is used to identify suitableneural network weights. However, theweights that are determined analyti-cally by linear means can be used asfirst guesses for nonlinear transforma-tions generated by neural networkssince complex behavior exhibited byneural systems is unlikely to be fullyexplained by a linear transformationsalone. Chapter 7 illustrates this pointby using the response of neurons inthe visual pathway. However, the con-cept of a multilayer neural networkand how to select weight-tuningschemes to approximate a given trans-formation are not covered.

Chapter 7 also characterizes neuralrepresentations and transformationsby comparing the standard algebraicnotion of a complete orthogonal basiswith the concept of an overcompletebasis. In an orthogonal basis, the dotproduct of any two vectors is zero. Incontrast, an overcomplete basis doesnot require linearly independent vec-tors, and thus an overcomplete basisis redundant. Descriptions employingovercomplete bases are not as suc-cinct as those employing complete,or orthogonal, bases. However, in theuncertain and noisy world of physicalsystems, such as neurobiological sys-tems, this redundancy is often valu-able for error correction and theefficient use of available resources.


These results are of interest tocontrol researchers since one-layerneural networks with basis functionsor multilayer neural networks withoutbasis functions are normally used inan ad hoc manner for designing con-trollers. The selection of basis func-tions for a single-layer neural networkto achieve a given approximationaccuracy is currently not well under-stood. Moreover, overcomplete basistechniques have not yet been appliedin neural network control. I am hope-ful that neural engineering can shedsome light in the selection of basisfunctions for a one-layer artificialneural network since a one-layerneural network is computationallyefficient and viable for controllingreal-world systems.

Neuroscientists and engineers willbe interested in studying the neurody-namic models presented in chapter 8. Inthis chapter, linear system techniquesare used to represent neurodynamics.The neural control diagram is intro-duced next, where the dynamics of ageneric population of neurons are rep-resented by a block diagram with inputsand outputs. Using Laplace transforms,a relationship is derived between thedynamics and input matrices of thegeneric neuron population. However,nonlinear neurodynamics and their sta-bility properties are not sufficiently dis-cussed. Instead in this chapter, thethree fundamental principles of neuralengineering are quantified, namely:

● Principle 1. Neural representa-tions are defined by a combina-tion of nonlinear encoding andweighted linear decoding.

● Principle 2. Transformations of neural representations are func-tions of many variables and are determined using analternatively weighted lineardecoding.

● Principle 3. Neural representa-tions can be considered asstate variables in system theo-ry, and thus neurobiologicaldynamics can be analyzedusing control theory.

Chapter 9 discusses statisticalinference, which is essential forexplaining the behavior of animals ina noisy and uncertain world. Thischapter outlines the ability of neuro-biological systems to implement thetransformations needed to supportcomplex statistical inference in thepresence of noisy information. Forexample, parts of the brain are repre-sented using a Kalman filter, whichcombines ideas from statistical infer-ence and control theory. In addition,this chapter includes preliminaryresults using Hebbian learning rulesfor transmitting a signal between twonetworks of neurons over a simplecommunication channel. In fact, it isinteresting to note that Hebbianlearning rules and modified Kalmanfilter update equations have beenused to tune artificial neural networkweights in the context of controllerdesign [1]. Finally, potential disserta-tion topics for neuroscience studentsare discussed.

Target AudienceNeural Engineering attempts to gener-ate a coherent understanding of neu-robiological systems from a systemsneuroscience perspective that catersto a wide audience ranging from phys-iologists to physicists. This book isintended primarily for neuroscientistsinterested in learning about methodsfor characterizing the neurobiologicalsystems that are studied experimen-tally. In-depth understanding of thedevelopment of computational mod-els and a simulation environment forlarge-scale neural models is included.The book’s target audience alsoincludes engineers, physicists, andcomputer scientists interested inlearning how quantitative tools relateto the brain. A good discussion onbiological systems using familiar toolssuch as linear algebra, signal process-ing, control theory, and statisticalinference is covered.

The book includes curriculum andprogramming tools for a graduate-level course in computational neuro-

science. Although the book does notinclude problem sets, solutions,course notes, or examples, thesematerials along with a code librarywritten in MATLAB are available in[3]. Novel simulation models of com-monly modeled biological systems,for instance, locomotor, vestibular,and working memory systemsincluded in [3], provide readers withthe means to compare the frameworkdescribed in this book with that ofother methods.

The book covers introductorymathematical analysis of neurobio-logical systems and thus will be use-ful as a reference text for engineers.By contrast, the book can be used asa graduate text for students in neuro-science. In my opinion, since suffi-cient details are not included onneural dynamics, stability issues, andhow the brain identifies and tunes thenetwork weights, control systemengineers may not be keen on usingthe book as a textbook for a neuralnetwork control systems course.Another weak point is the lack ofproblems at the end of each chapter.

Despite the lack of a sufficientmathematical treatment of neuralengineering concepts and the rele-vance of neural engineering to real-world control applications, I foundthis book to be interesting and usefulfor engineering students. Books suchas [4]–[8] require extensive back-ground on physiology and anatomyand thus are useful only to neurosci-entists. By contrast, Neural Engineer-ing provided me with ideas on how topresent neural networks and neuro-control to control students.

In conclusion, I recommend thisbook to advanced engineering stu-dents, to active researchers in neuro-science, and to those who may beinterested in moving into this field.This book is small enough to be man-ageable in a semester-long course butlarge enough to contain an abundanceof material. However, since the bookdoes not provide detailed coverage oflearning paradigms, stability analysis


[2], and control design, I recommendthat the book be considered only as areference text for undergraduate andgraduate engineering students.

References[1] K.S. Narendra and K.S. Parthasarathy,“Identification and control of dynamical sys-tems using neural networks,” IEEE Trans.Neural Networks, vol. 1, pp. 4–27, 1990.

[2] F.L. Lewis, S. Jagannathan, and A.Yesilderek, Neural Network Control of RobotManipulators and Nonlinear Systems. London,UK: Taylor and Francis, 1999.

[3] http://compneuro.uwaterloo.ca/

[4] W.Y. Lytton, From Computer to Brain. NewYork: Springer Verlag, 2002.

[5] M.I. Posner, Cognitive Neuroscience ofAttention. New York: Guilford, 2004.

[6] E. Gordon, Integrative Neuroscience.London, UK: Taylor and Francis, 2000.

[7] S. Zornetzer, An Introduction to Neural andElectronic Networks, 2nd ed. San Mateo, CA:Morgan, Kaufmann, 1995.

[8] G.M. Ederman, Ed., The Brain. NewBrunswick, NJ: Transactions, Oct. 2000.

Jagannathan Sarangapani is a pro-fessor in the Department of Electricaland Computer Engineering at the Uni-versity of Missouri-Rolla. His researchinterests include embedded neuralnetwork and adaptive control, com-puter, communication, and sensornetworks, diagnostics and prognos-tics, and autonomous systems.

Discrete-Time Markov Chains by G.George Yin and Qing Zhang, Springer,New York, 2005, ISBN: 0-387-21948-X,US$79.95. Reviewed by Vivek S.Borkar.

Markov Chains and Two Time ScalesDiscrete-time Markov chains are thebasic building blocks for understand-ing random dynamic phenomena, inpreparation for more complex situa-tions. Not surprisingly, there havebeen many textbook-level treatmentsof discrete-time Markov chains [2],[3]. The book by Yin and Zhang,

despite its title, is not one of these.Rather, the scope of this book is bet-ter captured by its subtitle Two-Time-Scale Methods and Applications. Asthis subtitle suggests, the book is amonograph on singular perturbationtheory involving Markov chains withtwo time scales.

Perturbation theory of dynamicalsystems deals with situations inwhich the dynamics can be viewed asa small perturbation of anotherunderlying dynamics by an additionalcomponent weighted by a small para-meter ε > 0. As ε ↓ 0, the dynamicsreduce to the underlying dynamics.The perturbation is said to be regularif there is no drastic qualitativechange in the dynamics during thepassage to the limit ε ↓ 0. If there issuch a change, then the perturbationis singular. The precise meaning ofthis condition depends on the context.For example, the perturbation of anordinary differential equation is sin-gular if, in the limit, the dimension ofits state space suddenly drops by oneor more. Thus the perturbation ofx(t ) = f(x(t )) to x(t ) = f(x(t )) +εg(x(t )) with a nice g(·) is regular,whereas the perturbation of thedifferential-algebraic system

x(t ) = f(x(t ), y(t )),

g(x(t), y(t )) = 0,

to

x(t ) = f(x(t ), y(t )),

ε y(t ) = g(x(t ), y(t ))

is singular. This distinction is due tothe fact that, as ε → 0, a higherdimensional differential equationchanges into a lower dimensionalalgebraic-differential system.

As another example, consider adiscrete-time Markov chain with theparameterized transition matrixP + εQ, which is irreducible for ε > 0.That is, for all ε > 0, the state spacedoes not split into two or more dis-joint classes for which the probabilityof moving from one to the other iszero. This situation can be consid-ered a regular perturbation of theMarkov chain with transition matrix Pif the latter is also irreducible. If not,the perturbation is singular since thestate space decomposes in the limitinto disjoint parts unreachable fromone another.

The singular case is usually moredifficult and also more interesting toanalyze. Singular perturbations basedon two time scales involve separationof the dynamics into fast and slowmodes. One major task of singular per-turbation analysis is to characterizethe limiting dynamics as ε ↓ 0. The lim-iting dynamics are typically simpler insome crucial aspect than the original(for example, due to lower dimension)and are therefore easier to analyze.The limiting system can then be a validapproximation to the perturbed sys-tem for small ε > 0. A harder problemis to quantify the errors in this approxi-mation. Asymptotic expansions associ-ated with the limiting procedure areused for this purpose.

Yin and Zhang have made manysignificant contributions to this area,and also to the corresponding issuesfor continuous-time Markov chains.Their contributions to the latter

106 December 20051066-033X/05/$20.00©2005IEEE


domain have already appeared as amonograph [5]. This book presentsits discrete-time counterpart. At theother end of the spectrum, one hascontinuous-time and state Markovprocesses with two time scales stud-ied in [4]. Between them, the threemonographs cover much of theground in singular perturbation theoryfor two time-scale Markov processes.Even so, there is a difference betweenthis book and some of the main-stream singular perturbation theoryof discrete-time Markov chains typi-fied by the recent survey [1]. The lat-ter also deals with averaging andasymptotic expansions in powers of εfor various averages related to thesingularly perturbed Markov chains.The present book looks at a differenttype of expansion, an example ofwhich is the probability distributionof the chain after k steps given by

pεk ≈

N∑

n=1

εnϕn(εk) +N∑

m=1

εmψn(k).

This distribution has the region ofvalidity, 0 ≤ k ≤ T/ε for some T > 0.A crucial feature is the two-fold role ofthe perturbation parameter ε, whichdefines the scale of the perturbationbecause, in the perturbation P + εQof the transition matrix P introducedearlier, the perturbation matrix Q isweighted by ε . In addition, ε is alsoused for scaling the time axis by con-sidering time steps εk,k ≥ 0, over thetime interval 0 ≤ k ≤ T/ε, whichincreases to infinity as ε decreases.The first sum on the right is the outerexpansion aimed at giving a goodapproximation at times away fromzero. The second sum is the initiallayer correction that ensures correctinitial conditions. The expansion isthus at a process level and not onlyfor aggregated averages.

This theory is appealing for appli-cations that involve a dynamic lawthat switches randomly between manyalternatives according to a singularly

perturbed Markov chain. The dynamiclaw may be prescribed, for example,by the driving vector field of an ordi-nary differential equation or the driftand diffusion coefficients of a stochas-tic differential equation. This formula-tion has been the main aim of theauthors’ research and of this book. Infact, the book naturally partitions intotwo parts. The first six chapters, alongwith chapter 14, is the theoreticalfoundation for the book. The rest ofthe book deals with applications to fil-tering, stability, and control of sto-chastic systems with singularlyperturbed Markov switching.

ContentsThe book has 14 chapters, of whichthree are background material, fourare devoted to developing the theory,and the remaining seven deal withspecific applications.

Chapter 1 builds up motivation bymeans of examples from areas suchas manufacturing and communica-tions. This chapter also gives anoverview of the book and a brief liter-ature survey to place the book in per-spective. Chapters 2 and 14 deal withthe mathematical background. Theformer deals with basic material suchas Markov processes and martin-gales, which are usually covered in anadvanced probability course. Thismaterial provides the essential frame-work in which the results of the bookcan be stated and proved. Chapter 14,in turn, covers more advanced andspecialized topics from probabilityand control theory such as weak con-vergence and the Hamilton-Jacobi-Bellman equation. These topics areused elsewhere in the book in variousspecific applications.

Of the theoretical chapters, chap-ter 3 develops the asymptotic expan-sions mentioned above and deriveserror bounds. Chapter 4 extends thistheory to analyze the limit behaviorof associated occupation measures.This analysis is done first for the irre-ducible case and then for the generalcase. In particular, the chapter

proves a functional central limit theo-rem for interpolations of suitably nor-malized occupation measures.Chapter 5 derives some exponentialmoment bounds. Chapter 6 recapitu-lates the contributions of these chap-ters and provides some extensions.

This material thus far is prepara-tion for chapters 7–13, each of whichdeals with a specific application.Chapter 7 looks at deterministic andstochastic difference equations,where the dynamics are modulatedby a singularly perturbed Markovchain as described above. This chap-ter establishes stability and the asso-ciated bounds on mean hitting timesfor these systems in terms of a sto-chastic Lyapunov condition on thelimiting system. Chapter 8 considersthe filtering problem for a linear sto-chastic system with the coefficientsof state and observation dynamicsmodulated by a singularly perturbedMarkov chain. It is shown that theexact filter for the limiting system isnear optimal for the original system.The chapter also considers approxi-mations of the Wonham filter for theMarkov chain when the chain itself ispartially observed.

Chapter 9 considers Markov deci-sion processes with singular pertur-bations. The main result shows thatthe value function of the limitingprocess provides a good approxima-tion to the value function of the origi-nal problem. Also, the correspondingoptimal policy for the limiting prob-lem is shown to be near optimal forthe original problem. An examplefrom manufacturing is given, wherethe objective is to optimize thescheduling of maintenance andrepair of machines subject to ran-dom breakdown.

Chapter 10 establishes analogousresults on approximation of value func-tions and near-optimal controls for thecontrol of a linear stochastic systemwith coefficients switching accordingto a singularly perturbed Markovchain, with quadratic cost. Continuingin a similar vein, chapter 11 considers


the mean-variance control problemfor portfolio optimization with -Markov-modulated coefficients. Thischapter uses the limiting problemand standard methods to establishnear optimality of controls. Both dis-crete-time and continuous-time prob-lems are considered. Chapter 12, inturn, derives more results in this spir-it for a control problem arising fromproduction planning. Chapter 13 con-siders a different class of problems,namely, least-mean-squares stochas-tic approximation algorithms associ-ated with an irreducible Markovchain whose transition probabilitiesare modulated by another singularlyperturbed Markov chain. A switchingdiffusion limit is obtained for interpo-lated iterates. The results are extend-ed to hidden Markov models.

SummaryAs should be clear from the forego-ing, this book is not a text, not even agraduate text. Rather, the book is aresearch monograph based largely onthe authors’ own work. The bookoffers little by way of a general back-ground in discrete-time Markovchains. Rather, the book focusesentirely on two-time-scale analysis fordiscrete-time Markov chains and thuscomplements but does not supersede

traditional works such as [1]. Thebook does, however, fill an importantniche in the literature on singularlyperturbed Markov chains.

A strong point of the book is thenumerical examples spread through-out to illustrate theoretical results. Ialso liked the way each chapter isorganized. The main results are stat-ed without proof first, whereas theproofs are given later at the end ofthe chapter. This format will be ofgreat help to researchers in otherareas who want to know and use theresults without going into the detailsof the proofs. Another strong pointof this book is the fact that many ofthe results are established for themost general case that allows fortransient states in the unperturbedMarkov chain.

The kind of problems analyzedhere occur frequently in areas suchas manufacturing and communica-tions. Hence, the book will be usefulto applied probabilists and engineerswho deal with such systems. Otherthan this, the book’s primary audienceis other researchers in singularly per-turbed Markov chains.

References[1] K.E. Avrachenkov, J. Filar, and M. Haviv,“Singular perturbations of Markov chains anddecision processes,” in Handbook of Markov

Decision Processes, E.A. Feinberg and A.Shwartz, Eds. Norwell, MA: Kluwer, 2002, pp.113–150.

[2] P. Brémaud, Markov Chains. New York:Springer, 1999.

[3] J.R. Norris, Markov Chains. Cambridge,UK: Cambridge Univ. Press, 1999.

[4] Y. Kabanov and S. Pergamenshchikov,Two-Scale Stochastic Systems. Berlin: Springer,2003.

[5] G. Yin and Q. Zhang, Continuous-TimeMarkov Chains and Applications: A SingularPerturbations Approach. New York: Springer,1998.

Vivek S. Borkar received his B.Tech.in electrical engineering from I.I.T.,Mumbai, in 1976, his M.S. in systemsand control from Case WesternReserve University in 1977, and hisPh.D. in electrical engineering andcomputer science from the Universityof California, Berkeley, in 1980. He waswith the TIFR Centre, Bangalore(1982–1989) and the Indian Institute ofScience, Bangalore (1989–1999),before joining the Tata Institute ofFundamental Research, Mumbai. He isa Fellow of the IEEE, Indian Academyof Sciences, Indian National ScienceAcademy, and Indian National Acade-my of Engineering. His research inter-ests are in stochastic optimizationand control and their applications.


Today, self-correcting mechanical systems surround us. So it’s hard for us toappreciate the impact of Wiener’s ideas in the mid-20th century, when hepointed out the similarity between machines with sensory systems that col-

lected information to fine-tune their behavior and biological systems—like humanbeings—that did the same thing. Cybernetics—Wiener’s theory of “control andcommunication in the animal and the machine”—made him a cultural figureprominent enough to be featured in Time magazine cover stories.

From “No Mean Mathematician,” by Mark Williams (Technology Review,June 2005), a review of the book Dark Hero of the Information Age:

In Search of Norbert Wiener, the Father of Cybernetics

Documents

BOOKSHELF The Book - IEEE Control Systems Society