Bridging scales with generating functions, and the large-deviations analysis of

stochastic evolutionary game theory

Eric SmithSanta Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501, USA

Supriya KrishnamurthyPhysics Department, Stockholm University, SE 106 91 Stockholm, Sweden

(Dated: May 9, 2011)

We review the problem of quantitative prediction of multiscale dynamical phenomena, for stochas-tic evolutionary games classified according to their symmetry. We first introduce a “maximum-ignorance” approach to identifying the many undetermined parameters of a stochastic process ifonly some coarse representation of the symmetries, such as the first-moment conditions, are knownreliably. We then construct the time-dependent generating function for any such process in gen-eral form as a perturbed dynamical system, following the methods of Freidlin and Wentzell. In acompanion review, we have introduced several game models as examples, showing how symmetrygroups and their representations determine the qualitative nature of multiscale dynamics. Thispaper shows how collective fluctuation effects are organized by symmetry, and how in turn theyproduce quantitative relations among scales or scale-dependent parameters, which go beyond thepredictions possible from symmetry alone. We derive both fluctuation distributions and the emer-gence of asymptotic large-deviations scaling, which defines the thermodynamic limit for evolutionarygames. Using an example that incorporates neutral fluctuations, we show that the thermodynamicdescription differs even at infinite population from the individual-scale interactions, by correctionsthat have the interpretation of entropies.

I. INTRODUCTION

Multiscale dynamics arises in evolutionary games whenbehavior at the population level separates in its timescaleor structure from the individual dynamics of the under-lying agents. The emergence of new scales can mark newforms of individuality, or simply multiple levels of se-lection essential to the dynamics. Because evolutionarygames capture a general relation between developmentand replication/transmission/selection at any level thatcan be described by a population process, they can beused to connect a chain of levels in hierarchical evolvingsystems.

In a companion paper, we show how symmetry groupsand their representations imply precise relations betweenpopulation-scale and individual-scale dynamics, whichhold across a range of variations in relative scale or de-tailed structure. The generality of results implied bysymmetry makes them robust with respect to both in-ference and prediction [1, 2], but for the same reason itdoes not enable us to predict many quantitative relationson the basis of symmetry alone.

In this paper we present methods to compute relationsamong multiple dynamical scales in stochastic evolution-ary games, when the relations are mediated by collec-tive fluctuations. The collective waves of type-change bymany agents in a population process are, like its symme-tries, the robust phenomena responsible for qualitativechanges in system character when the system has manyagents. Collective fluctuations are best studied with gen-erating functions [3, 4], which are the Laplace transformsof the probability distribution in the discrete basis thatcounts agents according to their type. As explained in

Ref. [5], generating functions provide a basis of distribu-tions in which any given fluctuation is typical, and theweight they assign to that fluctuation in the populationprocess is the projection onto its corresponding basis-distribution.The companion paper suggests the range of behav-

iors within evolution and development that can be cap-tured with games and understood in terms of symmetry.It also introduces a collection of models reflecting eachclass (continuous/discrete, simple/complex) of symme-try and symmetry breaking, as worked examples. Herewe draw on those examples and derive quantitative re-sults for separations of scale, scale-dependent parameterchanges, and characteristic fluctuation spectra.

Large-deviations scaling, concentration of measure,

and multiscale dynamics

We will compute fluctuation distributions at all scales,but the limit in which these lead to simple relations ofdynamics across scales – reflecting the simplicity of sym-metry relations – is associated with the emergence ofthe large-deviations property [6] for probability distribu-tions of fluctuations. The large-deviations property holdswhen log-probabilities separate into a scale factor thatcharacterizes all fluctuations in a system, and a factorcalled the rate function, which depends on the structureof fluctuations in a scale-independent way. The large-deviations property requires that it first be possible todescribe the structure of fluctuations in an invariant wayacross a range of scales. Therefore, large-deviations scal-ing is the defining characteristic of a thermodynamic level

2

of description separate [7] from the detailed underlyingstochastic dynamics of individuals.In addition to the qualitative separation of macro-

scopic from microscopic dynamics, large-deviations scal-ing implies a quantitative effect known as concentration

of measure [8, 9] in space and time, arising from the ex-ponential suppression of fluctuations. Concentration ofmeasure, like multiscale dynamics, has qualitative conse-quences that may be understood in terms of symmetry.The companion paper illustrates simple discrete symme-try breaking with a coordination game showing a pitch-fork bifurcation. In this game the ordered populationcan exist in two states, whose separation does not dimin-ish with large population size. The associated domain-switching trajectories cannot be made small, and the ratefunction is bounded below by a positive number. There-fore a part of the measure concentrates in time, makingevents of barrier-hopping exponentially rare.In contrast, a rock-paper-scissors game demonstrates

simple continuous symmetry breaking through a su-percritical Hopf bifurcation. The symmetry (time-translation) hidden by ordered population states impliesa continuous set of ordered limit-cycle histories, and arate function for fluctuations that is continuous down tozero. As a result of this continuity, fluctuations remaindense in time but have support in space that decreasesas O(1/N) with increasing population size N .

The Freidlin-Wentzell generating functional, and

various ways to construct it

The general class of techniques we will use are due toFreidlin and Wentzell [3], although many parallel formsexist and many construction methods may be used toarrive at them, as reviewed in Ref. [5]. These methodshave been applied to stochastic evolutionary games toprove concentration of measure in large-population lim-its [8–15], and they are widely used to compute prob-abilities and trajectories of escape from basins of at-traction [16–20]. Many approaches to large-deviationsscaling exist [21–23] which can be applied to evolution-ary games. We choose a particular construction due toDoi [24, 25] and Peliti [26, 27], which makes the connec-tions of dynamics across scales intuitive and easy to ex-press. Our treatment differs from the common approachin economics, which has tended to emphasize asymptoticconvergence and equilibrium refinement, in that we aremore concerned with long-term dynamics and the con-nections across multiple dynamical timescales.Freidlin-Wentzell theory provides a formally exact rep-

resentation of time-dependent generating functionals andtherefore a way to treat arbitrary fluctuations. It is alsoa point of departure for essentially all standard approx-imation methods in stochastic processes, including theKramers-Moyal expansion [28], the van Kampen expan-sion [29–31], the Langevin equation, and the stationary-point methods known collectively as the method of in-

stantons [32, 33].

Organization of the paper: model selection followed

by approximation regimes

Following the emphasis in the companion paper on theidentification of mechanism from robust observables, weseparate the problem of solving for fluctuation effects intotwo steps.The first step must be model selection from robust

statistics, performed in Sec. II. The standard observ-ables used in model choice are first-moment dynamics,interpreted in terms of fitness and mutation [34–36]).Estimating a model by its mean fitness and mutationrates leaves many parameters of the underlying stochas-tic process undetermined. We present an approach tochoosing minimal stochastic models which are consistentwith mean-field dynamics and introduce the fewest ad-ditional assumptions. From the stochastic process, wethen systematically construct the exact time-dependentgenerating functionals of Freidlin-Wentzell theory.Approximation methods to extract the quantities of in-

terest are then developed in Sec. III. These will be classi-fied according to the type of symmetry each reflects, andits associated macroscopic dynamics. The game modelsfrom the companion paper will then be solved as exam-ples of each approximation method. Supporting calcula-tions applying the general concepts to particular modelsare provided in Appendixes A–G.

II. FROM ROBUST STRUCTURE ESTIMATION

BASED ON LOW-ORDER MOMENTS, TO

MINIMAL UNDERLYING STOCHASTIC

MODELS

A. General properties and notation for population

processes drawn from the companion paper

From the companion paper, we draw the notation n ≡(n1, . . . , nD) for the counts of agents of types 1, . . . , D

in a population of size N ≡∑D

i=1 ni. We consider gen-eral branching-and-replacement processes that leave typenumber unchanged,1 and from the derivation of the Priceequation [37, 38], we know that these may always beparametrized in terms of a vector of fitnesses (fi) anda stochastic matrix [µij ] whose coefficients we generi-cally term “mutation” rates. Both the {fi} and {µij}will generally be functions of the population state. Inboth papers we use Roman indices n for discrete sam-ple values of population states, and math-italic n for thecorresponding continuous-valued variables which arise as

1 In examples, we will reduce these to Moran processes, as a kindof minimality assumption explained below.

3

mean values from distributions over samples n. Prob-ability density functions for the stochastic process aredenoted ρn, and we will use overbars (n etc.) for aver-ages under ρn, either exact or in various stationary-pointapproximations.

B. The association of minimal stochastic-process

models with first-moment recursions in mean-field

approximation

As long as all types in a population have fitnessesthat differ in relative magnitude by more than O(1/N),the population diverges exponentially from equal propor-tions and toward a well-defined asymptotic equilibrium(although which equilibrium may depend on initial con-ditions). In this situation the mean-field approximationfor closure of first-moment conditions is often valid, andtherefore it is adopted as the starting point to specifystructure in most discussions of evolutionary games.2 Al-though the methods we develop here become most usefulwhen convergence is not exponential due to degeneraciesin fitness – whether they are externally imposed or resultfrom broken symmetry – the deterministic replicator pro-vides a useful starting point to choose model structure inmany cases.As in the companion paper, we assume that the popu-

lation process is Markovian so that the distribution ρn iswell-defined on a complete space of types {n}. A generalMarkovian process is specified by its master equation,

d

dtρn =

∑

n′

Tnn′ρn′ . (1)

We assume that we may pass back and forth betweendiscrete- and continuous-time formulations, and will usu-ally write the continuous-time form for simplicity.3 (Inthis paper, n′ represents an alternative index of summa-tion, and does not carry the Price-equation conventionof population number in an offspring generation, whichwas used to introduce population-genetic models in thecompanion paper.) Tnn′ is called the transfer matrix ofthe stochastic process, and is a stochastic matrix (so∑

n Tnn′ ≡ 0, ∀n′) in order to conserve probability.The time-dependent first-moment of the distribution ρ

is denoted n, with components

ni =∑

n

niρn. (2)

2 The exponential divergence governed by the mean-field approx-imation also underlies the definition [39] of ESS through linearstability analysis in the replicator equation about its rest points.

3 The companion paper provides a discussion of the passage be-tween discrete and continuous time, as well as of other restric-tions on the form of the generator in relation to the synchroniza-tion or branching.

These may be averages under a model distribution suchas ρn, or the mean values that index a whole space ofPoisson distributions used as a set of basis functions, asin the methods of Peliti introduced below.If we can assume that first-moment closure by the

mean-field approximation is valid, we require that thetransfer matrix induce mean population dynamics whichmay be written in the form

dni

dt=∑

n,n′

niTnn′ρn′

= ni (fi − φ) +D∑

j=1

µij nj . (3)

The second line has the form of the replicator equation

with mutation, in which the mean-field approximationconsists of writing fi and µij as functions of n but not ofhigher-order correlations. If all relative fitness differencesare > O(1/N), the functions fi and µij used to define themodel will be close to the regression coefficients obtainedfrom samples generated by that model. The populationnumber is exactly preserved,

d

dt

D∑

i=1

ni ≡ 0, (4)

if we choose

φ =1

N

D∑

i=1

nifi. (5)

As noted in the companion paper, the conditions (3–5)assume more than just the replicator equation, becausefixed absolute population number implies fixed fluctua-tion strength, a simplification we will assume in our ex-amples.The deterministic replicator measures the antisymmet-

ric part of population change, known as drift in thestochastic-process literature. It is insensitive to the sym-metric part of population fluctuations, which leads todiffusion.4 It also only constrains mean drift; higher-order antisymmetric moments are left undetermined byEq. (3).However, just as the method of maximum entropy [40]

may be used to assign a maximum-ignorance distributionto any system of linear constraints (however simple), wemay assign a “maximum-ignorance” stochastic process

4 Collision of terminologies seems inescapable when population ge-netics encounters more general stochastic processes. “Drift” inthe terminology of neutral theory in population genetics corre-sponds to the “diffusion” term in the stochastic-process nomen-clature. The quantity known as “drift” in stochastic processes isassociated with “selection” in population genetics, though it canalso result from regularity in the transmission process.

4

to dynamical first-moment conditions. In the same man-ner as further constraints refine a maximum-entropy dis-tribution, higher-order polynomial expansions of fitness,or other constraints on fluctuation moments, may be in-corporated with the addition of terms in the stochastic-process description. The particular parameters that de-fine the stochastic process will appear as cumulants in afluctuation probability in the field theory defined below.We do not prove formal relations between our min-

imal stochastic processes and maximum-entropy condi-tions for the distributions they produce, although a for-malism of path entropies exists within which such rela-tions could be developed [5]. Rather, we define mini-mality through the two conditions that: 1) both driftand diffusion terms in the stochastic process introduceno new parameters beyond those in the mean-field repli-cator equation, and 2) the generator of the Markov chainmay be written as a Moran branching process [41] on asufficiently short, discrete timescale. In the Moran pro-cess, only one birth or death occurs within a time step,so all events of replication and replacement are indepen-dent, and we may write the transfer matrix Tnn′ withonly diagonal and one-index off-diagonal elements.To separate drift from diffusion, we divide a general

transfer matrix into its antisymmetric and symmetricterms as

Tnn′ = TMFnn′ + TNoise

nn′ . (6)

We labeled the drift term MF because we will estimateit in the mean-field approximation. The diffusion term islabeled “Noise” because it will be the source of fluctua-tions.If we now suppose linear frequency-dependent fitness5

fi =

D∑

j=1

aij (nj − δij) , (7)

corresponding to the second-order polynomial regressionmodel in Eq. (16) of the companion paper, with the in-trinsic fitness αi ≡ 0, and isotropic diffusion

µij = 1−Dδij , (8)

then the lowest-order part of the drift operator TMFnn′ is

uniquely specified by the dynamics (3). With the as-sumption that all population changes may be built upfrom single-individual type changes, Tnn′ 6= 0 only on in-dices n, n′ that differ by ±1 at two types j and l. Theindex-shifts in a general master equation of this formmay be written in terms of exponentials of the opera-tors ∂/∂nj , ∂/∂nl. The remaining terms in the transfermatrix may be systematically constructed as shown inApp. A.The result for the drift term, reproduced from

Eq. (A10), is

∑

n′

TMFnn′ ρn′ =

1

2

D∑

j,l=1

sinh

(

∂

∂nl− ∂

∂nj

)

{

nj (nl − δjl) aAjl +

1

N

D∑

k=1

nj (nk − δjk) aSjk (nl − δjl − δkl) + µjlnl

− 1

N

D∑

k=1

nl (nk − δlk) aSlk (nj − δlj − δkj)− µljnj

}

ρn, (9)

in which aAlk ≡ (alk − akl) /2 and aSlk ≡ (alk + akl) /2are respectively the asymmetric and symmetric parts ofthe payoff matrix. The antisymmetric combination ofshift operators sinh (∂/∂nl − ∂/∂nj) in Eq. (9) describesnet conversion from type l to type j, and the remainingquantity in braces gives the rate of conversion per unittime.The operator TMF

nn′ cannot define a full master equa-tion, because the quantity in braces is not positive defi-nite and cannot be interpreted as a probability. We mayassociate a unique symmetric operator TNoise

nn′ with thedeterministic replicator dynamic by requiring that the re-sulting Tnn′ contain only positive-definite rate constants,and no additional parameters beyond those appearing in

Eq. (9). While the resulting transfer matrix is not theonly one compatible with the first-moment conditions,it operationalizes our definition of a maximum-ignorancemodel choice.We construct TNoise

nn′ as follows: The lowest-order sym-metric shift operator which conserves probability, coun-terpart to the antisymmetric sinh (∂/∂nl − ∂/∂nj) inEq. (9), is [cosh (∂/∂nl − ∂/∂nj)− 1]. Similarly, theunique symmetric hopping rate that neither introducesnor suppresses parameters is the symmetric counterpart(under exchange of j and l) to the antisymmetric bracedterms in Eq. (9). If we adopt the product of these twosymmetric operators as the noise component of the trans-fer matrix,

5 Like the assumption that defines structure from the mean-fieldapproximation, the retention of only affine-order frequency de-pendence is an assumption that the lowest nontrivial order ex-

pansion is the most robust. These linear fitness and mutationterms define the Gaussian order of fluctuations, so these robust-ness assumptions have a basis in the central-limit theorem.

5

∑

n′

TNoisenn′ ρn′ =

1

2

D∑

j,l=1

[

cosh

(

∂

∂nl− ∂

∂nj

)

− 1

]

{

nj (nl − δjl) aSjl +

1

N

D∑

k=1

nj (nk − δjk) aSjk (nnl − δjl − δkl) + µjlnl

+1

N

D∑

k=1

nl (nk − δlk) aSlk (nj − δlj − δkj) + µljnj

}

ρn, (10)

then the right-hand side in the master equation (1) becomes

∑

n′

Tnn′ρn′ ≡D∑

j,l=1

(

e∂/∂nl−∂/∂nj − 1)

{

µjlnl + njajl2

(nl − δjl) +1

N

D∑

k=1

njaSjk (nk − δjk) (nl − δjl − δkl)

}

ρn. (11)

As long as all ajl and all off-diagonal components of µjl,are non-negative, the quantity in braces in Eq. (11) isnon-negative and has the interpretation of the probabilityper unit time for single-agent conversions from type lto type j, through any combination of replacement withreplication or mutation.For master equations in which the shift operators are

written this way (as functions of exp (∂/∂nl − ∂/∂nj) −1), it is particularly easy to extract the diffusion ap-proximation known as its Fokker-Planck equation. Thesecond-order Taylor expansion of the exponentials in∂/∂nl − ∂/∂nj directly yields this diffusion approxima-tion. Proper proofs of convergence in probability to thediffusion equation are provided in the probability litera-ture [21–23]. The Taylor’s-series explanation given heremay regarded as a notational short-cut to the formalderivation, if desired.The hopping rates in braces in Eq. (11) will re-

appear below in both the Liouville operator for generat-ing functions, and in the field-theoretic action of Freidlin-Wentzell theory. Master equations such as Eq. (1), withtransfer matrices of the form (11), may be written morecompactly as

d

dtρn =

D∑

j,l=1

(

e∂/∂nl−∂/∂nj − 1)

pjl(n) ρn, (12)

in which pjl(n) are the population-composition-dependent probabilities to change type from l to j. Wewill sketch the systematic derivation of the Liouvilleoperator and field action for the sake of completeness,but in the final field theory that results, the actionfunctional will have a form that may be written downdirectly from Eq. (12), with pjl given by the braced termin Eq. (11). The form is given in Eq. (43) below.

C. The Freidlin-Wentzell framework for

constructing the thermodynamics of non-equilibrium

stochastic processes

The Freidlin-Wentzell framework [3] for computingquite general properties of perturbed dynamical systems,is one presentation of a wide range of related two-field

methods, reviewed in Ref. [5]. As we will derive it here,it is essentially a method to compute the time-evolutionof the generating function associated with an arbitraryMarkovian stochastic process. However, it is equivalentto several approaches in field theory that have been de-veloped for particular application domains. These in-clude equilibrium [42] and non-equilibrium [43, 44] finite-temperature quantum mechanics, discrete stochastic pro-cesses [24–27], and systems with quantum superpositionbut classically definable entropy [45]. The field theoriesfall into the general class derived by Martin, Siggia, andRose (MSR) [46], and their common properties are re-viewed in Ref’s. [47–49].The Freidlin-Wentzell/MSR framework is formally ex-

act, but it also provides an efficient point of depar-ture to many approximation schemes ordinarily usedwith stochastic differential equations, including theKramers-Moyal expansion [28], the van Kampen expan-sion [29–31], the Langevin equation, and the stationary-point methods known collectively as the method of in-

stantons [32, 33]. Freidlin-Wentzell theory naturallyyields probabilities for macroscopically specified histo-

ries, which generalize the fluctuation theorems of equilib-rium thermodynamics [50]. In the large-population lim-its where field-theoretic methods become most useful, theleading large-deviations scaling of the probabilities gen-erated by the stochastic replicator is expressed directlythrough a rescaling of field variables.6

We provide a slightly unconventional introduction toFreidlin-Wentzell theory, by starting with the usual rep-resentation [4] of generating functions as analytic func-tions of a complex variable. We do this to make clear thestage at which the discrete indices of the master equa-tion are replaced by continuous variables (obscured inthe more conventional operator representation), and alsoto show why generating functions count identical parti-cles much as they are counted in quantum mechanics,

6 This simple approach to extracting large-deviations scaling issimilar to the extraction of the Fokker-Planck equation by mereTaylor’s series expansion in the previous section, and in fact thetwo approaches are related under the conversion from the discretemaster equation to the equivalent field theory performed here.

6

without any implied interpretation in terms of quantummechanics. We then convert the analytic representationof generating functions – which is often used only as aformal power series anyway [4] – to a representation byabstract operators and vectors in a linear space that ismuch more efficient for the treatment of time dependence.

D. From master equations to Liouville equations;

from function spaces to abstract Hilbert spaces

1. The ordinary-power-series generating function ofcomplex arguments

The generating function of a stochastic process re-places discrete number indices in the master equa-tion (1) with continuous-valued complex arguments. Theordinary-power-series generating function [4] for distribu-

tion ρ is defined as the polynomial

Ψ(z) ≡∑

n

(

D∏

m=1

znmm

)

ρn. (13)

Here D complex arguments are components of a vectorz ≡ (z1, . . . , zD). The monomials in the zm are basisfunctions for Ψ, and differentiation with respect to the zmyields the various moments of ρ. The master equation (1)induces dynamics for Ψ, which are also linear, and maybe written in terms of a differential Liouville equation ofthe form

d

dtΨ = −L

(

z,∂

∂z

)

Ψ. (14)

The map from ρ to Ψ is invertible, and the transfer ma-trix (11) determines the form for its associated Liouvilleoperator L, as shown in App. B. The result, reproducedfrom Eq. (B4), is

L(

z,∂

∂z

)

≡D∑

j,l=1

(zl − zj)

(

µjl + zjajl2

∂

∂zj+

1

N

D∑

k=1

zjzkaSjk

∂

∂zj

∂

∂zk

)

∂

∂zl. (15)

The δ-terms in the transfer matrix (11), which removeself-play or self-replacement, lead to a form for L withall z-factors to the left, and all ∂/∂z factors to the right.This ordering, known as normal ordering, allows deriva-tives in the Liouville operator to act on Ψ alone, withoutthe complexity of evaluating their effect on internal fac-tors of z within L itself. Any Liouville operator can beconverted to normal order; Eq. (15) is again a minimal

form because the coefficients in the normal-ordered ex-pression are the same as those in the deterministic repli-cator equation.

The Liouville operator is conventionally defined withthe minus sign as in Eq. (15). On the space of gen-erating functions, it is a positive-semidefinite operator,and thus behaves like a Hamiltonian in the field theo-ries corresponding to equilibrium thermodynamics [33].In the Freidlin-Wentzell theory, L is in fact a Hamil-tonian, and induces a volume-preserving time-evolutionmap on the space of probability distributions, familiarfrom Eikonal approaches to solutions of the diffusionequation on curved manifolds [16, 17, 51]. The role ofL as a Hamiltonian will connect the general domain ofirreversible stochastic processes, to the concept of a po-tential that governs escape from domains of attraction,and to the more general representation of symmetries.

2. Abstracting from analytic functions to a representationin linear algebra

Masao Doi observed [24, 25] that monomials znmm form-

ing a basis for Ψ obey the same laws of superposition asquantum-mechanical states, and that the derivatives inthe Liouville operator act on these states just as rais-ing and lowering operators act in quantum mechanics orquantum field theory.7 In particular, generating func-tions of a probability distribution ρn count the historiesof numbers of each type in a population, but do not trackhistories of individuals. Therefore the many construc-tions in linear algebra invented to facilitate quantum-mechanical calculations may be applied to stochastic pro-cesses. As in Ref. [5] we take care here to distinguish theoperator algebra, which merely represents counting con-ventions, from the choice of Hilbert space on which theoperators act. The latter distinguishes stochastic fromquantum applications.The algebraic relation that defines z and ∂/∂z as oper-

ators on a space of complex functions is the commutator

[

∂

∂zi, zj

]

= δij . (16)

7 This observation is again commonplace; the algebraic representa-tion of harmonic-oscillator states was originally constructed fromdifferential solutions to the Schrodinger equation.

7

It may be abstracted as a relation among formal opera-tors a and a†,

[

ai, a†j

]

= δij , (17)

which we refer to conventionally as raising and loweringoperators, respectively, under the map

zj ↔ a†j , (18)

∂

∂zi↔ ai. (19)

For completeness, we note that the space of complexmultinomials is “constructed” by application of the rais-ing operators, simply by multiplying factors of zm on thenumber 1. Therefore, we identify the right ground state

of the function space with notation

1 ↔ |0) . (20)

An essential feature of the complex functions as a linearvector space is that all lowering operators ai annihilatethe ground state |0). The requirement on an inner prod-uct is that the conjugate ground state (0| be normalizedagainst |0) (written (0 |0) = 1), and that the annihila-tion property ai |0) = 0 be preserved under conjugation,

so that (0| a†j = 0. The unique operator on complex func-tions that satisfies these requirements is

∫

dDz δ(z) ↔ (0| . (21)

App. C elaborates this notation to identify monomialsin z as the number states in the vector space, and thegenerating function Ψ as a corresponding vector

Ψ(z) ↔ |Ψ) . (22)

The inner product on |Ψ) in this space, preserved nomatter what its time evolution, is simply the trace ofthe original probability distribution ρn. In the analyticrepresentation, this corresponds to evaluation of the gen-erating function at argument 1; that is, Ψ(1) = 1 for alldistributions ρn. The inner product that produces thisevaluation is the Glauber norm, defined as

1 =

∫

dDz δ(z)Ψ(z + 1)

↔ (0| e∑

mam |Ψ) =

∑

n

ρn. (23)

The left-hand state in Eq. (23), with its product of shiftoperators exp (am), is called theGlauber state, and is nor-malized against each of the basis monomials separately.

The Liouville equation (14) in this notation becomes

d

dt|Ψ) = −L

(

a†, a)

|Ψ) . (24)

The Liouville operator remains that of Eq. (15), with theappropriate change in notation,

L(

a†, a)

=D∑

j,l=1

(

a†l − a†j

)

(

µjl + a†jajl2aj +

1

N

D∑

k=1

a†ja†ka

Sjkajak

)

al. (25)

It can sometimes fail to be apparent, in the operator rep-

resentation, that a†j is already, in some sense, a “contin-uous” argument, since it may be represented by the vari-able zj . Thus we have already replaced the discrete basisof the master equation with a continuous basis. The fur-ther conversion to an integral over field variables, whichwe will perform next, is only a small additional change.

E. Reduction to quadrature and the coherent-state

generating functional of time-dependent correlations

Formally, the Liouville equation can be integrated intime as an operator equation, or “reduced to quadra-ture”. Doing so provides a means to compute correlationsbetween moments of the probability distribution ρn atdifferent times. The bracket notation for inner productsin the operator representation replaces the more cumber-some complex integral that would be required by analytic

functions. The quadrature of Eq. (24) between times 0and T is written formally as

|ΨT ) = e−L(a†,a)T |Ψ0) , (26)

because L is not an explicit function of time.It is possible to work directly with the eigenvalues of

operators such as e−LT [47, 48]. We will find it more con-venient, however, to adopt the method of Peliti [26, 27],which is an expansion in an overcomplete set of coherentstates. These are defined as eigenstates of the loweringoperator (acting to the right),

ai |φ) = φi |φ) , (27)

or the conjugate raising operator (acting to the left),

(

φ†∣

∣ a†j =(

φ†∣

∣φ†j . (28)

Here φ ≡(

φ1, . . . , φD)

is a vector of complex numbers

(which we will treat as a column vector), and φ† its Her-

8

mitian conjugate. Like the original variables zi, the φi

are continuous-valued arguments of generating functions.App. C 2 provides constructions for the coherent states

in terms of the elementary number states, and shows thatthe right coherent state |φ) is simply the generating func-tion for a Poisson distribution over discrete states n, withmean value φ. The left coherent state

(

φ†∣

∣ is the distinc-tive feature of Freidlin-Wentzell theory. It is a weightedshift operator that extracts moments from the conjugatestate in an inner product. An important special case is(1|, which is the Glauber state of Eq. (23). The asymme-try between left/right number states and left/right coher-ent states distinguishes the space of generating functionsfrom the (symmetric) space of quantum states, whichmay share the same algebra of operators. The role of theleft coherent states in propagating information backward

in time, to identify most-probable paths associated witha final argument in a generating functional, is reviewedin Ref. [5].The coherent states provide an over-complete basis for

the space of generating functions. They also furnisha representation of the identity operator, as shown inApp. C 2,

∫

dDφ†dDφ

πD|φ)(

φ†∣

∣ = I. (29)

Therefore, the time-evolution operator in Eq. (26) maybe factored into a product of short-time evolution oper-ators,

e−L(a†,a)T =

T/dt∏

k=0

e−L(a†,a)dt, (30)

and a copy of the representation (29) of unity inserted be-tween each factor, to replace the operator equation (26)with a series of integrals over complex numbers φ and φ†

at a sequence of times.The initial conditions for any instance of evolution un-

der the master equation are represented in the state |Ψ0).

It is convenient (and has become conventional) to choose|Ψ0) itself to be the generating function for a Poissondistribution with some mean value n,

|Ψ0) = exp

∑

j

(

a†j − 1)

nj

|0) . (31)

For both systems with exponential convergence [8], andthe weaker diffusive behavior that we will consider, infor-mation in the initial state rapidly decays to that of theasymptotic late-time distribution, so nothing computedbelow will depend sensitively on the choice of |Ψ0).

With these procedures for evaluation, App. C 2 showsthat the Glauber norm of the state |ΨT ) may be writtenas a functional integral

(0| e∑

mam |ΨT ) =

∫ T

0

Dφ†Dφ e−Se(φ†

0−1)·(n−φ0), (32)

in which the measure is defined as a continuum limit ofthe product measure over discrete-time values of φ andφ†,

∫ T

0

Dφ†Dφ ≡ limδt→0

T/δt∏

K=1

∫

dDφ†K dDφK

πD, (33)

and the action which is the primary argument of theexponential is given by

S ≡∫ T

0

dt

{

D∑

i=1

(

φ†i − 1

)

∂tφi + L(

φ†, φ)

}

. (34)

(The shift of φ† to φ† − 1 is a convenience to absorb theshift-action of the Glauber state into a total derivative.)

We see that the weights φ†i in the moment-sampling op-

erator have become the conjugate momenta to the fieldsφi which are the means of Poisson distributions. The Li-ouville operator has become the Hamiltonian in S, andtakes the form

L(

φ†, φ)

≡D∑

j,l=1

(

φ†l − φ†

j

)

(

µjl + φ†jφj

ajl2

+1

N

D∑

k=1

φ†jφja

Sjkφ

†kφk

)

φl. (35)

The normal-ordered form of the operator version (25) ofL ensures that its functional counterpart (in which or-dering no longer matters) simply replaces coherent-statevariables for raising and lowering operators.

The advantage of the Peliti coherent-state expansion isthat the functional integral (32) may readily be approx-imated by the stationary paths of the action (34), cor-rected by Gaussian fluctuations of the integration vari-ables. This approximation recovers the mean-field repli-

cator equations in a rather trivial way, but with almostequal ease it gives the leading large-deviations formula forescape trajectories and escape probabilities from basinsof attraction of the stochastic process [18]. The ability toreduce quite complicated statistical escape probabilitiesin high-dimensional state spaces, to elementary problemsof Hamiltonian dynamics, is the strong motivation to usethe Freidlin-Wentzell approach to stochastic replicatordynamics.

9

If we use overbar to denote the stationary-point paths,and expand the general variables of integration φ ≡φ+ φ′, φ† ≡ φ† + φ, then S may be expanded in a func-tional Taylor’s series in φ′ and φ. Using superscript (k)to denote the collection of all terms of order k in thisexpansion, the exponential e−S may be systematicallydecomposed as

e−S = e−S(0)

e−(S(1)+S(2))

(

1− S(3) +S(3)2

2+ . . .

)

.

(36)

The first term, e−S(0)

, does not depend on the integrationvariables at all, and provides a deterministic approxima-tion to the expected field values φ and φ†. About a max-imum, S(1) ≡ 0, and the term S(2) in Eq. (36) defines a

kernel for Gaussian fluctuations of φ′ and φ. Wheneverthe further terms S(3) + . . . are multiplied by suitablepowers of a small parameter such as 1/N to ensure con-vergence of the series, the Gaussian approximation domi-nates the exponential, and the remaining corrections maybe evaluated as moments of this distribution, as indicatedby the large parenthesis in Eq. (36). The next two sec-tions show how stationary points are identified, and howGaussian fluctuations are evaluated and used.

F. A canonical transformation to number fields

There are now two standard approaches to using thefield theory of the Doi-Peliti construction: either to ex-pand in the coherent-state field coordinates, or to expandin fields corresponding directly to the number indices. Inthis section we note that the two approaches are relatedby a canonical transformation, which has the form of achange of variables to action-angle coordinates [52].The early literature from reaction-diffusion theory [47,

48, 53] emphasizes the coherent-state field variables φ andφ†. The two virtues of these variables are are that theyprovide a direct translation from the operator generatingfunction, and that the Liouville operator (35) for low-order frequency dependence is itself a finite-order poly-nomial in fields. The disadvantage is that the directlyobservable quantities – the expected numbers of agenttypes – are bilinear forms in the fields φ†φ, and not ele-mentary field variables. Thus, the most intuitive compo-nent in the Peliti construction – the mean of its Poissondistributions – cannot be directly interpreted as a parti-cle number, but requires multiplication with the weightsin the sampling operator to be assigned a meaning.An alternative approach, finding increasing use [54–

58], is to work directly with the characteristic functionof the distribution in number operators. The resultingaction contains transcendental functions and is formallyinfinite-order in the conjugate momenta to the numberfield. This is not usually a serious problem in practice,because the same low-order fluctuation calculations maybe performed in either representation, and the trans-

formed variables provide many advantages for identify-ing and understanding the stationary paths of the ac-tion. They also lead directly to fluctuation expressionsfor the number field, which is observable, rather than forthe coherent-state field φ which is an eigenvalue of thelowering operator.The action-angle transformation from fields to number

variables is given by

φ†i ≡ eηi ,

φi ≡ e−ηini, (37)

where ni is a number field corresponding directly to theexpectation of the number index ni of Sec.II B, and ηi isan “imaginary angle” variable. Under analytic continua-tion of η, the transformation (37) is equivalent to rotationfrom complex coordinates

(

φ†, φ)

to their equivalent po-lar coordinates (n, iη). The transformation is canonicalbecause it preserves the kinetic term in the action,

dφ†i

dtφi =

dηidt

ni. (38)

App. G checks that no problems arise in the functionalmeasure (33) under transformation to action-angle vari-ables, by verifying that the resulting functional integralrecovers the best Gaussian approximation to an exactlysolvable distribution: the case of neutral diffusion of apopulation between two types.8

In either coherent-state or action-angle variables, weimmediately extract the leading large-deviations approx-imation by descaling the number field by the total pop-ulation size N . In the polynomial expansion for fitness,1/N then serves as a small-parameter in which to ex-pand correlation functions. In field variables, descalingis applied to φ while φ† is left unchanged. In action-angle variables, descaling is applied to the number fields,leaving the angle variables η unchanged. We introducede-scaled number fields with the notation

ni

N≡ νi,

N dt ≡ dτ,µjl

N≡ µjl. (39)

In the deterministic replicator dynamic, the relativefitness fi − φ may be expressed as a linear function offrequencies with a “population-corrected” payoff matrix.In descaled variables, this adjusted payoff becomes inde-pendent of N , and is written

λjl ≡ajl2

+

D∑

k=1

aSjkνk. (40)

8 In this respect, the action-angle variables are actually muchbetter-behaved than their coherent-state counterparts. AsKamenev shows [49], certain terms which are essential to deriv-ing the correct fluctuation kernel seem to disappear upon takingthe continuous-time limit for the free diffusion theory. In action-angle variables, these terms remain explicit in the continuumlimit.

10

The total mean rate of conversion from type l to typej, introduced above in Eq. (12), corresponding to thebraced terms in Eq. (11), is then written in descaled vari-ables as

pjl ≡ (µjl + νjλjl) νl. (41)

pjl is non-negative for j 6= l, and contains both driftand diffusion contributions to type conversion. It doesnot generally vanish at equilibria, but rather satisfies acondition of detailed balance, pjl = plj , between any twotypes l and j. pjl = pjl/N

2 from Eq. (12).The combined transformation from coherent-state to

action-angle variables, and rescaling of the number fieldsand time, recasts the action (34) as

S =

∫

dt

(

D∑

i=1

φ†i

dφi

dt+ L

)

= N

∫

dτ

(

D∑

i=1

ηidνidτ

+ L)

≡ NS, (42)

in which the rescaled Liouville operator L takes the form

L ≡ LN2

=

D∑

j,l=1

(

1− eηj−ηl)

(µjl + νjλjl) νl

=D∑

j,l=1

(

1− eηj−ηl)

pjl. (43)

We now note the relation between the field Liouvilleoperator (43) and the original transfer matrix (11). Al-though a rather laborious sequence of transformationshas been required to take one into the other, their formsare identical, and the map from operators to field vari-ables clarifies the meanings of the latter. The field ηjcorresponds to the derivative −∂/∂nj . The term pjl isthe probability per unit time for all type conversions asa function of n from all processes, with the provisionthat simple products of field n correspond to products ofdiscrete-index n in which agents are never redundantlycounted. With these correspondences, one can writedown the field functional integral for a given transfer ma-trix by inspection.

III. THREE SYSTEMATIC APPROXIMATION

METHODS TO EXTRACT SYSTEM BEHAVIORS

FROM THE FREIDLIN-WENTZELL

GENERATING FUNCTIONAL

We will consider approximation methods in successiveorder of the terms in Eq. (36). This is also the orderin which game models are presented in the companionpaper. The stationary-point action itself has interestingstructure, related to domain flips, for the coordination

game with discrete symmetry-breaking. The stationary-path action lacks interesting structure, but the second-order kernel for fluctuations becomes interesting, in thecase of hidden time-translation in Rock-Paper-Scissors.Finally, the fluctuations feed back through third-orderderivatives to alter the stationary paths themselves, forthe canonical Prisoners’ dilemma.

A. Stationary-point expansions for mean-field flow

equations and large-deviations formulae

Stationary paths of the action S(0) are of two types,which we will term classical and non-classical. Classicalstationary paths satisfy S(0) ≡ 0. They form a continu-ous space, and comprise all solutions of the deterministicevolutionary game equation (3). The property S(0) ≡ 0implies that the probability of any such path is ∼ 1 toleading exponential order. Which path is followed thusdepends on boundary conditions. Classical stationarypaths converge on the stationary sets of the replicatordynamic (rest points, limit cycles, or chaotic attractors).

The non-classical stationary paths are solutions withS(0) > 0 of locally least action. They represent the most-probable paths for escape from the stationary sets of theclassical replicator dynamic, and correspond to the eikon-als in the ray approximation to solutions of the Fokker-Planck equation [51]. A continuum of the non-classicalstationary paths requires that any such path be anchoredby a boundary condition at the final time in the path in-tegral (32). In the absence of neutral directions, anysuch path will decay exponentially with time before thefinal boundary, and thus does not describe persistent,autonomous dynamics of the population.An isolated, discrete set of non-classical stationary

paths connects the rest points of the classical replica-tor dynamic. In the Hamiltonian description, these formthe heteroclinic network [59, 60] of the dynamical sys-tem. The orbits in this network connect stable rest points(the ESS) to saddle points, and form the instantons ofthe stochastic process. Because they terminate in in-terior points, any sufficiently-long trajectory may havean unlimited number of such transitions. They are sup-pressed at leading exponential order with probabilities

∼ e−S(0)

, which provides the large-deviations probabilityfor escape. Escapes have finite duration comparable tothe time for diffusion toward stable rest points, and there-fore may occur as a (Poisson-distributed) “dilute gas” inany long trajectories. The sum of probabilities from thisdistribution of possible escapes determines the character-istic lifetime of the stable rest points under perturbationfrom non-vanishing fluctuations [33]. We will considerthe trajectories and probabilities of a single escape inthis section, and return in Sec. III B 1 to the proper for-mulation of the Poisson distribution over escapes. Theheteroclinic network of the dynamical system – if one ex-ists – is important because it describes the autonomousdynamics of the aggregate population in the regime of

11

symmetry breaking.

The instantons of the stochastic process resemble thoseof equilibrium, but they differ in one important respect,which affects the interpretation of potentials in these the-ories. In equilibrium escape problems, trajectories whichare mirror images under time-reversal have the sameprobabilistic suppression, and the probability of a tra-jectory is therefore linked to its form. For irreversiblestochastic processes this association need not hold. Es-cape trajectories will often approximate (in one dimen-sion, they will equal) the time-reversed mirror imagesof classical diffusive paths. However, extra force termsin the Freidlin-Wentzell Hamiltonian serve to preciselycancel the probabilistic suppression of classical diffusionsolutions, while approximately doubling the path contri-bution (squaring the probability) for escapes.

1. Scaling properties of the stationary-point approximation:the large-deviations limit and thermodynamics

We noted in Eq. (42) that S = NS, where S is ap-proximately independent of N as N → ∞. Thus the

large-deviations scaling e−NS(0)

of escape probabilities isexponential in population size, corresponding to the ex-

tensive scaling familiar from equilibrium thermodynam-ics. Escape represents a shift by a fixed fraction of thepopulation to a saddle or unstable fixed point. Therefore,

writing e−NS(0)

=(

e−S(0))N

, we see that we may inter-

pret e−S(0)

as the conditional probability for each indi-vidual to follow the path to a saddle or boundary point,given the population state at each point, independentof the motion of other individuals. Although this scal-ing thus represents independent motion by individuals,because it is an essential singularity in the Taylor’s se-ries expansion in 1/N , it cannot be approximated by anyfinite-order expansion of the action as in Eq. (36) [32]. Inthis respect, escapes are less probable than all orders offluctuations about a rest point, and are inherently missedby perturbations about the deterministic description.

2. “Kinematic” description of the stationary-point flowequations, and the shape of the potential

The stationary point condition, if we do not attemptto incorporate fluctuation corrections, is S(1) = 0, equiv-alent to vanishing of the first variational derivatives of S.In coherent-state fields, vanishing first derivative givesthe Hamiltonian “equations of motion”

dφi

dt= − ∂L

∂φ†i

(44)

dφ†i

dt=

∂L∂φi

, (45)

while in the action-angle variables the same conditionreads

dνidτ

= − ∂L∂ηi

(46)

dηidτ

=∂L∂νi

. (47)

Overbars on partial derivatives indicate that they areevaluated at arguments φ†, φ, or their equivalents ν,η. The specific forms for derivatives of L (or L), usedto compute properties of the examples, are provided inApp. D.From either Eq. (D2) or Eq. (D9) in the appendix,

we find that φ† ≡ 1, or equivalently η ≡ 0, are al-ways solutions to the stationary point conditions. Fromthe forms (35) or (43) for the Liouville operator, it isalso clear that these leave S(0) ≡ 0. It is a generalfeature [49] of Freidlin-Wentzell/MSR action functionalsthat φ† ≡ 1 ⇔ η ≡ 0 are the stationary values for allclassical stationary paths.The structure of the Liouville operator reflected by the

classical stationary trajectories is not directly apparentin coherent-state coordinates, but if we introduce the log-ratio of flow rates between types j and l,

rjl ≡1

2log

(

pljpjl

)

, (48)

it is possible to recast the action-angle operator L ofEq. (43) in the form

L = 2D∑

j=1

j∑

l=1

√

pjlplj [cosh (rjl)− cosh (ηj − ηl − rjl)]

≈ −1

2(η − η0)

Tm−1 (η − η0) + V (ν) . (49)

The second line of Eq. (49) is an approximate expansionto second order in η, but keeping all orders in ν. It isapplicable to any Hamiltonian with the form (43)), com-patible with having been generated on sufficiently shorttimescales by a Moran process.In Eq. (49) we have introduced, at each configuration-

point ν, a function η0 (ν), defined as the maximizer of

L over η, and a kinematic potential V (ν) which is themaximizing value,

V (ν) ≡ L (ν, η0 (ν)) ≡ maxη

(

L (ν, η))

≈ 1

2ηT0 m

−1η0. (50)

From the ν equation of motion (46), it follows thatdν/dτ = 0 at (ν, η0 (ν)), so we see that η− η0 behaves asthe kinematic momentum for motion in potential V (ν).The second line in Eq. (50) follows as a condition for con-sistency of the second-order approximation in Eq. (49),

with the condition L ≡ 0 at η ≡ 0.

12

The potential for the coordination game is shown inFig. 10 of Sec. IV C 4 in the companion paper. In keepingwith the structure of instantons for diffusion processes, ithas three minima. The saddle path between these min-imal closely approximates the slowest trajectory for dif-fusive relaxation along the classical trajectories, and alsothe most-probable path for escapes from the ESS to thesaddle-fixed point.The final term in Eq. (49), which behaves as a matrix

whose eigenvalues are the inverses of the masses in thissystem, will be the source of Langevin noise in action-angle coordinates,

m−1ij (ν) ≡ − ∂2L

∂ηi∂ηj

∣

∣

∣

∣

∣

ν,η0(ν)

. (51)

From the second line in Eq. (42) for the action, and the

approximation (49) for its Hamiltonian L, it is clear thatthis decomposition is the standard Hamilton-Lagrangedecomposition familiar from mechanics, except for a termη0∂τν, which arises because the canonical momentum ηis offset from the kinematic momentum η−η0. In systemswhere escapes occur between fixed points, along hetero-clinic networks, this term often exercises little influenceon trajectories, but it is responsible for the asymmetry inthe value of S(0) between the classical and non-classicalsolutions.9

App. E develops the kinematic description of station-ary solutions, their relation to the contours of integrationfor η in the path integral, and the evaluation of the in-stanton probability S(0) on escape trajectories.

3. Conservation laws and non-classical stationary points

The expression of the large-deviation properties ofcomplex stochastic processes in terms of a determinis-tic Hamiltonian dynamical system is already a remark-able simplification. However, the heteroclinic network ofa general dynamical system may still be complicated toextract. The heteroclinic orbits are saddle points, so thatnumerical integration along them (with real η and realν) is generally unstable. Therefore it is useful to furtherrestrict the space of possible solutions using the conser-vation laws of the Hamiltonian system. Doing so leadsto a topological characterization of the heteroclinic net-work, and provides efficient steepest-descent algorithmsto identify the orbits.For Markovian systems of the kind we have con-

structed, two continuous symmetries always exist, whichimply the existence of two conserved quantities byNoether’s theorem [52]. First, in the Liouville opera-tor (49) only differences ηj−ηl arise, implying that global

9 This result does not necessarily extend to system which escapealong separatrices, for which see Ref’s. [19, 61].

shifts of all ηi by a constant is a symmetry of the Hamil-tonian. These shifts affect the diagonal component of η,

(1/D)∑D

i=1 ηi, which appears in Eq. (42) for S only in

the term∑D

i=1 ηidνi/dτ . The conjugate Noether charge,conserved along all stationary paths, is precisely the totalnumber of agents10

d

dτ

(

D∑

i=1

νi

)

= 0. (52)

The other global symmetry is any shift of τ by a con-stant, reflecting the stationarity of the transition prob-abilities in the Markov process. The conservation lawimplied by this symmetry is of the Liouville operator it-self,

dLdτ

= 0, (53)

which follows also from the equations of motion as11

dLdτ

=

D∑

i=1

(

∂L∂νi

dνidτ

+∂L∂ηi

dηidτ

)

. (54)

We may use these conservation laws to restrict themanifold of possible stationary paths as follows: Wehave observed that all classical stationary paths satisfyη ≡ 0 ⇒ L = 0, and that these paths terminate in thestable set of the replicator dynamic. Although the stableset is strictly singular in the ν simplex, actual trajectoriesinvolving escapes merely pass arbitrarily near the stableset, for arbitrarily long time intervals [33]. Thus they areformally continuous with the classical stationary paths,and satisfy the condition

L = 0 (55)

at all times.Referring to Eq. (49) – with the approximate form pro-

viding intuition – the set of solutions L = 0 at any ν

10 Because this symmetry was built in from the beginning, we havefactored out the scale parameter N to define the field ν. Eq. (52)verifies that this rescaling step is consistent with the dynam-ics. In a pedantic treatment we could have worked in the orig-inal fields n, in which case the conserved quantity would havebeen

∑

ini. It is important to understand that the conserva-

tion law (52) reflects a non-trivial aspect of the dynamics, and isnot the same as tautological conservation of the sum of normal-ized population fractions built into the deterministic replicatorequation.

11 We present both conservation laws as properties of the deter-ministic paths of the Hamiltonian dynamical system. In general,when such paths arise as stationary points within a functionalintegral, they reflect a larger symmetry associated with shifts inthe dummy-variables of integration. (These dummy variables are

the fields(

φ†φ)

or (η, ν).) The expression of the symmetry by

shifts of variables of integration, in the correlation functions ofthe theory, are known as its Ward identities [62].

13

is topologically a (D − 2)-sphere in η, centered approx-imately at η0. At rest points, η0 → 0 and the (D − 2)-spheres pinch off. Therefore all stationary paths, whetherclassical or non-classical, lie within a fiber bundle of(D − 2)-spheres on the base space

∑

i νi ≡ 1, and threadthrough the pinched fibers at the rest points. App. E 2uses these properties of the L = 0 manifold to efficientlyconverge to the heteroclinic orbits by gradient descent.The resulting escape trajectory for the coordination gameis shown in Fig. 1.

L R

M

0 10 20 30 40 50 60

0

2

4

6

8x 10

−3

η . ν

.

τ

FIG. 1. The result of the steepest-descent method, developedin App. E 2, gives the most-probable trajectory for escapefrom the stable ESS in the coordination game of Sec. IV Cin the companion paper. The heavy curve is the escape tra-jectory, which differs slightly from the contour of slowest flow(convergence of the thin lines) of the classical stationary tra-jectories. Plotted below is the numerical value of η · ν alongthe escape contour. Abscissa is rescaled time τ , and the in-tegral of

∫

dτ η · ν = S, which here evaluates to 0.1123. Wehave shown in Ref’s. [63, 64] this analytic approximation forlarge-deviations rate functions agrees well with numerical sim-ulations (which become costly for rare events).

By Eq. (55), we also see that the only contribution

to S(0) comes from the “kinetic” term∑D

i=1 ηidνi/dτ .The value of this quantity, along the escape trajectory,is plotted inset in Fig. 1. The integral under the curveis the rate function [6] S(0) for the escape probability,

which is the inverse log-residence time at rest points.B. The Gaussian approximation for fluctuations,

correlation functions, and corrections to mean-field

dynamics

We now turn to effects associated with the Gaussianfluctuation kernel e−S(2)

. We first consider the distinctiveform of this kernel associated with all MSR field theories,which expresses causality, and also determines the correctstructure for Langevin approximations. We then con-sider the divergence of fluctuations at the critical pointsfor symmetry breaking, first in the uniform and then inthe ordered phase. For brevity we will only develop theRock-Paper-Scissors game; the case of discrete symmetrybreaking in a system similar to the coordination game isdeveloped in Ref’s. [63, 64]. The emergence of Brownianmotion along the limit cycle in the Rock-Paper-Scissors(RPS) game leads to a more general discussion of time-translation symmetry, which allows us to compute theformal sum over instantons, completing the discussion ofnon-classical stationary points in the preceding section.

1. The tri-diagonal form and causality

We observed in Eq. (51) that the Hessian of L de-fines an effective-mass tensor for real-valued excursionsof η − η0 in the non-classical stationary paths. Aboutany stationary path, the remaining fluctuations of η mustgenerally be rotated to an imaginary contour, whichis the stable contour of integration for the path inte-gral (32). In this section we consider the effects associ-ated with fluctuations along the rotated contour, whoseimaginary part is analogous to the kinematic momen-tum in the real potential of a finite-temperature Hamil-tonian system [33]. The imaginary parts may be useddirectly, to compute an “imaginary Langevin equation”,or they may be integrated out, to form an action involv-ing only the fields ν and their time-derivatives, known asthe Onsager-Machlup action [65]. In App. F 1 we developa third formulation, which uses a Hubbard-Stratonovich

transformation [66] to produce the conventional, real-valued Langevin equation. Whatever approach is used,the quantities controlling the rate of injection of noise,and its rate of damping, are the second partial deriva-tives that make up S(2).We will emphasize the distinctive “tri-diagonal” form

of S(2), whose meaning was first clearly understood byKeldysh [44] in the context of quantum mechanics, butwhich enforces certain constraints on all orders of fluctu-ation corrections in either classical or quantum Hilbertspaces [49]. We consider only expansions about the clas-

sical backgrounds,(

φ†, φ)

=(

1, φ)

+(

φ, φ′)

, which are

sufficient to demonstrate the method and are computa-tionally simple. The tri-diagonal form comes from theability to organize all non-vanishing terms in S(2) intothe bilinear form

14

S(2) =1

2

∫

dtD∑

i,j=1

[

φ′i φi

]

0 −δijdt +∂2L

∂φi∂φ†

j

δijdt +∂2L

∂φ†

i∂φj

∂2L

∂φ†

i∂φ†

j

[

φ′j

φj

]

. (56)

The zero in the upper-left corner arises because φ† ≡ 1

causes vanishing of all terms at order (φ′)2. This property

is preserved even if fluctuation corrections are incorpo-rated back into the average dynamics to any order, be-cause it is an expression of causality of the dynamics [49].In any Gaussian integral with such a tri-diagonal kernel,the expected covariance of fields φ, φ′ (the matrix inverseof the kernel), also has tri-diagonal form,

⟨[

φ′

φ

]

t

[

φ′ φ]

t′

⟩

=

[

DKt;t′ DR

t;t′

DAt;t′ 0

]

. (57)

The D matrices are the Green’s functions of the func-tional integral, which are the propagating waves of re-sponse to pointlike sources. DA and DR describe, respec-tively, the time-Advanced and time-Retarded responsesof φ′ to φ. DK is the correlation function for fluctuationsφ′. From standard methods of matrix inversion [45, 49],it follows that the retarded and advanced Green’s func-

tions in the continuous-time limit satisfy the equations(

dtδij +∂2L

∂φ†i∂φj

)

DRj,t;m,t′ ≡ δimδ(t− t′) (58)

(

−dtδij +∂2L

∂φi∂φ†j

)

DAj,t;m,t′ ≡ δimδ(t− t′) . (59)

Their solutions are, respectively,

DRt;t′ = θ(t− t′) T exp

{

−∫ t

t′dz

∂2L∂φ†∂φ

∣

∣

∣

∣

∣

z

}

(60)

DAt;t′ = θ(t′ − t) T −1 exp

{

−∫ t′

t

dz∂2L

∂φ∂φ†

∣

∣

∣

∣

∣

z

}

=(

DR†)

t;t′.

(61)

Here T and T −1 denote forward and reverse time-

ordering operators on their respective exponential inte-grals, defined so that for a general time-dependent matrixAz,

12

T exp

{

−∫ t

t′dz Az

}

≡ limdz→0

(1− dzAt) (1− dzAt−dz) · · · (1− dzAt′) (62)

T −1 exp

{

−∫ t′

t

dz Az

}

≡ limdz→0

(1− dzAt) (1− dzAt+dz) · · · (1− dzAt′) . (63)

The correlation function DK , named afterL. D. Keldysh, has a formally simple relation tothe retarded and advanced Green’s functions due to thetri-diagonal kernel. Its general solution can be written

DKt;t′ = DR

t;t′Mt′ +MtDAt;t′ , (64)

in terms of a time-local matrix M satisfying

dtM +∂2L

∂φ†∂φM +M

∂2L∂φ∂φ†

= − ∂2L∂φ†2

. (65)

We have shown elsewhere [45] that Eq. (65) is an expres-sion of the fluctuation-dissipation theorem, in which dis-

sipation comes from the terms ∂2L/∂φ∂φ† and its conju-

gate, and the fluctuation source from the term ∂2L/∂φ†2.

12 In equations (60,61) and in what follows, it is necessary to ex-plicitly time-order as it was not in the original quadrature (26)of the Liouville equation, because L depends on the backgroundφ, which may be a function of time.

Because the diffusion kernels in equations (60,61)may have zero eigenvalues, we must generally integrateEq. (65) over finite rather than infinite times, so that thesolution for M at two different times t′′ and t > t′′ obeysthe relation

Mt = DRt;t′′Mt′′DA

t′′;t −∫ t

t′′dzDR

t;z

∂2L∂φ†2

∣

∣

∣

∣

∣

z

DAz;t. (66)

Then, completing the solution to Eq. (64), a similar re-lation for DK at (t, t′) both > t′′, is

DKt;t′ = DR

t;t′′Mt′′DAt′′;t′ −

∫ min(t,t′)

t′′dzDR

t;z

∂2L∂φ†2

∣

∣

∣

∣

∣

z

DAz;t′ .

(67)In equations (66,67), the positive eigenvalues of the

kernel ∂2L/∂φ†∂φ damp out correlations from earlier

times, while the noise kernel −∂2L/∂φ†2 continuouslycontributes new fluctuations, as it does in the Langevin

15

formulation of App. F 1. For damped fluctuations, thefinite-time integral (66) converges exponentially to theinfinite-time integral. In the case where the diffusionkernels have zero eigenvalues, the undamped fluctuationsaccumulate linearly in time as Brownian motion.

As we will show below, all time-dependent stationary-

path solutions lead to zero eigenvalues of ∂2L/∂φ†∂φ.Most of these will not concern us, as they result fromtransient relaxation of initial conditions, and cannot leadto universal or persistent fluctuation effects. The twocases that will concern us are the limit cycle of RPS,in which time dependence persists indefinitely, and thetransient instanton solutions of the coordination game,for which removal of the apparent fluctuations leads tothe correct measure for the dilute-gas instanton sum.

The fluctuation at equal times, 〈φ′tφ

′t〉, which will be

useful for studying corrections to the mean replicatorequations, is a degenerate case of the Keldysh correla-tion function,

DKt;t → Mt. (68)

2. Example: divergence of fluctuations about the uniformbackground near a critical bifurcation in Rock-Paper-Scissors

Critical bifurcations in the deterministic replicator willgenerally be associated with divergences in one or moremodes of the fluctuation correlation function (68). Herewe compute these for the cyclically symmetric Rock-Paper-Scissors game of Sec. IV D of the companion pa-per, beginning with the uniform background.

Recall that the critical bifurcation in RPS occurs whenthe steady-state radius r first takes nonzero real values.13

App. F 2 provides specific forms for the partial derivativesof L appearing in the Green’s function and noise source.In particular, Eq. (F14) shows that the two eigenvectors

of ∂2L/∂φ†∂φ have real parts proportional to −r2, whichpass through zero linearly in a− b at the critical point.

For the time-independent uniform background, thetime-ordered exponentials become simple exponentials,and the integrals for DK may be solved by diagional-ization. The resulting expression for the φ′ correlation

function becomes

〈φ′iφ

′l〉t = N

[

3(

a+ b−a2

)

− 2(b−a)D

]

∣

∣a− b− 2D2

N

∣

∣

(

δil −1

D

)

. (69)

This is a typical result for Gaussian fluctuations abouta mean-field background: a divergence proportional to1/ |a− acrit|, where a is a control parameter and acrit itsvalue at the bifurcation, in this case b + 2D2/N . Thisfunctional form (identical to its counterpart in action-angle variables at large N) is compared and shown toagree closely with statistics from simulations in Fig. 17of the companion paper.

3. Time translation of classical solutions and the Goldstonetheorem for transients and large deviations

Next we consider the spectrum of fluctuations in RPSin the ordered-population regime. For any stationarypath of S, if φ†(t), φ(t) is a solution so must be φ†(t− δt),φ(t− δt), for any finite δt. The resulting variation δSmust vanish identically, and we may expand the termsin δS in a power series in δt, with the coefficient of eachpower required to vanish independently. For classical sta-tionary points (the only cases we will consider explicitly),the leading term in δt yields the result known as Gold-

stone’s theorem for the inverse of DR,

0 =

D∑

j=1

(

δijd

dt+

∂2L∂φ†

i∂φj

)

dφj

dt. (70)

Here φ is the background satisfying S(1) = 0. The math-ematical statement of Goldstone’s theorem, in terms ofrepresentations of symmetry, is that the generator of the

symmetry hidden by the background (in this case, timetranslation, or d/dt), acting on that background, mustproduce a fluctuation with zero eigenvalue in S(2).The expression (67) for DK is solved in App. F 3 in

action-angle fields. For ordered populations near thecritical point, the limit cycle is approximately circular,and the symmetry axes for fluctuations are well approx-imated by polar coordinates in ν. More general cases ormore complex limit cycles may be treated with a Floquet

analysis, as demonstrated in Ref’s. [30, 31]. Letting r bethe radial coordinate on the ν-simplex, and θ the angleabout the uniform population that gives the “phase” onthe limit cycle, and expanding each of these in terms ofa background

(

r, θ)

and fluctuations (r′, θ′), the leadingsmall-r solution to Eq. (67) is taken from Eq. (F39), as

13 The mean-field equation for r is given as Eq. (41) in the com-panion paper, and shown to be consistent in the more general

Floquet analysis of App. F 3 below.

16

⟨[

r′

rθ′

]

[

r′ rθ′]

⟩

t

→[

01

]⟨[

r′

rθ′

]

[

r′ rθ′]

⟩

t′

[

01

]

+

(

a− a− b

3+

1

N

)

[

D2N

1∣

∣a−b− 2D2

N

∣

∣

t− t′

]

+O(

R)

.

(71)

The Goldstone mode is the tangential coordinate rθ′, inwhich noise accumulates linearly between any two timest and t′. Fluctuations in r′ are mean-regressing and ac-cumulate to a distribution of finite width. This widthdiverges on approach to the critical point from the or-dered phase.Eq. (F10) in the appendix shows, as an immediate

consequence of the symmetry of the trace of the diffu-sion kernel about the critical point, that the sum of thedamping eigenvalues is continuous through the bifurca-tion. Since, in the ordered phase, only fluctuations nor-mal to the limit cycle are damped, this damping rate istwice that of either fluctuation about the origin in the dis-ordered phase, as shown in Eq. (71) relative to Eq. (69).About the uniform background, by isotropy, a complexconjugate pair of eigenvalues is responsible for what havebeen termed quasi-cycles in the noise spectrum [67]. Un-der the transformation to the Frenet frame on the limitcycle [30], the eigenvalues become real, one of them ex-actly zero. The other is continuous with the sum of realparts of the eigenvalues in the uniform phase and thus adoubled rate of damping.With these results we have illustrated the main fea-

tures of Goldstone’s theorem for stochastic processes.The first is the linear accumulation of Brownian noise,which governs the relaxation of any initial populationstate toward the uniform distribution over the limit cy-cle at late times, restoring the time-translation symmetryof the underlying dynamics. We recognize that the clas-sical solution – which fails to account for decoherence –should be regarded as the leading-order term in a condi-

tional expectation, conditioned on the population stateat some reference time. Conditioning breaks ergodicityof the stationary distribution, and the conditional corre-lation may be analyzed to reveal quite intricate structureof fluctuations about the limit cycle [30, 31]. Our secondresult is the quantitative relation between this rate ofdecoherence and the width of the distribution for mean-regressing fluctuations. Our third result is the “conser-vation” of the rate of damping across the critical tran-sition, which causes the damped fluctuations in the or-dered phase to regress more strongly (because there arefewer of them) than their counterparts about the uniformpopulation background.

4. Fluctuations about non-classical stationary paths, andthe continuous sum over instantons

The same observations that lead to the StochasticGoldstone theorem for the limit cycle of RPS apply alsoto the non-classical stationary paths giving escape tra-

jectories in the coordination game. Like the backgroundof the limit cycle, instanton paths are time-dependent,and must create a zero-eigenvalue fluctuation. Unlike thelimit cycle, individual instantons have effectively finiteduration, and so do not accumulate Brownian noise for-ever. Instead, they do something more subtle [33]: theyintroduce into the fluctuation spectrum an apparently-divergent eigenvector. The resolution of the paradoxof this divergence is that this direction of fluctuationsshould not be integrated along the imaginary contour,but rather should be retained within the real contour ofdeformations of η, converting a naıve discrete sum overmultiple instantons into an integral over their positions aswell. The fluctuation corrections to non-classical station-ary paths thus bring together both roles of the Hessianof L, as effective-mass matrix and as noise source.The identification of the correct instanton sum goes

as follows: to ensure a convergent expansion in Gaus-sian moments [32, 33], the separation of the action fromEq. (36) should be made about each stationary back-ground φ†, φ, so that the generating functional (32) be-comes a sum

(0| e∑

mam |ΨT ) =

∑

φ†,φ

e−S(0)

∫ T

0

DφDφ′e−(S−S(0))e(φ†

0−1)(n−φ0). (72)

Each S(0) is to be evaluated at the appropriate back-ground, which may contain one or many instantons, andthe integral is to be left with only convergent Gaussianmoments about that background.The formal sum in Eq. (72) includes a discrete count

over the number k ∈ 0, . . . ,∞ of escape events, andfor each k the full set of allowed solutions must con-tain the integral over all possible time-ordered positions{t1, . . . , tk} for the event centers. Since this integral addsto each background φ†, φ its time derivative, it alreadyincludes the zero-eigenvalue mode of fluctuations, whichtherefore must not be redundantly included in the Gaus-sian integral. The time-derivative of the escape path isa finite and hence normalized mode, so that the individ-ual integrals over ti differ from the component dφ dφ′ forthe zero-eigenvalue mode only by a normalization, whichwe denote tesc. Denoting the functional measure with-out the zero-eigenvalue mode as D′φD′φ′, we have for asingle escape

∫ T

0

DφDφ′ =

∫ T

0

dt

tesc

∫ T

0

D′φD′φ′. (73)

(For methods to compute this normalization constant,see Ref. [33], Ch. 7, App. 2.)

17

Because the functional measure was defined byEq. (33) as a product, the normalization for removal ofa fluctuation with k escapes is simply tkesc. Similarly, thevalue of S(0) for a k-escape trajectory is k times that for

a single escape, which we denote S(0)esc . The geometric

restriction that escapes happen in order may be simpli-fied by introducing the time-ordering operator T , as was

used to define Green’s functions, this time applied to anunrestricted integral over the times of the escape events.One then divides by the factor k! by which a product ofunrestricted time integrals overcounts the configurationspace for sequential escapes. The result is that a fullyregular expansion of the generating functional includingall escapes may be written

(0| e∑

mam |ΨT ) =

∞∑

k=0

e−kS(0)esc

k!T∫ T

0

dt1tesc

, . . . ,

∫ T

0

dtktesc

∫ T

0

D′φD′φ′e−(S−S(0))e(φ†

0−1)(n−φ0). (74)

The mean-regressing fluctuations in the remaining Gaus-sian integral of Eq. (74) will be finite, and away fromthe critical point will lead to at most polynomial correc-tions to the exponential prefactors. Therefore the factors

e−S(0)esc and integrals over ti collapse to an exponential

sum of the form

e−rescT ,

in which the escape rate is given by

resc =e−S

(0)esc

tesc. (75)

The correction of the Gaussian measure was the stepneeded to relate the dimensionless probability term

e−S(0)esc to a rate by defining the characteristic timescale

tesc over which fluctuations sample the basin of attrac-tion. Because tesc depends at most polynomially on N ,

S(0)esc ≡ NS

(0)esc defines the leading exponential dependence

of the escape rate on N . The coefficient S(0)esc is given, for

example in the coordination game, by the integral shownin Fig. 1.Escapes from domains of attraction, in systems whose

population states break a discrete symmetry, have a sim-ilar effect to Brownian motion around the limit cycle forsystems that break time-translation. They cause condi-tional expectations on the short term to relax toward theergodic distribution over population states at late time,restoring the expression of the hidden symmetry. Bothmay be used as probabilities of elementary transitions ina coarse-grained description, in which populations be-come the elementary entities. The calculations madetractable by Freidlin-Wentzell theory therefore providethe needed connection between individual-level dynam-ics, and the aggregated dynamics of emergent entitiesensured by hidden symmetries on slower timescales.

C. Fluctuation-induced corrections to average

dynamics, and the evolutionary entropy

Our last fluctuation effect concerns corrections to themean stationary paths, in models that contain neutral

directions.S(2) is only the correct kernel for Gaussian fluctua-

tions, on the background identified by S(1) = 0, in thecase that S(3) and higher-order terms in the Taylor’s se-ries for the action do not on average produce values linearin φ or φ′, in the distribution that they themselves helpshape. If this approximation is not good enough, ratherthan use Eq. (36), e−S in the generating functional (32)can be expanded as

e−S = e−(S(0)+S(2))

(

1− S(1) − S(3) + . . .)

. (76)

The expansion is initially performed about a general

background, which is then chosen so that the linear termsin the parenthesis vanish on average. For correctionsto classical stationary points where φ† ≡ 1, the iden-tification of self-consistent backgrounds is particularlyeasy. For all backgrounds φ, S(0) ≡ 0, and the onlynon-vanishing term in S(3) leads to the correction to theequation of motion for φ,

dφi

dt+

∂L∂φ†

i

+1

2

D∑

m,n=1

∂3L∂φ†

i∂φm∂φn

〈φ′nφ

′m〉 = 0. (77)

The fluctuation strength 〈φ′nφ

′m〉 is given by the equal-

time correlation function in equations (57,66,68), com-puted about the background φ. The forms for the deriva-tives of L required to evaluate Eq. (77) are provided inApp. D 1.Stationary paths of the form (77) are the minima

of a function of the background fields known as theeffective action. Effective actions are computed byfirst introducing time-dependent sources into the pathintegral (32) to produce a moment-generating func-

tional, whose logarithm is called the cumulant-generating

functional. The functional Legendre transform of thecumulant-generating functional is the effective action. Inquantum field theory, this functional (of classical back-ground fields) is known as the quantum effective ac-

tion [68]. It is the basis for a wide variety of background-field methods, and renormalization schemes based onthem. We have termed its counterpart for classicalstochastic processes the “stochastic effective action”, and

18

developed its role in large-deviations theory and a ther-modynamics of histories in Ref. [5]. The stochastic ef-fective action is also known in Freidlin-Wentzell theoryas the quasipotential [3], because it is a path-space gen-eralization of the equilibrium (Helmholtz or Gibbs) freeenergy. Usually S(0) is used as a leading approximationto the quasipotential.The arguments of the stochastic effective action are the

leading-order summary statistics for the correlation func-tions of Markovian stochastic processes. Elegant graph-ical methods exist [68] for defining and computing thecumulant-generating functional and effective action, butto leading order in this problem, the only result is a shiftin the Liouville operator to

Leff = L+1

2

D∑

m,n=1

∂2L∂φm∂φn

〈φ′nφ

′m〉+ . . . . (78)

We will not attempt to compute here the correctionsto dynamical stationary paths, which require the evalua-tion of time-ordered exponentials about dynamical back-grounds in the solution (67) for 〈φ′

nφ′m〉. We will com-

pute corrections to asymptotic steady states, by solvingEq. (77) for the flow to fixed points, but evaluating bothderivatives of L and 〈φ′

nφ′m〉 as if the backgrounds were

constant. The resulting flow will deviate from the truefluctuation-corrected equations of motion, but will con-verge to it in neighborhoods of the fixed points wherechange of the true backgrounds becomes slow.The fluctuation corrections in stochastic evolutionary

games have the same origin as the entropy correctionsin thermodynamics, which distinguish internal energiesU from free energies F ≡ U − TS [69]. Averaging aGibbs function e−H/kBT (where H is the Hamiltonian),in an ensemble whose fluctuations are governed by H,yields the partition function e−F/kBT . In the Freidlin-Wentzell sum over histories, L is the Hamiltonian, andits self-consistent average is Leff.

To make this correspondence more explicit, noticethat, in the generating functional (32), if all values(

φ†, φ)

are integrated except those at a single time, theremaining integral defines a density over a single set ofcoherent-state fields,

1 =

∫ T

0

Dφ†Dφ e−Se(φ†

0−1)·(n−φ0) ≡∫

dDφ†t d

Dφt

πDWφ†

t ,φt.

(79)Wφ†

t ,φtis called theWigner function corresponding to the

probability density ρn at time t. It is the distributionover the continuous, but overcomplete, set of coherent-state indices at a single time which gives the equivalentweight function to the one given by ρn for number states.At late times in a system whose stationary paths con-verge to a stable rest point, both ρn and Wφ†,φ becomet-independent. The stationary-path condition computedby varying the stochastic effective action, with time-independent backgrounds, simply equals the average of

the classical stationary-path condition in the Wigner dis-tribution for fluctuations about that average:

∂Leff

∂φ†i

≡ ∂L∂φ†

i

+1

2

D∑

m,n=1

∂3L∂φ†

i∂φm∂φn

〈φ′nφ

′m〉

=

∫

dDφ†dDφ

πDWφ†,φ

∂L∂φ†

i

. (80)

All the low-order approximations that have been de-rived here for coherent-state fields have counterparts inaction-angle variables, and specific forms are provided inApp. D 2. The corrections to the rest points due to thesefluctuations are shown in Fig. 2, and are reproduced insimplified form and compared to simulation results inthe companion paper. The circles without axes drawnthrough them in the figure are the naıve mean-field restpoints for a sequence of increasing N .14 The mean popu-lation compositions, when self-consistently averaged overfluctuations, are shown as the circles with crosses. Thecrosses identify both the major and minor axes of thenoise covariance as it would appear in a Langevin equa-tion for this system (heavy small axes), and the result-ing fluctuations in population state from the combinationof noise accumulation and relaxation (light large axes).Both sequences have regular limits at large N

The naıve mean-field rest points differ from the evalu-ation of Eq. (77) by ∼ 10%. In contrast, Eq. (77) con-verges at large 1/N to simulation values within ∼ 1%.Since we lack a formal small parameter such as 1/N tocontrol the fluctuation expansion, it was not assured thatthe first-order correction alone would produce such closeagreement.15 Proofs of convergence of the all-orders ex-pansion remain a topic for future work.We have thus shown how fluctuations in a model, de-

fined at the individual level with parameters that re-flect the actual individual interaction strengths, givesrise to a best-fit description at the population level withshifted parameters, even for infinite populations. As forthe microscopically-defined L, Leff may be decomposedinto fitness and transmission terms according to the re-gressions of the Price equation, and these will specify anormal-form game and mutation matrix. The parame-ters in those functions will not be the ones directly expe-rienced by agents, but we have shown how the relationsbetween the two may be systematically computed.

14 N values used were, from left to right in the figure: 90, 100, 150,200, 250, 300, 350, 400, 450, 500, 600, 700, 900.

15 Note that the significance of convergence between the Gaussianfluctuation model and simulations at large N is not due to ex-pansion of the orders of fluctuation, but rather to the fact thatthe model only is only nearly neutral within a domain of distance1/N from the boundary. It is necessary that the mean-field non-neutrality in the interior give way to control by fluctuations,before the simplified Gaussian fluctuation expansion is expectedto become a good approximation.

19

D C

T

FIG. 2. Self-consistent solutions for rest points of the meanbehavior of the (infinite-population) stochastic replicator withneutral directions show systematic deviation from the restpoints of the deterministic replicator. These deviations existfor all finite mutation rates and converge to a finite differ-ence in the zero-mutation limit, where both rest points lie onthe boundary of absorbing states. The persistence of bothdifferences in the mean behavior, and fluctuations involving afinite but non-vanishing fraction of the total population in thislimit, can be handled by including the non-equilibrium coun-terpart to “entropy” corrections to the frequency-dependentfitness functions.

IV. DISCUSSION AND CONCLUSIONS: THE

NON-EQUILIBRIUM STATISTICAL

MECHANICS AND THERMODYNAMICS OF

STOCHASTIC EVOLUTIONARY GAMES

In this pair of papers – the companion paper thatdiscusses symmetry, and the current paper’s presenta-tion of Freidlin-Wentzell methods – we have introduceda fairly complete suite of stochastic-process methods forthe framing and analysis of evolutionary games. We havecovered the definition and estimation of game models,methods of analysis, the roles played by collective fluctu-ations, and a fairly broad and representative set of casesin which fluctuations can cause the stochastic formula-tion to differ in important ways from naıve deterministicapproximations.

Our methods have not been directed at proofs of con-vergence of large-deviation limits, but rather at trans-parent and intuitive expressions for multiscale dynam-ics. The principle elements that benefit from this sim-plification are extraction of the Fokker-Planck approxi-mation from the master equation (12), the equivalenceof the discrete-index master equation to the field inte-gral (43), and the extraction of both classical diffusionsolutions, and escape paths suppressed by exponentiallarge-deviation probability terms, from the stationary-

point equations (44–47).We summarize here the sense in which the construc-

tions of these papers define the statistical mechanics andthermodynamics of evolutionary games, and the way afully scale-dependent treatment of symmetry breakingand symmetry restoration is related to the analytic anddynamical-system properties of the different approxima-tion methods.

A. Entropy corrections, and the systematic

derivation of effective theories for games

We have shown how the regression approach to as-signing mechanisms can depend on scale, and why itoften must. In particular, we have shown the impor-tance of symmetry as both a source of scale-dependence,and a constraint on its forms. We have limited bothpapers to purely technical problems of analysis, but webelieve that the discipline enforced by consistent fram-ing and interpretation of stochastic evolutionary gamesalso contributes to a clearer understanding of the (scale-dependent!) concepts of agency and structured interac-tion.The tools we have presented for evaluating corrections

from fluctuations also may be used to define consistentclosure conditions for models, under the interpretationof self-consistent, renormalized effective theories [48, 49].The effective Liouville operator introduced in Sec. III C,although only renormalized in a trivial sense, demon-strates how the average of nonlinear interactions involv-ing fluctuations can be re-absorbed in the stationary-path conditions. It has the interpretation of the firstvariation of a quantity we have termed the stochastic ef-

fective action [5], which is a functional Legendre trans-form of the Freidlin-Wentzell generating functional.Through the explicit definition of the Wigner distri-

bution at a single time, we have shown how the correc-tions in the stochastic effective action arise as entropyterms. The fact that the Wigner distribution can as-sign correlated fluctuation to both the field values andtheir conjugate momenta – where the conjugate mo-menta have the interpretation of expectations of moment-sampling weights – shows how an entropy over pathsmay be well-defined and yet differ from the equilibriumentropy even at a single time. In this way we demon-strate both the form and the meaning of the Lagrange-Hamilton dual structure for the thermodynamics of non-equilibrium processes.

B. Multiscale dynamics, and the breaking and

restoration of ergodicity

Our analysis of fluctuation effects – for Brownian mo-tion in the Hopf bifurcation in Sec. III B 3 and for domainflips in the pitchfork bifurcation in Sec. III B 4 – providesthe timescale-dependent treatment of symmetry break-

20

ing and symmetry restoration promised in the compan-ion paper. An approach to defining ergodicity breakingwith respect to explicit timescales comes, in large part,from the interaction of ray approximations with diffusionequations.The diffusion equation with moment corrections is a

power-series expansion (usually in 1/N , though, as wehave seen in the RPD example, not always). The ray ap-proximation captures terms that do not converge to theirTaylor’s series expansion in any radius about 1/N → 0.Therefore effects associated with the two approximationsare distinguished by their analytic structure as well as bycharacteristic scaling.The ray approximation describes conditional probabil-

ities of observables over entire histories, and inherentlytreats these probabilities as if coherent order persists in-definitely. Fluctuations about the ray background thenprovide the characteristic timescale on which the coher-ence approximation becomes invalid, and the mechanismby which population dynamics restores the underlyingsymmetry hidden by population states on the short term.The contrasting examples of the pitchfork and Hopf

bifurcations show that the broken ergodicity of discretepersistent states can be restored by other trajectories(domain flips) which are themselves ray-approximate inorigin, whereas the continuous persistent histories of alimit cycle are restored to an ergodic distribution aroundthe cycle by simple diffusion. In the former, the time torestore ergodicity grows exponentially in population sizeN , while in the latter it approaches a constant of orderunity as N → ∞.

Acknowledgments

We are grateful for discussion and references to CosmaShalizi, Duncan Foley, and Martin Nowak. DES thanksInsight Venture Partners for support of this work, andthe Program for Evolutionary Dynamics at Harvard forhospitality in 2006. SK is funded by the Swedish Re-search Council.

Appendix A: Systematic construction of transfer

matrices

Here we derive, in the notation appropriate to linear-order normal-form evolutionary games, a construction fora minimal continuous-time Moran stochastic process,16

from its first-moment conditions. The criteria of mini-mality are lowest polynomial order of the transition prob-abilities, and no new parameters beyond those in the fit-ness and mutation matrices. The derivation follows the

16 The Moran process is characterized by single birth/death eventsin sufficiently short time intervals [41].

general form of Ref. [5]. The construction readily gen-eralizes to k-player normal form games. The approachcan also be extended to processes incompatible withthe Moran model, involving simultaneous birth/deathevents, by the addition of higher-order “contact terms”for multi-agent shifts.First the payoff matrix is written as a sum of antisym-

metric and symmetric parts,

aij ≡ aAij + aSij . (A1)

Then Eq. (3) is expanded in terms of these as

dni

dt=

D∑

j=1

niaAij nj

+D∑

j=1

niaSij (nj − δij)−

ni

N

D∑

k,j=1

nkaSkj (nj − δkj)

+

D∑

j=1

µij nj . (A2)

(The matrix µ is stochastic, so that its transpose vanishesin the sum over j.) The exchange antisymmetry betweenthe (summed) index j and the (not-summed) index i is

further exposed by introducing a factor of∑D

k=1 nk/N ≡1 into the first term on the second line, to give

dni

dt=

D∑

j=1

niaAij nj

+ni

N

D∑

j,k=1

nj

[

aSik (nk − δik)− aSjk (nk − δjk)]

+D∑

j=1

µij nj . (A3)

The sum of Eq. (A3) on i now vanishes manifestly by theantisymmetry of its argument.Population size N regulates the strength of fluctua-

tions, which we wish to hold at a fixed scale for ease ofanalysis. It is as easy to conserve N exactly as in an en-semble average, so we will restrict attention to transfermatrices that conserve probability over configurations ofseparate N independently. All such matrices must trans-fer probability through equal numbers of death and re-production events. The operator for the Moran process(type conversion of a single agent), which acts on anyfunction of n to shift the index by +1i − 1j , is

e∂/∂ni−∂/∂nj .

For continuous-time master equations, we require thatall terms be built from single-agent shift operators ofthis form.17 Because only net conversion from types

17 Including multiple-agent type conversions that are correlated

21

i to j appears in the deterministic replicator, only theantisymmetric combination sinh

(

∂/∂ni− ∂/∂nj

)

will beconstrained by the first-moment equation (A3).

The correspondence is then constructed as follows. Forany function An with the expectation

∑

n Anρn denoted〈A〉, these shift operators, by shifting the index of sum-mation, lead to the identity

∑

n

nj sinh

(

∂

∂ni

− ∂

∂nj

)

Anρn

=∑

n

Anρn sinh

(

∂

∂nj

− ∂

∂ni

)

nj

= 〈A〉 . (A4)

In particular, taking An to be the function∑D

j,l=1 nlaAljnj

defined from the decomposition (A1), yields

1

2

∑

n

ni

D∑

j,l=1

sinh

(

∂

∂nl

− ∂

∂nj

)

njaAjlnlρn =

D∑

j=1

⟨

niaAijnj

⟩

.

(A5)

If correlated fluctuations are ignored, and 〈ninj〉 isreplaced by ninj , Eq. (A5) yields the first line ofEq. (A3). A similar expansion with An set to∑D

j,k,l=1 njaSjk (nk − δjk)nl gives

1

N

∑

n

ni

D∑

j,k,l=1

sinh

(

∂

∂nl

− ∂

∂nj

)

njaSjk (nk − δjk)nlρn =

D∑

l,k=1

⟨

ninl

NaSik (nk − δik)

⟩

−D∑

j,k=1

⟨ni

Nnja

Sjk (nk − δjk)

⟩

,

(A6)

which in mean-field approximation is the second line inEq. (A3). Finally, the antisymmetric component of thecombination µijnj is extracted with the identity

∑

n

ni

D∑

j,l=1

sinh

(

∂

∂nl

− ∂

∂nj

)

µjlnlρn =

D∑

j=1

〈µijnj〉 ,

(A7)giving the third line of Eq. (A3).The sum of the three terms (A5 - A7) gives an un-

symmetrized representation for the mean-field compo-nent of the transfer matrix,

∑

n′

TMFnn′ ρn′ ≡

D∑

j,l=1

sinh

(

∂

∂nl

− ∂

∂nj

)

{

nj

[

1

2aAjl +

1

N

D∑

k=1

aSjk (nk − δjk)

]

+ µjl

}

nlρn. (A8)

Since only the antisymmetric part of the braces in Eq. (A8) avoids cancellation against the sinh, and since it willbe useful to have explicit antisymmetric forms to motivate the noise component of Tnn′ in the text, we extract theantisymmetric part of the braces explicitly as

∑

n′

TMFnn′ ρn′ =

1

2

D∑

j,l=1

sinh

(

∂

∂nl

− ∂

∂nj

)

{

nj (nl − δjl) aAjl +

1

N

D∑

k=1

nj (nk − δjk) aSjknl + µjlnl

− 1

N

D∑

k=1

nl (nk − δlk) aSlknj − µljnj

}

ρn. (A9)

Eq. (A9) implicitly includes self-play (and selection of a player against itself) in the combinations of n coefficientsmultiplying aSjk and aSlk. A pair of completely cancelling terms, which remove double-counting of any agent from

both of these aS coefficients, may be introduced to put the mean-field transfer matrix in a form that includes onlyinteractions of distinct agents,

∑

n′

TMFnn′ ρn′ =

1

2

D∑

j,l=1

sinh

(

∂

∂nl

− ∂

∂nj

)

{

nj (nl − δjl) aAjl +

1

N

D∑

k=1

nj (nk − δjk) aSjk (nl − δjl − δkl) + µjlnl

− 1

N

D∑

k=1

nl (nk − δlk) aSlk (nj − δlj − δkj)− µljnj

}

ρn. (A10)

over arbitrarily short times is a kind of “fine tuning”. It requires specific empirical motivation to be introduced at the individual

22

reproduced as Eq. (9) in the text. Although Eq. (A10)appears cumbersome with its extra δ-terms, it will yieldthe simplest form for the algebraic and field-theoreticgenerating functionals, derived in App. B.

Appendix B: Conversion between transfer matrices

and Liouville operators

Here we construct the mapping between a transfer ma-trix acting on a distribution ρ and the Liouville opera-tor acting on its associated generating function Ψ. Inthe Liouville equation (14), the left-hand side dΨ/dt ismade a function of dρ/dt by definition (13), with dρ/dtevaluated according to the master equation (1) and the

form (11) for the transfer matrix. Factors of z and ∂/∂zare chosen on the right-hand side so that L will repro-duce the terms in Tnn′ , multiplying each basis function∏D

m=1 znmm . The elementary terms that contribute to the

transfer matrix (11) are factors of ni, and the single-agentshift operators exp(∂/∂nl − ∂/∂nj)− 1.To extract these, again let An be any function of the

components of n defining the context in which an indexappears. The combination

∑

n

(

D∏

m=1

znmm

)

niAnρn = zi∂

∂zi

∑

n

(

D∏

m=1

znmm

)

Anρn

(B1)simply extracts the number index. When multiple deriva-tives act in turn, they recover number indices with cor-rections so that no factor of zm is multiply counted,

∑

n

(

D∏

m=1

znmm

)

ni (nj − δij)Anρn = zizj∂

∂zj

∂

∂zi

∑

n

(

D∏

m=1

znmm

)

Anρn. (B2)

The shift operator in any context is even more simply obtained as

∑

n

(

D∏

m=1

znmm

)

(

e∂/∂nl−∂/∂nj − 1)

Anρn =

(

zjzl

− 1

)

∑

n

(

D∏

m=1

znmm

)

Anρn. (B3)

Repeated application of suitably ordered z and ∂/∂z as in equations (B1,B2), followed by Eq. (B3), shows that thetransfer matrix (11) is produced by the Liouville operator

L(

z,∂

∂z

)

≡D∑

j,l=1

(zl − zj)

(

µjl + zjajl2

∂

∂zj+

1

N

D∑

k=1

zjzkaSjk

∂

∂zj

∂

∂zk

)

∂

∂zl. (B4)

reproduced as Eq. (15) in the text.

Appendix C: From complex functions to abstract

vector spaces

A systematic treatment of the transformation from an-alytic functions to abstract Hilbert-space representationsof generating functions is provided in Ref. [5]. Here wereproduce the essential relations to make the presenta-tion self-contained.

1. The correspondence of elementary objects

Only a finite number of moments can be extracted fromany basis polynomial, because the lowering operators an-

level, and in many cases we do not expect differences from theMoran process be preserved under aggregation.

nihilate the right ground state, represented as

0 =∂

∂zi1 ↔ ai |0) . (C1)

The conjugacy of this relation for the definition (21) ofthe left ground state is checked as

0 =

∫

dDz δ(z) zj ↔ (0| a†j . (C2)

The basis polynomials count the number of agents ofeach type without distinguishing them, and are associ-ated with right number states,

(

D∏

m=1

znmm

)

· 1 ↔D∏

m=1

(

a†m)nm |0) = |n) , (C3)

on which the raising and lowering operators act by

a†j |n) = |n+ 1j) , (C4)

ai |n) = ni |n− 1i) . (C5)

23

The number states are eigenvalues of the number opera-

tor, which corresponds to the operator zi∂/∂zi that ex-tracts the index ni from complex functions,

a†iai |n) = ni |n) . (C6)

The state corresponding to a generating function is there-fore the sum of number states weighted by their proba-bilities,

Ψ(z) ↔∑

n

ρn |n) ≡ |Ψ) , (C7)

defining Eq. (22) in the text.A unique inner product evaluates to unity on all num-

ber states, which is known as the Glauber norm, or alsothe inner product with the Glauber state. Its representa-tion in analytic functions and abstract vectors are

1 =

∫

dDz δ(z) e∑

m∂/∂zm

(

D∏

m=1

znmm

)

· 1

↔ (0| e∑

mam

D∏

m=1

(

a†m)nm |0) .

= (0| e∑

mam |n) . (C8)

From the expansion of the exponential eam in Eq. (C8),we can extract the definition of the normalized conjugatenumber state,

(n| ≡ (0|D∏

m=1

anmm

nm!, (C9)

From the commutation (17) and annihilation (C2) rela-tions it follows that number states are orthogonal andnormalized,

(n′ | n) = δDn′n. (C10)

2. Coherent states: construction, inner product,

and action of the Liouville operator

The coherent states are defined in Eq. (27) as eigen-states of the lowering operator. They are constructedfrom the number states as

|φ) = e−(φ†φ)/2ea

†·φ |0)

= e−(φ†φ)/2

∞∑

M=0

(

a† · φ)M

M !|0)

=D∏

m=1

(

e−(φ†mφm)/2

∞∑

nm=0

φnmm

nm!a†m

nm

)

|0)

=∑

n

D∏

m=1

(

e−(φ†mφm)/2φ

nmm

nm!

)

|n) . (C11)

Because all a†m commute, the exponential may be ex-panded either as a simple sum on M or a multinomialover {nm}.The conjugate coherent states, indexed by Hermitian

conjugate vector φ†, and satisfying the conjugate eigen-value relation (28), are then

(

φ†∣

∣ = (0| eφ†·ae−(φ†φ)/2

= (0|∞∑

M=0

(

φ† · a)M

M !e−(φ

†φ)/2

=∑

n

(n|D∏

m=1

(

(

φ†m

)nme−(φ

†mφm)/2

)

. (C12)

(See Ref. [5] for the interpretation of the left coherentstates as moment-sampling operators, and their role inpropagating information from final conditioning contextsbackward in time to extract most-likely trajectories con-sistent with those contexts.) In particular, the valueφ† = 1 recovers the Glauber state

(0| e∑

mam ,

up to an overall normalization.To compute the quadrature (26) of the Liouville equa-

tion we need the inner product between two coherentstates at adjacent times separated by an interval dt.This inner product may be evaluated from the defini-tions (C11,C12) as(

φ†t+dt

∣

∣

∣φt

)

= eφ†

t+dt·φt−(φ†

t ·φt+φ†

t+dt·φt+dt)/2. (C13)

The coherent states are readily checked to furnish arepresentation of the identity equivalent to that providedby the number operators,

∫

dDφ†dDφ

πD|φ)(

φ†∣

∣ =∑

n

|n) (n| = I. (C14)

The cancellation of phase integrals in Eq. (C14) collapsestwo summations into one, and the evaluation of the re-sulting Γ-functions as integrals in φ† ·φ cancels factorials,leaving only the number states.To evaluate the action of the Liouville operator, we

begin with the expectation of the number operator,

(

φ†∣

∣

D∑

i=1

a†iai |φ) =

D∑

i=1

φ†iφ

i ≡ φ† · φ. (C15)

Because L – like the number operator – is normal-ordered, the value from its insertion between coherentstates is elementary to compute, as(

φ†t+dt

∣

∣

∣L(

a†, a)

∣

∣

∣φt

)

= L(

φ†t+dt, φt

)(

φ†t+dt

∣

∣

∣φt

)

.

(C16)These computations may be assembled to convert the

Doi operator expression [24, 25] for the generating func-tion into its coherent-state equivalent, introduced by

24

Peliti [26, 27]. The combination of the factorization (30)of the evolution operator with the insertion of coherent-state representations of unity (29), the choice (31) for

the initial state, and inner product with the Glaubernorm (C8), gives the integral representation for the time-evolved generating function

(0| e∑

mame−L(a†,a)T e(a

†−1)·n |0) =∫ T

0

Dφ†Dφ e1·φT e−∫

T

0dt{φ†·∂tφ+L(φ†,φ)}−φ†

0·φ0e(φ†

0−1)·n. (C17)

The left-most factor of 1·φT in Eq. (C17) results from theinner product of the Glauber state with the left-most co-herent state. It may be shifted to the right by absorbinga total derivative into the main integral in Eq. (C17),

e1·φT = e

∫

T

0∂t(1·φt)e1·φ0 (C18)

leaving Eq. (32) in the text.

3. The abstract inner product in place of parallel

complex integrals

The most important reason to pass from analytic func-tions to an abstract representation of the Doi Hilbertspace is that it simplifies the inner product. Since we willneed to compute this inner product an infinite number of

times in the functional integral, such simplifications arevaluable. Also, the abstract Hilbert space captures prop-erties of the formal power series representation of gen-erating functions [4], which remain fundamental whetheror not the power series converge to analytic functions.

The definition (21) of the left ground state (0| as anintegral, beyond being cumbersome to write, requires aseparate variable of integration for each inner product tobe taken. To see how the multiplicity of integration vari-ables is simplified by the formal inner product, we mayevaluate the coherent-state representation of unity (C14)as it would appear directly in terms of differential opera-tors. Expanding the exponentials in the coherent states,and using the phase integral to eliminate all but equal-power terms, the identity acting on inner products be-comes

∫

dDφ†dDφ

πD|φ)(

φ†∣

∣ =

∫

dDφ†dDφ

πDe−(φ

†·φ)ezt+dt·φ

∫

dDzt δ(zt) eφ†·∂/∂zt

=

∫

dDzt δ(zt) ezt+dt·∂/∂zt

= zt 7→ zt+dt, (C19)

that is, a map that removes the former integration vari-able and substitutes the latter.

Appendix D: Taylor’s series expansion of the

Liouville operators

1. The derivative expansion in coherent-state fields

The stationary-point conditions (44,45) are defined byvanishing of the first variational derivative of S. Thecomplete algebraic forms for this derivative at general(

φ†, φ)

are

dφi

dt= − ∂L

∂φ†i

=

D∑

j=1

(µijφj − µjiφi) +

D∑

j=1

φiφj

(

φ†iλij − φ†

jλji

)

− φi

D∑

j=1

(

φ†j − φ†

i

)

φjλij +1

N

D∑

j,l=1

(

φ†l − φ†

j

)

φlaSijφ

†jφj

(D1)

25

dφ†i

dt=

∂L∂φi

=

D∑

j=1

(

φ†i − φ†

j

)

µji +

D∑

j=1

(

φ†i − φ†

j

)

φ†jφjλji + φ†

i

D∑

j=1

(

φ†j − φ†

i

)

φjλij +1

N

D∑

j,l=1

(

φ†l − φ†

j

)

φlaSijφ

†jφj

.

(D2)

Because only difference terms(

φ†i − φ†

j

)

appear in Eq. (D2), it is immediate that φ† ≡ 1 is always a consistent

solution. The terms grouped in parentheses are those which cancel in the stationary-point equation for the total

number, d(

φ†iφi

)

/dt, which corresponds to dνi/dt.

The second partial with respect to φ, which is the quantity averaged to compute the stochastic effective action (78),and whose variation is averaged to compute the corrections to the mean-field equations of motion (77), is

∂2L∂φm∂φn

=(

φ†m − φ†

n

)

[

(

φ†n

anm2

− φ†m

amn

2

)

+1

N

D∑

k=1

(

φ†na

Snk − φ†

maSmk

)

(

φ†kφk

)

]

+1

N

D∑

k=1

(

φ†kφk

) [(

φ†m − φ†

k

)

aSknφ†n +

(

φ†n − φ†

k

)

aSkmφ†m

]

− φ†mφ†

naSmn

1

N

D∑

k=1

(

φ†m + φ†

n − 2φ†k

)

φk. (D3)

Because we will not pursue fluctuation corrections to the response and correlation functions (57), and will considerthese only about classical backgrounds, we evaluate the mixed partial that defines the diffusion kernel only in thebackground φ† ≡ 1,

∂2L∂φ†

i∂φm

= −µim − δim

D∑

j=1

aijφj −1

N

D∑

j,k=1

φjajkφk

+ φi

1

N

D∑

j=1

(

2aSmj − aSij)

φj − aim

. (D4)

The trace of this kernel, which gives the sum of eigenvalues as a function of the background, evaluates to

D∑

i=1

∂2L∂φ†

i∂φi

= −D∑

i=1

µim −D∑

i,j=1

(aij + δijaii)φj +D + 1

N

D∑

i,j=1

φiaijφj . (D5)

The Hessian which is the noise source for coherent-state field fluctuations, appearing in equations (65 - 67) and alsoin the equivalent Langevin source term (F7), is given (also at φ† ≡ 1) by

∂2L∂φ†

i∂φ†l

= −δilφl

D∑

j=1

(2alj + ajl)φj + φiφl

2

N

D∑

j=1

(

aSij + aSlj)

φj − aSil

. (D6)

Finally, the third partial responsible for the leading corrections (77) to the stationary-point conditions is

∂3L∂φ†

i∂φm∂φn

= δim

1

N

D∑

j=1

(

2aSnj − aSmj

)

φj − amn

+ δin

1

N

D∑

j=1

(

2aSmj − aSnj)

φj − anm

+φi

N

(

2aSmn − aSin − aSim)

.

(D7)

We evaluate this quantity for the examples in the text atφ† ≡ 1, because all expectations except those of 〈φ′φ′〉 atequal times will cancel [49].

2. The derivative expansion in action-angle

coordinates

The stationary-point condition (46) for the numberfield ν, resulting from first variation of the normalizedaction S, may be expressed variously in terms of the

26

population-corrected fitness λij of Eq. (40) or the averageflow rates ρij of Eq. (41) as

dνidτ

= − ∂L∂ηi

=

D∑

j=1

eηi−ηj (µij + νiλij) νj − eηj−ηi (µji + νjλji) νi

=

D∑

j=1

eηi−ηjρij − eηj−ηiρji. (D8)

In particular, the last line, at η ≡ 0 (or indeed η =any constant) yields the condition of detailed balance

ρlj = ρjl between all pairs jl, expressed in the text asthe fixed point condition of vanishing of rjl in Eq. (48).

The corresponding Eq. (47) for η, which we do not

need to use except to implement steepest descents of Sin App. E 2, evaluates to

dηidτ

=∂L∂νi

=

D∑

j=1

(

1− eηj−ηi)

(µji + νjλji) +(

1− eηi−ηj)

λijνj +

D∑

j,l=1

aSijνj(

1− eηj−ηl)

νl. (D9)

Again we verify that η ≡ 0 is always a consistent solu-tion. All remaining quantities will be evaluated at η ≡ 0,because they will only be applied here to classical sta-tionary trajectories.The noise source and inverse mass matrix (51) has

a considerably simpler form in action-angle coordinatesthan its counterpart (D6) in coherent-state fields,

∂2L∂ηi∂ηj

= (ρij + ρji)− δij

D∑

k=1

(ρik + ρki) . (D10)

The diffusion kernel is also simple, and may be expressedin a manner that emphasizes the antisymmetry of its ar-gument in i and j, corresponding to the antisymmetry ofthe deterministic replicator in Eq. (A3),

∂2L∂ηi∂νm

= δim

D∑

j=1

[µjm + νj (λjm − λmj)]

− [µim + νi (λim − λmi)]

+ νi

D∑

j=1

νj(

aSjm − aSim)

(D11)

Eq. (D11) evaluates to the same quantity as its coherent-state counterpart (D4), up to an overall factor of N re-flecting the definitions of rescaled number fields and time.Finally, the third partial derivative that appears in

fluctuation corrections to the stationary-point equations,counterpart to Eq. (77) in the text, but for the numberfield ν, is

∂3L∂ηi∂νm∂νn

= (λnm − λmn) (δmi − δni)

+

D∑

j=1

νj[

δmi

(

aSjn − aSmn

)

+ δni(

aSjm − aSnm)]

− νi[

aSin + aSim − 2aSnm]

. (D12)

This is the form used numerically to evaluate the cor-rections to the stationary point in going from the naivemean-field limit to the self-consistent rest points inFig. 20 of the companion paper.

3. Relations between field and action-angle

variables

Only the diffusion kernels and noise source appear inthe evaluation of fluctuations in Sec. III B 1, and App. D 2has shown that the diffusion kernels about classical back-grounds are the same in coherent-state and action-anglevariables. Therefore, although the correlation function〈νν〉 is formally fourth-order in φ† and φ, it differs onlyby the addition of noise-source terms. The exact relationbetween the Hessian forms (D6) and (D10) is

− ∂2L∂ηi∂ηj

= − 1

N2

∂2L∂φ†

i∂φ†j

− (µijνj + µjiνi)

+ νiνj

D∑

k=1

(

aSik + aSjk − 2aSij)

νk.

(D13)

Usually the correlation function for coherent-state fieldsis used as a proxy for fluctuations in the numberfield [48, 49, 53]. The evaluations in Eq. (F37) belowshow that this difference, near a critical point, is thesame as the O

(

R2)

correction in passing from the uni-form background to the small-radius limit cycle, and isO(1/N).

27

Appendix E: Hamiltonians and instanton methods

for systems with multiple equilibria

The computation of instanton trajectories and proba-bilities for irreversible stochastic processes has both im-portant similarities, and important differences, to thecorresponding computation for barrier penetration inequilibrium thermal or quantum systems. These concernthe role of saddle points in relation to stable rest points,and the asymmetry of probabilities for relaxation and es-cape. They also lead to a “kinematic” description of thestochastic process which is different from its canonicaldescription. It is the kinematic description that providesthe most direct comparison of potentials and symmetriesto the equilibrium problem.We consider first the structure of the dynamical system

defined by the Liouville operator, and then the efficientcomputation of trajectories and probabilities for escape.

1. Forms using a quadratic approximation of the

Liouville operator

We begin with the structure of the Liouville operatorderived in Sec. IIIA 2 in the text. We will suppose thatthe quadratic expansion in η − η0 in the text can betreated as the exact form of an approximation to theLiouville operator, so that fluctuations in η may alwaysbe removed by Gaussian integration.For such quadratic forms, it is possible to remove the

explicit index notation and use a condensed inner prod-uct notation for vectors and matrices. Thus we will letηTm−1η stand for

∑Di,j=1 ηim

−1ij ηj , η · ν stand for ηT ν,

and we will use overdot to denote d/dτ . With these con-

ventions, the starting form for the action S in Eq. (42)is written

S =

∫

dτ(

η · ν + L)

. (E1)

Treating the second-order expansion of Eq (49) as an

exact form, the Liouville functional L in Eq. (E1) be-comes

L = −1

2(η − η0)

Tm−1 (η − η0) + V (ν) . (E2)

Although this form is quadratic in η, we permit it toretain arbitrary dependence on ν. Within the samequadratic expansion, ensuring L(η ≡ 0, ν) = 0 requires

V (ν) =1

2ηT0 m

−1η0. (E3)

η0 and m−1 are defined by Equations (50) and (51) inthe main text. The potential V (ν) is bounded below byzero and has local minima at rest points given by η0 ≡ 0.This is the “trapping” potential that corresponds to theordinary kinematic potential in equilibrium thermody-namics; it is responsible for the non-negative spectrumof the Liouville operator, and convergence of the densityρn to a stationary distribution.We expect, from equilibrium treatment of instantons,

that escapes will be described by the heteroclinic orbitsof a configuration ν moving on potential −V (ν). To see

how this description arises, we expand S as a bilinearform using three different groupings of terms:

S =

∫

dτ

[

η · ν − 1

2(η − η0)

Tm−1 (η − η0) + V (ν)

]

=

∫

dτ

[

−1

2(η − η0 −m⊥ν)

Tm−1 (η − η0 −m⊥ν) +

1

2νTm⊥ν + V (ν) + η0 · ν

]

=

∫

dτ1

2

[

−(η − η0 −m⊥ν)Tm−1 (η − η0 −m⊥ν) +

(

ν + η0m−1)T

m⊥

(

ν +m−1η0)

]

. (E4)

In order to invert the matrix m−1 in Eq. (E4), it hasfirst been necessary to remove the zero-eigenvalue asso-ciated with shifts of all ηi by a constant, mentioned inSec. IIIA 3. The projection of m−1 onto the (D − 1)-

dimensional hyperplane∑D

i=1 ηi = 0 is positive-definite,and m⊥ is its matrix inverse.

We now verify a number of properties mentioned in thetext, from these forms for S. The first line in Eq. (E4) isthe standard Lagrange-Hamilton definition of the actionwith canonical momentum η, and a Hamiltonian havingnon-canonical dependence on η and potential −V (ν).

It is clear that integrating η−η0−m⊥ν along a real con-

tour in the second line of Eq. (E4) produces a divergence,and that η − η0 −m⊥ν must be rotated to an imaginarycontour to define the Gaussian integral by steepest de-scent. After such an integration, the remaining terms onthe second line give the Lagrangian for potential −V (ν).This Lagrangian corresponds to the so-called Euclidean

action in the equilibrium method of instantons [33]. Foran equilibrium barrier-crossing problem, η0 would equalzero. The real offset it creates for η, and the final ex-pression η0 · ν that it adds in the second line of Eq. (E4)will lead to different probabilities for relaxation and es-cape trajectories, which are nearly time-reverses of oneanother, as we will see below.

28

The third line in Eq. (E4), after integration overη− η0−m⊥ν, gives the stochastic action first derived byOnsager and Machlup [65]. It is the small-fluctuation ap-proximation to the rate function for the large-deviationsprinciple for histories [5].The stationary-path condition for η from the second

line of Eq. (E4),

m⊥ν = η − η0, (E5)

verifies that η− η0 is the kinematic momentum, and m⊥

the mass matrix relating velocities ν to momenta. Inparticular, the classical stationary-path trajectories (forwhich η ≡ 0) express the mean-field replicator dynamicthrough the relation

m⊥νCl = −η0. (E6)

Thus we recognize −m−1η0 as the tangent vector to theclassical flowfield at each ν.To see that the Markovian term η0 · ν in the second

line of Eq. (E4) does not qualitatively alter the kinematicinterpretation of barrier crossing in the potential −V (ν),we may evaluate the stationary-point condition for ν fromthe third line of Eq. (E4) to obtain

d

dτ(m⊥ν) =

∂

∂νV (ν) + η0

∂m−1

∂ν

(

m⊥ν +η02

)

. (E7)

The leading term in Eq. (E7) is the acceleration dueto the gradient of −V (ν), and the second term is smallwhenever m−1 is not a strong function of ν. An exactevaluation of m−1 is plotted in Fig. 3, showing weak tomoderate ν-dependence and anisotropy. The correctionproportional to ∂m−1/∂ν contains a viscosity linear in νas well as a force term not captured by the potential, dueto the momentum offset η0.

We note that the combination of the conservation lawL = 0, the equation of motion (E6), and the quadratic ex-

pansion (E2) for L, implies that the velocity fields alongall stationary paths must satisfy

νm⊥ν = (η − η0)m−1 (η − η0) = η0m

−1η0. (E8)

The solutions to this equation for ν are (D − 2)-spheres,which pinch off at the rest points where η0 = 0.

2. A steepest-descent algorithm to extract

non-classical stationary points

Local minima of the action can be found directly witha method of steepest descent, which we demonstratein this subsection. The key observation is that locallyvalid momentum values, which are needed to evaluatethe stationary-point equations (46,47), can be uniquelyinferred for all points along any smooth trajectory inν, from the equation of motion (E6) and the condition

L = 0. If we know the endpoints of a trajectory, we may

L R

M

FIG. 3. In-plane projection of the fluctuation kernel ∂2L/∂η2

for the parameters that produce the flowfield shown in Fig. 8for the coordination game of the companion paper. The ma-jor and minor principle axes of fluctuation (variances) for η,at a regular sample of ν values, are plotted as crosses centeredat ν (and divided by 20 from actual magnitudes to aid view-ing). The fluctuation spectrum is only weakly ν-dependentand only moderately anisotropic at any ν.

vary ν-paths pinned at those endpoints, minimizing thedifference between the path-derivative for the stationary-point momentum, and the derivative implied by the equa-tion (47).Consider a general variation about an arbitrary, τ -

dependent, pair of functions η and ν. Ignoring totalderivatives, the resulting variation in S will be

δS = N

∫

dτ

[(

∂L∂ν

− η

)

· δν + δη ·(

ν +∂L∂η

)]

.

(E9)Now define a trajectory as a smooth curve of points

{ν}, and let dν at any point be a vector tangent to thecurve at that point. The normalization of the tangentvector may be arbitrary, but its orientation is not. Foreach point ν in the trajectory, choose a correspondingvalue for η, denoted η (ν), such that

L(

ν, η (ν))

= 0 , and

dν = −dτ∂L∂η

∣

∣

∣

∣

∣

ν,η(ν)

, (E10)

for some positive constant dτ . As long as the surfaceL = 0 is convex, η (ν) and dτ will be uniquely specified.dτ in Eq. (E10) is the time interval appropriate to make

dν

dτ= − ∂L

∂η

∣

∣

∣

∣

∣

ν,η(ν)

(E11)

29

a velocity field. The integral of Eq. (E11), ν (τ), is awell-defined time-dependent trajectory, satisfying the νstationary-path condition (D8). Similarly, for the func-tion η (ν), a time derivative dη (ν) /dτ along the trajec-tory is implied as well. In general it will not satisfy the ηequation of motion (D9), since the foregoing constructioncan be carried out for any smooth curve in ν-space.

Now we reconsider the variation (E9) of S, amongthe restricted class of functions that admit general δνand δη, but which start from paths satisfying equa-tions (E10,E11). The second term in the integral vanishes

by Eq. (E11). The remaining gradient ∂L/∂ν ≡ ηEOM isthe velocity of the η-field along a true stationary point in-stantaneously tangent to the trajectory {ν}, by Eq. (D9).The variation of S then becomes

δS = N

∫

dτ

[(

ηEOM − dη (ν)

dτ

)

· δν]

. (E12)

Note that, starting from(

ν, η (ν))

, the variationsηEOMdτ and dη (ν) (along with dν), both leave their

paths in the surface L = 0 at ν + dν. Therefore the dif-ference ηEOMdτ −dη (ν) must satisfy

(

ηEOMdτ −dη (ν))

·∂L/∂η = 0 at ν + dν, or equivalently, from Eq. (E11)

(

ηEOM − dη (ν)

dτ

)

· dνdτ

= 0. (E13)

In other words, the functional variation of the actionabout this class of curves yields a vector field instan-taneously perpendicular to the trajectory at each point.Thus the gradient in Eq. (E12) may be used in a New-ton’s method to converge on the non-classical stationarypaths.Fig. 4 shows a portion of the ν-simplex from the flow-

field Figure 8 for the coordination game in the compan-ion paper, with the classical diffusion paths in the back-ground. Overlaid on these are segments of actual solu-tions to the stationary-path equations of motion, startingfrom arcs that begin in the stable rest point and termi-nate in the saddle point. The arcs are chosen to lie aboveand below the contour of slowest approach in the classi-cal diffusion solution, which serves as a proxy for thesaddle path of the potential −V (ν). As expected fromthe kinematic approximation (E7), these trajectories di-verge from the saddle path between the two rest points.The actual escape trajectory, obtained by the method ofsteepest descent described here, is shown in Fig. 1 in thetext.To estimate the actions associated with reverse-

direction paths, which give their probability of occur-rence, we use the condition L = 0, both explicitly inEq. (E1), and through equations (E2,E5) to remove τ asan integration variable, leaving

νTm⊥ν ≈√

dνTm⊥dν

dτ

√

2V (ν). (E14)

Then S of Eq. (E1) may be approximated

S = N

∫

dτ η · ν

0 0.05 0.1 0.15 0.2 0.25−0.2

−0.1

0

0.1

(nR - nL) / sqrt(2)

(2 n

M -

nR

- n

L)

/ sq

rt(6

)

FIG. 4. Solutions to the non-classical stationary point con-ditions (D8,D9) for the coordination game with the payoffsof Eq. (26) from the companion paper. Symmetry is weaklybroken (a = 1, a = 0.5, N = 10). Flowfields of the classicalsolution are shown as light lines in the background for ref-erence, and the path of slowest diffusion is indicated by theconfluence of these lines. Two ν trajectories are tested (heavyblack lines), one above and one below the classical directionof slowest flow, which approximates the saddle contour of thepotential −V (ν). Exact solutions initially tangent to thesetrajectories (the comb of diverging heavy lines) curve awayfrom the saddle of −V (ν), as expected from the potentialcontours shown in Fig. 10 of the companion paper.

= N

∫

dτ(

(η − η0) · ν + η0 · ν)

≈ N

∫

dτ(

νTm⊥ν − νTClm⊥ν)

≈ N

∫ √

dνTm⊥dν√

2V (ν)

(

1− νTClm⊥ν

νTm⊥ν

)

,

(E15)

in which the approximations correspond to the second-order expansion of L in η, used in the previous subsection.√

dνTm⊥dν is a line element along the trajectory ν,and by Eq. (E8), the term in the fraction on the fourthline of Eq. (E15) is a direction cosine between the tan-gent fields ν and νCl in the inner product defined by m⊥.For the classical stationary paths, this fraction equalsunity by definition, giving the required S ≡ 0. To theextent that the non-classical solutions approximately re-

verse νCl, the direction cosine is ≈ −1, and the actionevaluates to the positive quantity

S ≈ 2N

∫ √

dνTm⊥dν√

2V (ν), (E16)

twice the WKB value that would have been expected forequilibrium barrier crossing in the same potential. We see

30

that the last term η0 · ν in the second line of Eq. (E15)has only weakly affected the stationary trajectories, butprovides the offset to the value S(0) that permits free clas-sical diffusion, doubly penalizing reverse-direction diffu-sion responsible for escape from the stable fixed points.

Appendix F: Calculating Gaussian-order response

and correlation functions

This appendix provides calculations of all Gaussian-order fluctuation effects used in the main text. Sec. F 1reviews the connection between the field-integral gen-erating function and the Langevin equation. Sec. F 2provides specific forms for the correlation function offluctuations about the uniform background in the Rock-Paper-Scissors game in coherent-state fields, emphasizingthe divergence on the approach to instability of the uni-

form population background. Sec. F 3 provides the cor-responding derivation, this time using action-angle vari-ables and polar coordinates in the ν simplex, for thelimit cycle of ordered population states. This sectionprovides both the radial-fluctuation divergence in the or-dered state, and the accumulation rate for phase noisearound the limit cycle.

1. Auxiliary fields and the Langevin equation

The tri-diagonal form (56) for S(2) in the main textresults from splitting the mixed-partial derivatives in φ†

and φ into two symmetric terms, whose inverses are theadvanced and retarded response functions. If we do notsplit these terms, the same action, up to a total deriva-tive, may be written

S(2) =

∫

dt

D∑

i,j=1

{

φi

(

δijd

dt+

∂2L∂φ†

i∂φj

)

φ′j +

1

2φi

∂2L∂φ†

i∂φ†j

φj

}

. (F1)

Its counterpart in action-angle coordinates reads

S(2) = N

∫

dτ∑

i,j

{

η′i

(

δijd

dτ+

∂2L∂ηi∂νj

)

ν′j +1

2η′i

∂2L∂ηi∂ηj

η′j

}

. (F2)

The following construction may be carried out in eithervariables, but the coherent-state representation will beused as an example.

The Hessian for either φ or η′, in equations (F1,F2), isnegative-semidefinite, requiring the rotation of the mo-mentum variable to an imaginary integration contouras we saw in the expansion over stationary points inApp. E 1. The direct use of the momentum variable asan “imaginary Langevin field” is sometimes seen, but anolder approach is to cancel the unstable Hessian term

with an offset produced by an auxiliary field, which wouldbecome the Langevin field. The procedure, known as theHubbard-Stratonovich transformation [68] is to introducea representation of unity as the Gaussian integral

1 = det

(

∂2L∂φ2

)−1/2∫

Dζe−SAux

, (F3)

in which the “auxiliary field action” is the non-dynamical quadratic form (meaning that it involves noτ -derivatives)

SAux = −1

2

∫

dt

D∑

i,j=1

(

ζi +

D∑

m=1

φm

(

∂2L∂φ2

)

mi

)(

∂2L∂φ2

)−1

ij

ζj +

D∑

n=1

(

∂2L∂φ2

)

jn

φn

. (F4)

(Eq. (F3) is normalized with the square root of a functional determinant because a single field, ζ, is introduced to

correspond to φ, rather than a complex-valued pair(

ζ†, ζ)

.)When the expression (F3) is inserted into the generating functional (32), the resulting integral remains Gaussian,

with a kernel in which the original Hessian terms cancel, to be replaced with a convergent Hessian in the fields ζ,

S(2) + SAux =

∫

dt

D∑

i=1

φi

D∑

j=1

(

δijd

dt+

∂2L∂φ†

i∂φj

)

φ′j − ζi

− 1

2

D∑

i,j=1

ζi

(

∂2L∂φ2

)−1

ij

ζj

. (F5)

31

If φ is now integrated along an imaginary contour, it produces a functional δ-function,

∫

Dφ exp

∫

dt

D∑

i=1

φi

D∑

j=1

(

δijd

dt+

∂2L∂φ†

i∂φj

)

φ′j − ζi

=

D∏

i=1

δ

D∑

j=1

(

δijd

dt+

∂2L∂φ†

i∂φj

)

φ′j − ζi

, (F6)

which restricts field variations to a hypersurface withinthe remaining integral over φ′ and ζ.The argument of the δ-functional,

D∑

j=1

(

δijd

dt+

∂2L∂φ†

i∂φj

)

φ′j − ζi

is called the Langevin equation from stochastic differen-tial equation theory. It expresses φ′, through the inverseof the retarded Green’s function, in terms of the Langevinfield ζ, which has the time-local correlation function

⟨

ζitζjt′⟩

= −(

∂2L∂φ2

)

ij

δ(t− t′) . (F7)

The form of the Hessian ∂2L/∂φ2 is given in Eq. (D6)as a function of the original probabilities in the masterequation. Thus, from the Gaussian approximation to thefunctional integral, we may derive the Langevin equationfor an arbitrary stochastic process, with the correct noisestructure and the correct damping.

2. Fluctuations about the uniform background in

coherent-state fields

Here we provide the algebraic forms needed to com-pute the fluctuation quantities in Sec. III B 1, for the RPSgame defined in Sec. IV E of the companion paper, whenit is expanded about the uniform-population background.General algebraic forms for the coherent-state expansionare given in App. D 1, and the payoff matrix to whichthey are applied is that of Eq. (48) in the companion pa-per. Diffusion is taken to be isotropic, as in Eq. (8). Wewill keep the number of types D explicit, even thoughin the examples D = 3, to distinguish it from numericalfactors related to the order of derivatives, and to indicatethe scaling of magnitudes with D.The uniform background satisfies φm = N/D for each

m. In this background, the diffusion kernel (D4) evalu-ates to

∂2L∂φ†

i∂φm

= N

(

D

N+

b− a

2D

)(

δim − 1

D

)

− aAim. (F8)

Continuity in the sum of the eigenvalues about more gen-eral backgrounds will also be of interest. From Eq. (D5)for the general trace, and the definition

r2 ≡ n2R + n2

P + n2S

N2− 1

D, (F9)

of r2 on the population simplex,

D∑

i=1

∂2L∂φ†

i∂φi

= (D − 1)

[

D +N

(

b− a

2D

)]

+ (D + 1)N

(

a− b

2

)

r2

→ 2

∣

∣

∣

∣

D +N

(

b− a

2D

)∣

∣

∣

∣

, (F10)

where in the last line we have used the leading non-oscillatory expansion for r2 in the phase with a limitcycle, corresponding to the estimate (F21) in App. F 3.Returning to the uniform background, the noise source

corresponding to Eq. (F8), which is the Hessian (D6),becomes

− ∂2L∂φ†

i∂φ†j

= N2

[

3

D

(

a+b− a

2

)

− 2 (b− a)

D2

](

δij −1

D

)

.

(F11)The matrix in both equations (which has the same

form as the mutation matrix up to a scale factor) is a pro-

jector into the transverse simplex∑D

m=1

(

φm + φ′m

)

=N . Using an expansion similar to that used for the pitch-fork bifurcation Eq. (B1) of App. B in the companionpaper, but expressed in a basis v and v∗ of eigenvectorsof the diffusion kernel (F8),

11

1

− 1

D

111

[

1 1 1]

= vv† + v∗vT . (F12)

In terms of the population frequencies,

v ≡ 1

2√3

2−1−1

+i

2

01

−1

, (F13)

and v∗ is its complex conjugate. The eigenvalue of v inthe diffusion equation is given by

∂2L∂φ†∂φ

v = N

[(

D

N+

b− a

2D

)

+ i(b+ a)

2√D

]

v, (F14)

and the eigenvalue for v∗ is the complex conjugate.About a uniform background both the eigenvalues and

eigenvectors are time-independent, so the time-orderedexponential integrals in the Green’s functions (60,61) re-duce to simple scalar exponents in (t− t′). These arereadily integrated in the formula (66) for Mt, in whichwe can take t′′ → −∞ because all noise is damped. The

32

result is simply to divide the noise source (F11) by thesum of the eigenvalue in Eq. (F14) and its complex con-jugate (one factor from DR and one from DA), yieldingEq. (69) for 〈φ′φ′〉t.

3. Rotating backgrounds, polar coordinates, and

accumulating Brownian noise

We now carry out the parallel derivation, above thecritical point where the stationary solutions for RPS con-verge on a limit cycle of nonzero radius. The treatmenthere, which makes use of a rotation to a Frenet coordi-nate system to simplify the form of the action, generalizesthe Floquet analysis in App. C of the companion paper.The calculations are performed in action-angle vari-

ables, for convenience in converting to polar coordinatesin the number field ν. Again, explicit algebraic formsare provided in App. D 2. We note, as an aside, thatthe correlation function in action-angle variables differsfrom that in coherent state variables only by correctionsof order 1/N relative to the leading magnitudes. These

arise, as they must, from expectations⟨

φφ′⟩2

, which the

action-angle correlator includes but the coherent-statecorrelator does not. The closed-form relation betweenthe noise sources for these two forms, for general payoffmatrices, is provided in App. D 3.We begin with the action form for S(2) given in

Eq. (F2) with fluctuating fields (η′, ν′). In the limit-

cycle background, the diffusion kernel ∂2L/∂η∂ν is time-dependent and does not generally have any zero entries,which makes direct evaluation of time-ordered exponen-

tials complicated. Yet we know from Goldstone’s theo-rem that the time-derivative of the limit cycle must be azero mode, and we know by symmetry that near the crit-ical point it is approximately a circle. To exploit thesefacts, we consider the effect of general (time-dependent)coordinate transformations on simplifying the form of thediffusion kernel. When coordinates can be chosen to ap-proximately trace the limit cycle, the zero eigenvalue maybe made explicit, and the resulting lower-triangular formfor the kernel makes evaluation of 2 × 2 time-orderedexponentials elementary. This construction is a simpleversion of the transformation to a Frenet frame used inRef. [30].

Consider, therefore, the effect of an invertible transfor-mation V from the rectilinear basis

(

ηT , ν)

to variables

ηTV and V −1ν. We may always choose V so that

dV

dτ+

∂2L∂η∂ν

V = V Λ, (F15)

for some Λ which is lower triangular, by letting the sec-ond column of V be the time derivative of the limit-cycletrajectory. For now, however, assume only that V andΛ are smooth, i.e., V ΛV −1 has a continuous first deriva-tive so that in the continuum limit the retarded responsefunction becomes

〈ν′τη′τ ′〉 = θ(τ − τ ′)1

NVτT

(

e−∫

τ

τ′duΛu

)

V −1τ ′ , (F16)

Under the same basis transformation, the correlationfunction (67) at equal times becomes

〈ν′τν′τ 〉 = VτT(

e−∫

τ

τ′duΛu

)

V −1τ ′ 〈ν′τ ′ν′τ ′〉V −1

τ ′

†T −1

(

e−∫

τ

τ′duΛ∗

u

)

V †τ

− 1

NVτ

∫ τ

τ ′

dzT(

e−∫

τ

zduΛu

)

V −1z

∂2L∂η2

∣

∣

∣

∣

∣

z

V −1z

†T −1

(

e−∫

τ

zduΛ∗

u

)

V †τ (F17)

We now construct V through a series of successiveapproximations, beginning with the transformation tostatic polar coordinates, which can be performed directlyin the action, and for which the notation of a time-dependent V is not yet needed. Let the normalized radiusr introduced in Eq. (F9), and the angle θ in the simplex,be polar coordinates for ν, and perform a similar trans-formation on η as

[

ν1ν2

]

≡[

r cos θr sin θ

]

;

[

η1η2

]

≡[

h cosφh sinφ

]

. (F18)

The only combinations of these that actually appear inS(2) refer η to the direction of ν, not to the rectilinear

frame, so define radial and tangential components of η as

[

ηrηt

]

≡[

h cos (φ− θ)h sin (φ− θ)

]

. (F19)

For small r2, we will approximate the noise ker-nel (D10) by its value in the uniform background for sim-plicity, because it is not a strong function of ν near thecenter of the simplex, as was shown in Fig. 3. In polarcoordinates, the action to all orders in ν, and to secondorder in η, then takes a simple form corresponding to theexpansion (E2) used to study stationary points,

33

S ≈ N

∫

dτ[

ηr ηt]

[

drdτ + ∂L

∂ηr

r dθdτ + ∂L

∂ηt

]

+1

2

[

ηr ηt] ∂2L∂η2

[

ηrηt

]

, (F20)

in which ∂2L/∂η2 is computed with respect to the (ηr, ηt)basis.

The stationary point conditions are most easily decom-posed in terms of a set of scaling parameters which arefunctions of the payoffs. The reference scale for radius,R,18 will be the value suggested by Eq. (43) of the com-panion paper,

R2 ≡ 1

D− 2D

N (a− b). (F21)

The mean rate of advance of the phase θ is

ω =a+ b

2√3

(F22)

A normalized radius coordinate will be denoted

ρ ≡ r

R(F23)

only in this appendix (the notation is distinguished fromρn used for the original probability density). Three fur-ther combinations of the game parameters appear in theexact stationary point equations:

A1 ≡ R2 a− b

2ω

2A22 ≡ R2 +A2

1

tanα ≡ 1

3

a− b

2ω(F24)

The small expansion parameter will be R, in terms ofwhich A2 ∼ O

(

R)

, A1 ∼ O(

R2)

.Then the stationary point condition for ν in (ρ, θ) co-

ordinates, from the action (F20), becomes

dρ

ω dτ= − 1

Rω

∂L∂ηr

= ρ[

A2ρ sin (3θ + α) +A1

(

1− ρ2)]

ρ dθ

ω dτ= − 1

Rω

∂L∂ηt

= −ρ [1− η2ρ cos (3θ + α)] (F25)

The limit-cycle trajectories are not perfect circles, asshown in Fig. 14 of the companion paper, and the angleof inclination to a pure radial vector due to the variationof ρ is

∂L/∂ηr∂L/∂ηt

=ρ

ρθ= −A2ρ sin (3θ + α) +A1

(

1− ρ2)

1− η2ρ cos (3θ + α)

≡ tan ξ

≡ d log ρ

dθ. (F26)

The second-order action (F2) may then be written inpolar coordinates in terms of solutions to these equationsas

S(2) = N

∫

dτ[

ηr ηt]

d

dτ−[

∂ρ∂ρ

∂ρρ∂θ

ρ∂θ∂ρ

∂θ∂θ + ρ

ρ

]

[

r′

rθ′

]

+1

2

[

ηr ηt] ∂2L∂η2

[

ηrηt

]

. (F27)

Having removed most of the frame dependence of thelimit cycle with this transformation, we may now returnto the systematic approximation of V and Λ as a smallparameter expansion. Eq. (F15) becomes

d

dτ−[

∂ρ∂ρ

∂ρρ∂θ

ρ∂θ∂ρ

∂θ∂θ + ρ

ρ

]

V = V Λ (F28)

18 We require a slightly changed notation from App. C in the com-panion paper, because the notational requirements in this paperare more elaborate. Because overbar denotes a general ensembleaverage, the notation r includes dynamical ensemble averages forthe radius on the type simplex. Therefore, for the mean mag-nitude of the limit cycle radius as an expansion parameter, weintroduce the notation R.

If we take the time-derivative of the trajectory(

ρ, θ)

tobe the second column of V , and choose the first columnsimply to be orthogonal, then

V = − 1

ω

[

ρθ ρ

−ρ ρθ

]

=1

ω

√

ρ2 +(

ρθ)2[

cos ξ sin ξ− sin ξ cos ξ

]

, (F29)

with ξ defined in Eq. (F26). Solving Eq. (F28) then gives

Λ =

[

Λ(11) 0Λ(21) 0

]

, (F30)

in which

−Λ(11) = ω[

4A2ρ sin(

3θ + α− 2ξ)

− 2A1ρ2 cos (2ξ)

]

34

−Λ(21) = ω[

4A2ρ cos(

3θ + α− 2ξ)

− 2A1ρ2 sin (2ξ)

]

.

(F31)

Time-ordered exponentials of 2× 2 lower-triangular ma-trices reduce to elementary scalar integrals, as

T(

e−∫

τ2

τ1duΛu

)

=

e−∫

τ2

τ1duΛ(11)

u 0

−∫ τ2τ1

duΛ(21)u e

−∫

u

τ1du′Λ

(11)

u′ 1

.

(F32)V and Λ may be constructed in this way for any sta-

tionary solution. Polar coordinates lead to a simplesmall-parameter expansion if we solve Eq. (F26) for thelimit-cycle trajectory, to give

ρ(

θ)

= 1 +A2

3cos(

3θ + α)

+O(

R2)

(F33)

The leading order in V is then

V =

[

11

]

+O(

R)

(F34)

so the polar coordinates turn out to be adequate by them-selves.19 The time-ordered exponential (F32) similarlysimplifies to

T(

e−∫

τ2

τ1duΛu

)

=

[

e−2A1ω(τ1−τ2) 00 1

]

+O(

R)

. (F35)

Now it remains only to compute the noise kernel. Eval-uation of Eq. (D10) about a uniform background gives

− ∂2L∂ηi∂ηj

= − 1

N2

L∂φ†

i∂φ†j

+

[

2

N+

b− a

D2

](

δij −1

D

)

=

[

3

D

(

a+b− a

2

)

+2

N+

a− b

D2

](

δij −1

D

)

(F36)

The leading constant correction in small R2 can readilybe included, leaving out oscillatory terms as they wereleft out of V in Eq. (F34),

− ∂2L∂ηi∂ηj

∣

∣

∣

∣

∣

R2 6=0

= − ∂2L∂ηi∂ηj

∣

∣

∣

∣

∣

R2=0

+(a− b)

2DR2

(

δij −1

D

)

= − 1

N2

L∂φ†

i∂φ†j

∣

∣

∣

∣

∣

φi=N/D

+

[

1

N+

b− a

2D2

](

δij −1

D

)

=

[

3

D

(

a+b− a

2

)

+1

N− 3 (b− a)

2D2

](

δij −1

D

)

,

(F37)but this result differs from Eq. (F36) only by terms oforder 1/N near the critical point. Now using D = 3for the particular RPS example, the noise kernel (F37)reduces to the simple expression

− ∂2L∂ηi∂ηj

=

[

a− a− b

3+

1

N

](

δij −1

D

)

. (F38)

Incorporating the source (F38) and the time-orderedexponentials (F35) in the expression (F17) for the corre-lation function, and performing the elementary integralsover τ − τ ′, gives

⟨[

r′

rθ′

]

[

r′ rθ′]

⟩

τ

=

[

e−2A1ω(τ−τ ′)

1

]⟨[

r′

rθ′

]

[

r′ rθ′]

⟩

τ ′

[

e−2A1ω(τ−τ ′)

1

]

+

(

a− a− b

3+

1

N

)

[

1−e−4A1ω(τ−τ′)

4A1ωNτ−τ ′

N

]

+O(

R)

(F39)

Restoring the combination of aggregated constants totheir expression in terms of the parameters of the prob-lem,

4A1ωN =2N

D

(

a− b− 2D2

N

)

, (F40)

and using large (τ − τ ′) to ignore exponentials in thenonzero damping radial eigenvalue, we obtain Eq. (71)

19 This demonstration justifies the use of simple polar coordinatesto approximate the Frenet frame on the limit cycle in the maintext of the companion paper.

in the main text, used as Eq. (45) in the text of the com-panion paper.

Appendix G: Action-angle variables and the number

fluctuation for drift between two types

The transformation (37) to action-angle variables inthe text satisfies the criteria for a canonical transforma-tion with respect to the stationary-path solutions. It isnecessary to check, however, that it generates the cor-rect correlation functions from the functional integral as

35

well.20 When we have verified this, we may performGaussian integrals by inverting the kernel in the action,equally well in coherent-state or action-angle variables.This appendix verifies the absence of anomalies, by com-paring the Gaussian integral to an exact solution for theproblem of pure mutational drift between two types.Consider two types, for which the population structure

is given by a vector n ≡ (n1, n2). The master equationfor pure mutation becomes simply

dρndt

=ω

2

[

(n1 + 1) ρn+(1,−1) − n1ρn]

+ω

2

[

(n2 + 1) ρn+(−1,1) − n2ρn]

. (G1)

(The factor of ω has been added by hand as a reminderthat time has explicit dimensions, and also to check thatits associated characteristic timescale cancels from thefinal fluctuation strength.)Steady-state solutions to Eq. (G1) separate to inde-

pendent distributions along the diagonals n1 + n2 = Nfor each value of N . Any one such diagonal solution isexactly solvable, to the binomial

ρn =1

2N

(

Nn1

)

. (G2)

The leading N -dependence of the equal-time fluctuationcorrelation function in this distribution is

⟨

(

n1 − n2

2

)2⟩

=N

4, (G3)

displaying the N -linear scaling characteristic of eitherPoisson statistics or the central-limit theorem.Following the methods in the text, the master equa-

tion (G1) corresponds to the coherent-state Liouville op-erator

L =ω

2

(

φ†1 − φ†

2

)

(φ1 − φ2) , (G4)

and to the action in the full generating functional

S =

∫

dt{

φ†1∂tφ1 + φ†

2∂tφ2 +ω

2

(

φ†1 − φ†

2

)

(φ1 − φ2)}

.

(G5)This action, corresponding to pure diffusion, is exactly

quadratic – that is, it has no second-order term in φ†. ByNoether’s theorem it implies conservation of φ1+φ2 = N ,which we may make explicit with a basis rotation to write

S =1

2

∫

dt{(

φ†1 + φ†

2

)

∂t (φ1 + φ2)

+(

φ†1 − φ†

2

)

(∂t + ω) (φ1 − φ2)}

, (G6)

20 This is equivalent to checking that the action-angle transforma-tion generates no anomaly terms from the measure.

Now we encounter the subtlety remarked byKamenev [49] for free-field theories in coherent-state vari-ables: in the tri-diagonal continuum limit corresponding

to Eq. (56), no Hessian term for(

φ†1 − φ†

2

)2

appears to

define the functions M and DK that specify the corre-lation function for (φ1 − φ2). This lacuna is partly anotational artifact of the continuum limit, in the sensethat the fluctuation kernel can be correctly recovered bycareful evaluation of the more precise discrete functionalintegral [45, 49]. It is also physical, however, in the sensethat fluctuations are extensive when they are not regu-lated by terms larger than 1/N in the action.

The transformation to action-angle variables actuallysolves both of these problems, but only if it is performedin the original variables (n1, n2), so that it respects thediscrete permutation symmetry of the problem. Theaction-angle transformation will not generally respect thelarger basis-rotation symmetry displayed by the kineticterm in Eq. (G6). The form for S obtained by trans-forming Eq. (G5), and then rotating to a diagonal basis,is

S =1

2

∫

dt {(η1 + η2) ∂tN + ωN [1− cosh (η1 − η2)]

+ (η1 − η2) ∂t (n1 − n2) + ω (n1 − n2) sinh (η1 − η2)} .(G7)

In Eq. (G7) we have substituted N ≡ n1 + n2 for theconserved total number. The nonlinear cosh creates anexplicit second-order term in (η1 − η2), where there was

none in(

φ†1 − φ†

2

)

. Note that such a term would not have

been produced by introducing action-angle variables forthe rotated linear combinations of coherent-state fields inEq. (G6).

The tri-diagonal second-order expansion of the ac-tion (G7) now contains all three terms in the continuumlimit, which are needed to define the response and corre-lation functions. Following through the constructions ofSec. III B 1 recovers the result of the exact solution,

⟨

(

n1 − n2

2

)2⟩

=

∫ ∞

0

dte−2ωt

(

ωN

2

)

=N

4. (G8)

This is therefore the prescription for introducing action-angle variables that has been used throughout the text.

36

[1] K. G. Wilson and J. Kogut. The renormalization groupand the ε expansion. Phys. Rep., Phys. Lett., 12C:75–200, 1974.

[2] Joseph G. Polchinski. Renormalization group and effec-tive lagrangians. Nuclear Physics B, 231:269–295, 1984.

[3] M. I. Freidlin and A. D. Wentzell. Random perturbationsin dynamical systems. Springer, New York, second edi-tion, 1998.

[4] Herbert S. Wilf. Generatingfunctionology. A K Peters,Wellesley, MA, third edition, 2006.

[5] Eric Smith. Large-deviation principles, stochastic effec-tive actions, path entropies, and the structure and mean-ing of thermodynamic descriptions. Rep. Prog. Phys.,74:046601, 2011. http://arxiv.org/submit/199903.

[6] Hugo Touchette. The large deviation approach tostatistical mechanics. Phys. Rep., 478:1–69, 2009.arxiv:0804.0327.

[7] Enrico Fermi. Thermodynamics. Dover, New York, 1956.[8] Michel Benaim and Jorgen W. Weibull. Deterministic

approximation of stochastic evolution in games. Econo-metrica, 71:873–903, 2003.

[9] Nicolas Champagnat. Large deviations for singularand degenerate diffusion models in adaptive evolution.arXiv:0903.2345v1.

[10] Dean Foster and Peyton Young. Stochastic evolutionarygame dynamics. Theor. Pop. Biol., 38:219–232, 1990.

[11] David Canning. Average behavior in learning models.J. Econ. Theory, 57:442–472, 1992.

[12] Michihiro Kandori, George J. Mailath, and Rafael Rob.Learning, mutation, and long run equilibria in games.Econometrica, 61:29–56, 1993.

[13] H. Peyton Young. The evolution of conventions. Econo-metrica, 61:57–84, 1993.

[14] H. Peyton Young. The economics of conventions.J. Econ. Perspectives, 10:105–122, 1996.

[15] Conventional contracts. Young, h. peyton.Rev. Econ. Studies, 65:776–792, 1998.

[16] R. Graham and T. Tel. Existence of a potential for dis-sipative dynamical systems. Phys. Rev. Lett., 52:9–12,1984.

[17] R. Graham and T. Tel. Weak-noise limit of fokker-planckmodels and nondifferential potentials for dissipative dy-namical systems. Phys. Rev. A, 31:1109–1122, 1985.

[18] Robert S. Maier and D. L. Stein. Escape problem forirreversible systems. Phys. Rev. E, 48:931–938, 1993.

[19] Robert S. Maier and D. L. Stein. Effect of focusing andcaustics on exit phenomena in systems lacking detailedbalance. Phys. Rev. Lett., 71:1783–1786, 1993.

[20] Robert S. Maier and Daniel L. Stein. Asymptotic exit lo-cation distributions in the stochastic exit problem. SIAMJ. App. Math., 57:752, 1997.

[21] Thomas G. Kurtz. Approximation of population pro-cesses. Soc. Ind. Appl. Math., Philadelphia, PA, cbms-nsfvol.36 edition, 1981.

[22] Stewart N. Ethier and Thomas G. Kurtz. Markov Pro-cesses: Characterization and Convergence. Wiley, NewYork, 1986.

[23] Jin Feng and Thomas G. Kurtz. Large Deviations forStochastic Processes. Providence, Rhode Island, 2006.

[24] M. Doi. Second quantization representation for classicalmany-particle system. J. Phys. A, 9:1465–1478, 1976.

[25] M. Doi. Stochastic theory of diffusion-controlled reaction.J. Phys. A, 9:1479–, 1976.

[26] L. Peliti. Path-integral approach to birth-death processeson a lattice. J. Physique, 46:1469, 1985.

[27] L. Peliti. Renormalization of fluctuation effects in a+a →

a reaction. J. Phys. A, 19:L365, 1986.[28] Erwin Frey. Evolutionary game theory: non-

equilibrium and non-linear dynamics of interact-ing particle systems. Boulder lecture notes, pages1–39, 2009. http://boulder.research.yale.edu/Boulder-2009/ReadingMaterial-2009/Frey/frey lecture notes games.pdf.

[29] N. G. van Kampen. Stochastic Processes in Physics andChemistry. Elsevier, Amsterdam, third edition, 2007.

[30] Richard P. Boland, Tobias Galla, and Alan J. McKane.How limit cycles and quasi-cycles are related in systemswith intrinsic noise. 2008. arXiv:0805.1607v1.

[31] Richard P. Boland, Tobias Galla, and Alan J. McKane.Limit cycles, complex Floquet multipliers, and intrinsicnoise. arXiv:0903.5248v1.

[32] J. L. Cardy. Electron localisation in disordered systemsand classical solutions in ginzburg-landau field theory.J. Phys. C, 11:L321 – L328, 1987.

[33] Sidney Coleman. Aspects of symmetry. Cambridge, NewYork, 1985.

[34] John Maynard Smith. Evolution and the theory of games.Cambridge, London, 1982.

[35] Josef Hofbauer and Karl Sigmund. Evolutionary Gamesand Population Dynamics. Cambridge U. Press, NewYork, 1998.

[36] Martin A. Nowak. Evolutionary Dynamics: exploring theequations of life. Belknap Press, New York, 2006.

[37] G. R. Price. Fisher’s ‘fundamental theorem’ made clear.Ann. Hum. Genet., 36:129–140, 1972.

[38] Steven A. Frank. George Price’s contributions to evolu-tionary genetics. J. Theor. Biol., 175:373–388, 1995.

[39] John Maynard Smith and George R. Price. The logic ofanimal conflict. Nature, 246:15–18, 1973.

[40] E. T. Jaynes. Probability theory : the logic of science.Cambridge U. Press, New York, 2003.

[41] Warren J. Ewens. Mathematical Population Genetics.Springer, Heidelberg, second edition, 2004.

[42] Takeo Matsubara. A new approach to quantum-statistical mechanics. Prog. Theor. Phys. (Kyoto),14:351–378, 1955.

[43] J. Schwinger. Brownian motion of a quantum oscillator.J. Math. Phys., 2:407–32, 1961.

[44] L. V. Keldysh. Diagram technique for nonequilibriumprocesses. Sov. Phys. JETP, 20:1018, 1965.

[45] Eric Smith. Quantum-classical correspondence principlesfor locally non-equilibrium driven systems. Phys. Rev. E,77:021109, 2008. originally as SFI preprint # 06-11-040.

[46] P. C. Martin, E. D. Siggia, and H. A. Rose. Statisticaldynamics of classical systems. Phys. Rev. A, 8:423–437,1973.

[47] Gregory L. Eyink. Action principle in nonequilibriumstatistical dynamics. Phys. Rev. E, 54:3419–3435, 1996.

[48] Daniel C. Mattis and M. Lawrence Glasser. The usesof quantum field theory in diffusion-limited reactions.Rev. Mod. Phys, 70:979–1001, 1998.

[49] Alex Kamenev. Keldysh and doi-peliti techniquesfor out-of-equilibrium systems. pages arXiv:cond–

37

mat/0109316v2, 2001. Lecture notes presented at Wind-sor NATO school on “Field Theory of Strongly Corre-lated Fermions and Bosons in Low-Dimensional Disor-dered Systems” (August 2001).

[50] C. Jarzynski. Nonequilibrium work relations: founda-tions and applications. Eur. Phys. J. B, 64:331–340,2008.

[51] R. Graham. Path integral formulation of general diffusionprocesses. Z. Phys. B, 26:281–290, 1977.

[52] Herbert Goldstein, Charles P. Poole, and John L. Safko.Classical Mechanics. Addison Wesley, New York, thirdedition, 2001.

[53] J. Cardy. Field theory and non-equilibriumstatistical mechanics. 1999. http://www-thphys.physics.ox.ac.uk/users/JohnCardy/home.html.

[54] S. Pilgram, A. N. Jordan, E. V. Sukhorukov, and

M. Buttiker. Stochastic path integral formulation of fullcounting statistics. Phys. Rev. Lett., 90:206801, 2003.

[55] N. A. Sinitsyn and Ilya Nemenman. Universal geomet-ric theory of mesoscopic stochastic pumps and reversibleratchets. Phys. Rev. Lett., 99:220408:1–4, 2007.

[56] N. A. Sinitsyn and J. Ohkubo. Hannay angle and geo-metric phase shifts under adiabatic parameter changes inclassical dissipative systems. J. Phys. A: Math. Theor.,41:262002:1–4, 2008.

[57] N. A. Sinitsyn. The stochastic pump effect and geometricphases in dissipative and stochastic systems. J. Phys. A:Math. Theor., 42:193001:1–33, 2009.

[58] N. A. Sinitsyn, Nicalas Hengartner, and Ilya Nemenman.Adiabatic coarse-graining and simulations of stochas-tic biochemical networks. Proc. Nat. Acad. Sci. USA,106:10546–10551, 2009.

[59] John Gluckheimer and Philip Holmes. Structurally sta-ble heteroclinic cycles. Math. Proc. Cam. Phil. Soc.,103:189–192, 1988.

[60] John Gluckheimer and Philip Holmes. Nonlinear Oscil-lations, Dynamical Systems, and Bifurcations of VectorFields (Applied Mathematical Sciences vol.42). Springer,Berlin, 2002.

[61] Robert S. Maier and D. L. Stein. Oscillatory behaviorof the rate of escape through an unstable limit cycle.Phys. Rev. Lett., 77:4860–4863, 1996.

[62] Steven Weinberg. The quantum theory of fields, Vol. I:Foundations. Cambridge, New York, 1995.

[63] Supriya Krishnamurthy, Eric Smith, David C. Krakauer,and Walter Fontana. The stochastic behavior of a molec-ular switching circuit with feedback. Biology Direct, 2:13,2007. doi:10.1186/1745-6150-2-13.

[64] Eric Smith, Supriya Krishnamurthy, Walter Fontana,and David C. Krakauer. Non-equilibrium phase transi-tions in biomolecular signal transduction. Phys. Rev. E,2007.

[65] L. Onsager and S. Machlup. Fluctuations and irreversibleprocesses. Phys. Rev., 91:1505, 1953.

[66] Nigel Goldenfeld. Lectures on Phase Transitions and theRenormalization Group. Westview Press, Boulder, CO,1992.

[67] R. M. Nisbet and W. S. C. Gurney. Modelling fluctuatingpopulations. Wiley, New York, 1982.

[68] Steven Weinberg. The quantum theory of fields, Vol. II:Modern applications. Cambridge, New York, 1996.

[69] Kerson Huang. Statistical mechanics. Wiley, New York,1987.