Evolution & Learning in GamesEcon 243B

Jean-Paul Carvalho

Lecture 10.Stochastic Dynamics

1 / 31

The Stochastic Evolutionary Process

I Now we shall directly analyze the stochastic evolutionarydynamic rather than its deterministic approximation.

I Let the population be large but finite, with N members.

I The set of feasible social states is then a discrete gridembedded in X:

XN = X⋂

1N Zn = {x ∈ X : Nx ∈ Zn}.

2 / 31

Markov PropertySuppose choices depend only on current state x and currentpayoff vector π.

Then the stochastic evolutionary process {XNt } is a

continuous-time Markov process on the finite state space XN.

I The process is fully characterized by its transitionprobabilities {PN

xy}x,y∈XN .

I PNxy(t) is the probability of transiting from state x to y in

exactly one revision.

I If PNxy(t) is independent of t, then the Markov process is

time homogenous, and we write PNxy. This is the case we will

be dealing with.

3 / 31

Transition Matrix

I PN is a |XN| × |XN|matrix, called the transition matrix,whose elements are the transition probabilities.

I For all x, ∑y∈XN PNxy = 1.

I This means that PN is a row stochastic matrix, i.e. its rowelements sum to one.

I For example, consider the following transition matrix for atwo-state Markov process:

PN =

( 12

12

14

34

).

4 / 31

Representation

Another way to describe this process is via a graph:

5 / 31

Population Game Version

In the standard population game we have focussed on, strategyrevisions are determined by independent rate 1 Poisson clocksand the learning protocol ρ:

I When an agent playing i ∈ S receives a revisionopportunity, it switches to j with probability ρij.

I The probability that the next revision involves a switchfrom strategy i to j is then xiρij.

I The transition involves one less agent playing i and onemore agent playing j, i.e. the state is shifted by 1

N (ej − ei).

6 / 31

Population Game Markov Process

Definition. A population game F, a revision protocol ρ, arevision opportunity arrival rate of one, and a population sizeN define a Markov process {XN

t } on the state space XN.This process is described by some initial state XN

0 = xN0 and the

transition probabilities:

PNx,x+z =

xiρij

(F(x), x

)if z = 1

N (ej − ei), i, j ∈ S, i 6= j1−∑i∈S ∑j 6=i xiρij

(F(x), x

)if z = 0

0 otherwise

7 / 31

Limiting Behavior

I Over finite-horizons, the focus is on the mean dynamic.

I In the long run, however, the object of interest is thestationary distribution µ of the process {Xt} (we are nowdropping the N superscript).

I The stationary distribution tells us the frequencydistribution of visits to each state as t→ ∞, i.e. almostsurely the process spends proportion µ(x) of the time instate x.

I Before analyzing stationary distributions, let us introducesome definitions.

8 / 31

CommunicationI State y is accessible from x if there exists a positive

probability path from x to y, i.e. a sequence of statesbeginning in x and ending in y in which each one steptransition between states has positive probability under P.

I States x and y communicate if they are each accessiblefrom the other.

I A set of states E is closed if the process cannot leave it, i.e.for all x ∈ E and y /∈ E, Pxy = 0.

I An absorbing state is a singleton closed set.

I Every state in X is either transient or recurrent; and a stateis recurrent if and only if it is a member of a closedcommunication class.

9 / 31

Recurrence

Theorem 9.1. Let {Xt} be a Markov Process on a finite set X .

Then starting from any state x0, the frequency distribution ofvisits to states converges to a stationary distribution µ, whichsolves µP = µ. In such a vector, µ(x) = 0 for all transient states.

I This is why closed communication classes are commonlycalled recurrence classes.

10 / 31

Stationary Distributions

I FACT: every non-negative row stochastic matrix has atleast one left eigenvector with eigenvalue one, i.e. thereexists a µ such that µP = µ.

I In other words there exists at least one stationarydistribution.

Theorem 9.2.(i) If a stationary distribution µ is unique, it is the long runfrequency distribution independent of the initial state.

(ii) If there are multiple stationary distributions, then the longrun frequency distribution is among these, and can be any oneof them.

11 / 31

Stationary DistributionsI In our previous example, µ = ( 1

3 ,23 ) (check by showing

that µ solves µP = µ).

I Consider the following Markov process:

I The solutions to the stationarity equation are:

µ1 = ( 13 ,

23 , 0, 0), µ2 = (0, 0, 0, 1),

or any convex combination of µ1 and µ2.12 / 31

Irreduciblity

I A Markov Process is irreducible if there is a positiveprobability path from each state to every other, i.e. if allstates in X communicate, or equivalently if X forms asingle recurrent class.

Theorem 9.3. If the Markov process {Xt} is irreducible then ithas a unique stationary distribution, and this stationarydistribution is independent of the initial state.

In this case, we say that the process {Xt} is ergodic, itslong-run behavior does not depend on initial conditions.

13 / 31

k-Step Ahead ProbabilitiesI We may not only want to know the proportion of time the

process spends in each state (given by µ), but also theprobability of being in a given state at some future point intime.

I Recall that P(X1 = y|X0 = x) = Pxy is a one step transitionprobability.

I Two step transition probabilities are computed bymultiplying P by itself:

P(X2 = y|X0 = x) = ∑z∈X

P(X2 = y,X1 = z|X0 = x)

= ∑z∈X

P(X1 = z|X0 = x)P(X2 = y|X1 = z,X0 = x)

= ∑z∈X

PxzPzy

=(P2)

xy.

14 / 31

AperiodicityI By induction, the t-step transition probabilities are given

by the entries of the tth power of the transition matrix:

P(Xt = y|X0 = x) =(Pt)

xy.

I Let Tx be the set of all positive integers T such that there isa positive probability of moving from x to x in exactly Tperiods.

I The process is aperiodic if for every x ∈ X , the greatestcommon denominator of Tx is 1.

This holds whenever the probability of remaining in each stateis positive, i.e. Pxx > 0 for all x ∈ X .

15 / 31

Aperiodicity

I If {Xt} is irreducible and aperiodic, then with probabilityone:

for all x0, x ∈ X limt→∞

(Pt)

x0x = µ(x).

I Therefore, from any initial state x0 both the proportion oftime the process spends in each state up through time tand the probability of being in each state at time t convergeto the stationary distribution µ.

I Hence the stationary distribution provides a lot ofinformation about the long run behavior of the process.

16 / 31

Full Support Revision Protocols

I Irreducibility and aperiodicity are desirable properties of aMarkov process.

I They are both generated by full support revision protocols,i.e. revision protocols in which all strategies are chosenwith positive probability.

I Let us consider two examples which are extensions of bestresponse protocols.

17 / 31

Full Support Revision Protocols

I Best response with mutations:

I A revising agent switches to its current best response withprobability 1− ε, and chooses a strategy uniformly(mutates) with probability ε > 0.

I Let us refer to this protocol as BRM(ε).

I In case of best response ties, it is often assumed that anon-mutating agent sticks with its current strategy if it is abest response; otherwise it chooses at random among fromthe set of best responses.

18 / 31

Full Support Revision ProtocolsI Logit Choice:

ρij(π) =exp(η−1πj)

∑k∈S exp(η−1πk).

I We can define ε−1 = exp(η−1), where ε is an increasingfunction of η. As η → 0, ε→ 0.

I In this way, we can rewrite the revision protocol as follows:

ρij(π) =ε−πj

∑k∈S ε−πk.

19 / 31

Stationary Distributions

I There are two problems:

I It may not be possible to compute the stationarydistribution explicitly,

I Even if it is possible to do so, the stationary distributionmay spread weight widely over the state space.

20 / 31

Analyzing Large-Dimensional Markov Processes

I In this lecture:

I We shall study a class of reversible Markov processeswhose stationary distributions are easy to compute;

I In the next lecture, we shall:

I Introduce the concept of stochastic stability, which candrastically reduce the number of states which attractpositive weight in the stationary distribution.

21 / 31

Reversible Markov Processes

I Reversible Markov processes permit easy computation ofµ even if the state space X , and hence the |X | × |X |transition matrix P, is large.

I A process {Xt} is reversible if it admits a reversibledistribution, i.e. a probability distribution µ on X thatsatisfies the following detailed balance conditions:

µxPxy = µyPyx for all x, y ∈ X. (1)

I Such a process is reversible in the sense that it looks thesame whether time is run forward or backward.

22 / 31

Reversible Markov ProcessesI Recall that a stationary distribution µ satisfies:

∑x∈X

µxPxy = µy for all y ∈ X. (2)

I Summing (1) over x we get:

∑x∈X

µxPxy = ∑x∈X

µyPyx

= µy ∑x∈X

Pyx

= µy.

(3)

I Therefore, a reversible distribution is also a stationarydistribution.

23 / 31

Reversible Markov Processes

I There are two contexts in which the stochasticevolutionary process {Xt} is known to be reversible:

1. two-strategy games (under arbitrary revision protocols),

2. potential games under exponential protocols.

I We shall now study the first and leave the second to later.

24 / 31

Two-Strategy Games

I Let F : X→ R2 be a two strategy game, with strategy set{0, 1}, full support revision protocol ρ : R2 ×X→ R2×2,and finite population size N.

I This defines an irreducible and aperiodic Markov Process{Xt} on the state space XN.

I For this class of games, let x ≡ x1. The state of the processis fully described by x.

I Therefore, the state space is XN = {0, 1N , . . . , 1}, a

uniformly spaced grid embedded in the unit interval.

25 / 31

Birth and Death Processes

I Because revision opportunities arrive independently incontinuous time, agents switch strategies sequentially.

I This means that transitions are always between adjacentstates.

I If in addition the state space is linearly ordered (which it isin a two-strategy game), then we refer to the Markovprocess as a birth and death process.

I We shall now show that a stationary distribution for suchprocesses can be calculated in a straightforward way.

26 / 31

Birth and Death Processes

I In a birth and death process, there are vectors p, q ∈ R|X |

with p1 = q0 = 0 such that the transition matrix takes thefollowing form:

Pxy ≡

px if y = x + 1

N ,

qx if y = x− 1N ,

1− px − qx if y = x,0 otherwise

I The process is irreducible if px > 0 for all x < 1 and qx > 0for all x > 0, which is what we shall assume.

27 / 31

Birth and Death ProcessesI Because of the “local” structure of transitions, the

reversibility condition reduces to:

µxqx = µx−1/Npx−1/N

for all x > 0.I Applying the formula inductively, we have:

µxqxqx−1/N . . . q1/N = µ0p0p1/Np2/N . . . px−1/N.

I That is, the process running ‘down’ from state x to zeroshould look like the process running ‘up’ from zero to statex.

28 / 31

Stationary DistributionI Rearranging, we see that the stationary distribution

satisfies:

µx

µ0=

Nx

∏j=1

p(j−1)/N

qj/Nfor all x ∈ { 1

N , . . . , 1}. (4)

I For a full support revision protocol ρ, the upward anddownward probabilities are given by:

px = (1− x)ρ01(F(x), x

)qx = xρ10

(F(x), x

).

(5)

29 / 31

Stationary Distribution

I Substituting the expressions in (5) into (4) yields:

µx

µ0=

Nx

∏j=1

p(j−1)/N

qj/N=

Nx

∏j=1

1− j−1N

jN

.ρ01

(F(

j−1N

),

j−1N

)ρ10

(F(

jN

),

jN

) (6)

for all x ∈ { 1N , . . . , 1}.

30 / 31

Stationary Distribution

Simplifying, we have the following result:

Theorem 9.4. Suppose that a population of N agents plays thetwo-strategy game F using the full support revision protocol ρ.Then the stationary distribution for the evolutionary process{XN

t } on XN is given by:

µx

µ0=

Nx

∏j=1

N− j + 1j

.ρ01(F( j−1

N ), j−1N

)ρ10(F( j

N ), jN

) for x ∈ { 1N , . . . , 1}, (7)

with µ0 determined by the requirement that ∑x∈X µx = 1.

31 / 31