THE CO-EVOLVABILITY OF GAMES IN

COEVOLUTIONARY GENETIC ALGORITHMS

林偉楷 2009.3.12

Taiwan Evolutionary Intelligence Laboratory

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

2

Outline

Introduction Mixed strategy games, less co-evolvable Pure strategy games, more co-evolvable

Needle-in-a-haystack gameSimple state gameCoevolutionary GA with nichingThe optimal number of opponents

A less co-evolvable pure strategy game Conclusions

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

3

Introduction: Coevolution (1) Coevolutionary algorithm evaluates the

fitness of a solution by other candidate solutions.Coevolutionary fitness is absolute or

subjectiveTraditional EC fitness is relative or objective

No objective measures exist Objective measure difficult to formalize

or unknown

4

Introduction: Coevolution (2)

Ref: de Jong et al. Introductory tutorial on coevolution. GECCO ’07.

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

CEA

Population

Player 1

Player 2

Game Playing Strategies

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

5

Introduction: Coevolution (3) Coevolution can also be applied to

certain types of structure in search space

CEA

Population

Agent 1

Population

Agent 2

Population

Agent 3

Multiagent Teams

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

6

Introduction: Pure Strategies In game theory, a game can be defined

by a set of players, the valid strategies of each player, and the payoff of all players in each strategy profile.

Pure strategy refers to such deterministic strategy (and thus payoff).

Example: rock, scissor and paper are 3 pure strategies.

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

7

Introduction: Mixed Strategies In contrast to pure strategy, a mixed

strategy plays each pure strategy with a probability.

The payoff is than considered as an expected value.

Example: randomly plays rock, scissor and paper with probability 0.4, 0.3 and 0.3.

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

8

Introduction: Some Properties of Games We Interested In

Some properties are common in real-world games:Two-player: games involving more players

are very complexZero-sum: no win-win strategySymmetric: both player are unbiased

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

9

Mixed Strategy, Less Co-evolvable For a two-player, zero-sum and

symmetric game with a non-pure mixed strategy Nash equilibrium, the equilibrium strategy does not get a higher payoff than other strategies.

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

10

Rock-paper-scissors

In a population with equal probability to play R/P/S in average, different strategies have the payoff:

Strategy The probability of each outcome

Rock Paper Scissor Win Tie Lose

1/3 1/3 1/3 1/3 1/3 1/3

1 0 0 1/3 1/3 1/3

0 1 0 1/3 1/3 1/3

0 0 1 1/3 1/3 1/3

All their linear combinations 1/3 1/3 1/3

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

11

Turn-Based Games

We focus on turn-based games, which has huge solution space.

Turn-based games can be modeled in a state-transition concept: for each state, the player makes an action and it goes to another state.

For a board game, a state may contains the board position and whose turn

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

12

An Uninteresting Game:Needle-in-a-Haystack Consider a game with two players, each

player has N strategies and the payoff is:P1 \ P2 1 2 3 ... N-1 N

1 (1,-1) (1,-1) (1,-1) (1,-1) (-1,1)

2 (1,-1) (1,-1) (1,-1) (1,-1) (-1,1)

3 (1,-1) (1,-1) (1,-1) (1,-1) (-1,1)

⁞ ⁞

N-1 (1,-1) (1,-1) (1,-1) ... (1,-1) (-1,1)

N (1,-1) (1,-1) (1,-1) (1,-1) (0,0)

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

13

A Simple State Game

Moving a flag along a straight line, one unit a turn for both players, and terminates in t turns

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

14

Coevolutionary GA with RTS

Sample a random population If terminating condition is not satisfied,

repeatGenerate a new population by crossoverEvaluate the fitness of all individuals by

playing games with a set of opponents in the population

For each individual in the new population, find the nearest one in the old population and replace it if the fitness is higher.

15

Coevolution with Niching is Better

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

16

The Optimal Number of Opponents (1)

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

17

The Optimal Number of Opponents (2)

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

18

Evolving a Heuristic

In real-world, board games is large:The number of steps is proportional to the

board size.The number of states is exponential to the

number of steps.The number of strategies is still exponential

to the number of states! Thus we prefer to evolve a heuristic.

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

19

Less Co-Evolvable Games A perfect strategy need to “remember”

the decision at an exponential number of states.In the worst case, we need to perform an

exponential number of tournaments. A perfect heuristic may not evolvable

due to the above large number.

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

20

Bit-Flipping Game

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

21

The Required Population Size

The Co-Evolvability of Games in Coevolutionary Genetic Algorithms

22

Conclusions

The mixed strategy games is not co-evolvable

Niching is a helpful technique to pure strategy games.

The optimal number of opponents used to evaluate a strategy exists.

The existence of less co-evolvable pure strategy games.