Video Games As Environments For Learning And Planning:

What’s Next?Julian Togelius

A very selective history• Othello

• Backgammon

• Checkers

• Chess

• Go

• Poker

• Super/Infinite Mario Bros

• Ms. Pac-Man

• Crappy Atari games

• Montezuma’s Revenge

• Dota 2

• StarCraft 2

What’s next?

What’s left?

How we will think of this

Characteristics of games

Characteristics of games

• Number of players1, 1.5, 2, many; competitive, cooperative, both?

• Stochasticity

• Observability

• Action space and branching factor

• Time granularityFew turns, many turns, continuous?

Branching factor• Checkers: 3

• Pac-Man: 4

• Super Mario Bros: 16

• Chess: 35

• Go: 350

• Civilization: ?

• Fortnite: ?

Characteristics of the game-AI interface

• How is the game state represented?Pixels, text, simulated sensors, some internal representation?

• Is there a forward model?Is it fast? Accurate?

• Do you have time to train?How much?

• How many games are you playing?

Characteristics ofcommon benchmarks

• Classic board games: lowish branching factor, few turns, two-player adversarial, complete information, no stochasticity, structured game state representation, forward model readily available

• Poker: lots of hidden information, stochastic, otherwise like above

• Atari 2600: low branching factor, longish, single-player, mostly complete information, no stochasticity (!), pixels, no/slow forward model

• StarCraft: insane branching factor, long games, two-player adversarial, hidden information, some stochasticity, pixels (more or less), no forward model

Another way of thinking

Which cognitive abilities are exercised by a particular game?

Which cognitive abilities that humans have are not tested by current game-based AI benchmarks?

If we create AI that can play games that require certain abilities of humans, will agents that solve these games need to have the same abilities?

Which challenges are next?

Wait a minute, why are we doing this?

To improve specific AI capabilities?

To achieve AGI?

To make games better?

To keep unemployed AI researchers off the streets?

Which challenges are next?• Role-playing games

• Roguelikes

• Multiplayer games

• 4X games

• Open world games

• Gardening/building games

• Generalization/Transfer

• Human-like playing

• Resource-constrained playing

• Things that are not game-playing

Role-playing games

Text as input; very long game length; reasoning about motives, emotions and social practices

Text adventure games

Roguelikes

Handling radical uncertainty, generalization

Multiplayer games

Multiplayer dynamics (example: Pommerman)

4X games

Example: Civilization 6. Ridiculous branching factor, n-player, extremely long game time.

So many game genres• Dating simulators

• Rhythm games

• Word games

• Grand strategy war-games

• Life simulations

• Weirdo casual games of all kinds

• Things that come out of the global game jam

Generalization/Transfer

Example: Obstacle Tower.Small branching factor, single-player, pixel inputs with very complex visuals, thoroughly randomized puzzles and visuals, potentially endless game time

Generalization/Transfer

Example: General Video Game AI

Human-like playing

• Playing games to win is one thing - how about playing them like a human?

• Same strengths, same weaknesses, same preferences

• Very useful for the game industry

Resource-constrained playing

• How to do it without 5000 TPUs

• Academics like this

• Obvious practical applications

Things that are not game-playing

• Procedural content generation

• Player experience modeling

• Game modeling

• Game generation

• Ultimately, inventing new games for AI to play automatically