Application of Monte Carlo Tree Search in a Fighting Game AI (GCCE 2016)

Application of Monte-Carlo Tree Search in a Fighting Game AI

Application of Monte-Carlo Tree Search in a Fighting Game AIShubu Yoshida, Makoto Ishihara, Taichi Miyazaki,Yuto Nakagawa, Tomohiro Harada, and Ruck ThawonmasIntelligent Computer Entertainment LaboratoryRitsumeikan University

Hello everyone. My name is shubu yoshida of Intelligent Computer Entertainment Lab, Ritsumeikan University.Id like to talk about Application of Monte-Carlo Tree Search in a Fighting Game AI .

1

OutlineBackground of this researchMonte-Carlo Tree SearchMonte-Carlo Tree Search for a Fighting GameExperimental EnvironmentExperimental MethodResultCompetition result in 2016Conclusion

This is the outline of my presentation.Id like to talk about these contents.

2

Background (1/2)A Fighting Game AI Competition is held every year [1]High-ranking AIs = Rule-based(until 2015)Rule-based : a same action in a same situationHuman player can easily predict the AIs action patterns and outsmart it

[1] http://www.ice.ci.ritsumei.ac.jp/~ftgaic/

A Fighting Game AI Competition is held every year.In this competition, High-ranking AIs are mainlywell-tuned rule-based AIs which always conduct a same action in a same situation. Rule-based AIs take predetermined actions. Human players can easily predict the AIs action patterns and outsmart it.And if the parameters of the action changed, Rule-based AIs strength be changed

3

Background (2/2) Apply the Monte-Carlo Tree Search (MCTS) to a fighting game AIDecides a next own action by stochastic simulations Already successful in many games [2][3] We evaluate the effectiveness of MCTS on a ghting game

[2] S. Gelly, et al. The Grand Challenge of Computer Go: Monte Carlo Tree Search and Extensions, Communications of the ACM, Vol. 55, No. 3, pp. 106-113, 2012.

[3] N. Ikehata and T. Ito. Monte-carlo tree search in ms. pac-man. In Computational Intelligence and Games (CIG), 2011 IEEE Conference on, pp. 39-46, 2011

In order to solve this problem, we apply MCTS to a Fighting Game AI.MCTS decides a next own action by stochastic simulations. MCTS based approach produces a signicantly promising result not only in a board game like Go [2], but also in a realtime based game like Ms.Pac-Man [3]. Then, it is expected that it performs better in a ghting game because this kind of game is similar to Ms.Pac-Man in terms of real-time based. it is expected that it performs better in a ghting game.In this paper, we evaluate the effectiveness of MCTS on a ghting game.

4

Monte-Carlo Tree Search (1/5)

selectionsimulationbackpropagation

repeat until the set time has elapsedexpansion

We modified traditional MCTS for a fighting game.This figure is an overview of traditional MCTS.Ill explain you about this.And after having explained this, Ill explain you about MCTS for fighting game. MCTS combines the game tree search and the Monte Carlo method.Each node represents a state of the game. Each edge an action.

5



repeat until the set time has elapsed

expansion

First, MCTS selects the child node with the highest UCB1 value until it reaches a leaf node.Each child node has a UCB1 value.

6

Formula of UCB1Preferentially select a child node that has been visited less The evaluation value ExploitationExploration

7




expansion

Second, after arriving at a leaf node, if its number of visits exceeds a pre-defined threshold and the depth of the tree has not reached the upper limit, MCTS will create child nodes from it.8




expansion

Third, it performs random simulation from the root node to the leaf node.And it simulate until the end of game.In this part, opponent actions are selected randomly and my actions are used in that path. After do these actions, we get reward and state.

9




expansion

Finally, it propagates a result of simulation from the leaf node to the parent node and calculates UCB1 values and repeat propagation until the root node.The above 4 steps are repeated during allowed time budget in MCTS. Then, the child node is chosen with the highest number of visits from the root node.

10

MCTS for a Fighting Game (1/2)

11

MCTS for a Fighting Game (2/2)

Expansion normal fighting game

Simulation

In the expansion part, traditional MCTS expands only one node at a time.In this paper, we expand all actions or nodes that the AI can act.Fighting games have a lot of actions, and real time games have search time limit.We want to explore all of the nodes once at least.So we expand all actions that the AI can act.In the simulation part, in board games, simulation is done until the end of the game.But real-time games have limited thinking time. So we put restrictions on tree depth.These are the main changes in MCTS for fighting games.12

Experimental Environment

In an experiment, We used FightingICE as the fighting game platform. FightingICE is a 2D fighting game developed by our laboratory for game ai researches. It is used as the platform of international fighting game AI competitions recognized by IEEE CIG.The player AI score or My score is calculated by this formula.If more than 500, my AIs performance is superior to the opponent AI

13

Experimental MethodMCTSAI(AI applying MCTS) vs high ranking 5 AIs of 2015 tournament5 AIs : Rule-based100 games (50games each side) TABLE I THE PARAMETERS USED IN THE EXPERIMENTS

Notations MeaningsValuesCBalance parameter3Threshold of the number of visits10Threshold of the depth of tree2The number of simulations60 frames

Next , experimental method.We let MCTSAI fight 100 times against high ranking 5 AIs of 2015 tournament, while switching each side .Action behaviors of each AI are rule-based.And we used these parameters.14

Result (1/5)Fig. 1. The average scores against high ranking 5 AIs of 2015 tournament

The average score against each AI is shown in Fig. 1.In this figure, the horizontal axis lists the name of high ranking AI.And from left to right, there are 1st ranked to 5th ranked Ais.The vertical axis represents the average scores of MCTSAI againtw high ranking Ais.

15

Result (2/5)

Fig. 1. The average scores against high ranking 5 AIs of 2015 tournament

From this result, the proposed AI outperformed all opponent AIs, except for the 1st ranked AI Machete.

16

Result (3/5)

P1 : MCTSAI P2 : RatioBot

This video is a fighting game scene where P1 is MCTSAI and P2 is RatioBot.RatioBot is the 4th ranked ai in 2015 tournament.As we can see from this video,MCTSAI has been able to dodge the behavior of RatioBot.It can be said that the simulation of Monte Carlo tree search has been working well.So MCTS is an effective method in this fighting game.

////

17

Result (4/5)

Fig. 1. The average scores against high ranking 5 AIs of 2015 tournament

But the proposed AI did not show a good performance against Machete.

18

Result (5/5) P1 : MCTSAI P2 : Machete

This video is a fighting game scene where P1 is MCTSAI and P2 is Machete.Machete is a well tuned rule-based AI that repeatedly conducts short actions, requiring less number of frames, which are not well simulated by MCTS RANDOM simulation.

19

Competition result in 2016Orange1stBlue2ndGreen3rd

Total RankRANKBANZAI11DragonSurvivor12iaTest7IchibanChan9JayBot20165KeepYourDistanceBot10MctsAi3MrAsh4Poring8Ranezi2Snorkel13Thunder011Tomatensimulator6Triump14

This is the competition result in 2016.the horizontal axis lists the name of AI.And these numbers represent these AIs Ranking.In this competition, our MctsAI came 3rd.So it can be said that Mcts showed good results also in an actual tournament.

20

ConclusionApplied MCTS to fighting game AIShowed that MCTS in fighting game AI is effective

Future workIn fighting game, random simulation of the enemy behavior is not effective

Predict the behavior of the enemy and use this information in simulation

In conclusion we applied MCTS to a fighting game AI.Results showed that MCTS in fighting game AI is effective.In this paper, we have found that random simulation of the enemy behavior is not effective in fighting games.So, in the future, we plan to add a mechanism such as behavior prediction of the enemy and use it in simulation.Use of this kind of mechanism should better simulate the opponent.

21

Thank you for listening

22

Technology

Application of Monte Carlo Tree Search in a Fighting Game AI (GCCE 2016)