Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Game-based Data
Capture for Player
Metrics
Aline Normoyle,
John Drake
University of Pennsylvania
Maxim Likhachev
Carnegie Mellon University
Alla Safonova
Disney Research, Pittsburgh
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 1
Player Metrics
• What are they?
– Player statistics
– Objective description of player behaviors
– Examples
• Deaths per level
• Percentage of time spent in an activity
• Why collect them?
– Playtesting and design
– Train Bots and Non-Player Characters
– Customize gameplay
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 2
Valve, GDC 2009
Gathering Metrics
• Passively record players
• Potential drawbacks
– Desired player behaviors may be infrequent
– Data acquisition costly, laborious, memory intensive
• Can we reduce the number of data capture
sessions we need to run?
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 3
Our approach
• Investigate active approach for acquiring metrics
– Make automated data capture more efficient
– Focus on metrics which most require data
– Model correspondence between player metrics and
game configurations
• Use model to choose the best game configuration to run next
• No need to know player behaviors a priori
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 4
Related Work –
Playtesting
• Identifying problems and improving gameplay [Kennerly 2003; DeRosa 2007; Luban 2009; Ambinder 2009]
• Designing more effective development/test cycles [Medlock et al. 2002]
• Designing automated methods for logging, tracking, and reporting [Kennerly 2003; DeRosa 2007; Kim et al. 2008]
• Organizing and visualizing the large data sets collected with logging [Chittaro, Ranon, and Ieronutti 2006; Andersen et al. 2010; Moura, el Nasr,and Shaw 2011]
• Designing better models for understanding player data [Tychsen and Canossa 2008; Drachen and Canossa 2009; Mahlmann et al. 2010]
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 5
Related Work –
Active Learning
• Goal: Reducing manual effort or cost [Chaloner and Verdinelli 1995;
Settles 2012]
• Reducing manual labeling – text classification [Roy and Mccallum 2001; Hoi, Jin, and Lyu 2006]
– image classification and retrieval [Tong and Chang 2001; Hoi et al. 2006]
– speech recognition [Tur, Hakkani-Tur, and Schapire 2005; Riccardi and Hakkani-Tur 2005]
• Optimal experiment design – “robot-scientist” [King et al. 2004]
– animation controller [Cooper, Hertzmann, and Popovic 2007]
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 6
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 7
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 8
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Input: Desired metrics
Input: Game configurations
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 9
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 10
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 11
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 12
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 13
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 14
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 15
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 16
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 17
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 18
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 19
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Active data acquisition
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 20
• GAMELAB: autonomously collects player metrics
• Iteratively collect data to improve the estimates for each metric
Game
Scheduler
Run
Game data
game configuration
Input: Desired metrics
Input: Game configurations
Output: Metric models
Run
Game
Game Scheduler –
Overview
• Goal: Minimize confidence intervals across all metrics using as few games as possible
• Idea: – Estimate how accurate our current metrics are
• Confidence intervals
– Model of how player metrics correspond to game configurations
• Game configurations = {(Game type) x (Environment) x (Game Settings)}
– Model the decision of which game to run next as a Markov Decision Process (MDP)
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 21
Game Scheduler –
Markov Decision Process
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 22
pn-1
pn
• Allows us to reason about decisions when outcomes are uncertain
Metric: Standing
in narrow spaces
?
p1
p2
Game choices
N1 samples
N2 samples
Nm samples
Nn-1 samples
…..
Game Scheduler –
MDP Design
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 23
• States si: – Numbers of samples we have so far for each metric
Game Scheduler –
MDP Design
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 24
• Actions gk: – Game configuration to run
– configurations = {(type) x (settings) x (environment)}
Game Scheduler –
MDP Design
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 25
• Transition probabilities Pk(si, sj) – Probability of receiving a set of additional samples if we run game gk
)|(),( ,
1
mk
b
mm
M
m
a
mkjik pPssP
Game Scheduler –
MDP Design
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 26
• Reward function R(si, sj) – Measure of how accurate our metric estimates are in sj
])()([),(1
M
m
mmmji pCIpCIssR
Game Scheduler –
MDP Design
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 27
• Solving this MDP gives our scheduling policy – States are represented in a tree structure
• Solving policy is linear in number of states
– To keep the numbers of transitions and states tractable, we segment counts into bins
– Depth of tree determines how far into the future we look for scheduling
Proof of Concept -
Mini-games in Second Life
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 28
• Scavenger Hunts
• Configurable environments
• Collected 5 metrics
Proof of Concept -
Second Life
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 29
Proof of Concept –
Scavenger Hunts
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 30
Proof of Concept – Metrics
Standing distance (Wide)
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 31
Proof of Concept – Metrics
Standing distance (Narrow)
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 32
Proof of Concept – Metrics
Crossing timing (slow traffic)
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 33
Proof of Concept – Metrics
Crossing timing (fast traffic)
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 34
Proof of Concept – Metrics
Health kit usage
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 35
Proof of Concept -
Environments
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 36
Office Space Alien Teleportation Facility
Game configurations
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 37
Experiments –
Data Collection
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 38
• Game activities were advertised through the Second Life event calendar
• 5 weeks
• 70 games
• ~23 hours online recording
• 179 participants – 80 unique based on Avatar ID
Experiments –
Participants
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 39
• 39% male
• 61% female
• Ages – 18-25: 53%
– 25-45: 29%
– 45-65: 16%
– 65+: 2%
• Residence – 75% US
Experiments –
Effect of Scheduling
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 40
Comparison of a policy against a brute force schedule
TF8
TF12 HK HKW
HKN HWF HNF T TW TN TWF TNF TF HF
TF12 TF8
TF12 TF8 HF HF HF HF TF8
TF12 HF
HK
HK H
TF8 - Slow Traffic
TF12 – Fast Traffic
HK – Kit
HKW – Health Kit (Wide Only)
HKN – Health Kit (Narrow Only)
HF – Health Kit (Limited Furniture)
HWF – Health Kit (Limited Furniture, Wide)
HNF – Health Kit (Limited Furniture, Narrow)
T – Tokens
TW – Tokens (Wide Only)
TN – Tokens (Narrow Only)
TF – Tokens (Limited Furniture)
TWF – Tokens (Limited Furniture, Wide)
TNF – Tokens (Limited Furniture, Narrow)
Brute Force vs. GAMELAB schedule (14 Games)
Brute Force
Experiments –
Effect of Scheduling
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 41
Comparison of a policy against a brute force schedule
TF8
TF12 HK HKW
HKN HWF HNF T TW TN TWF TNF TF HF
TF12 TF8
TF12 TF8 HF HF HF HF TF8
TF12 HF
HK
HK H
Brute Force vs. GAMELAB schedule (14 Games)
Brute Force
Both schedules were
initialized with the same
set of 14 experiments
Experiments –
Effect of Scheduling
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 42
Comparison of a policy against a brute force schedule
TF8
TF12 HK HKW
HKN HWF HNF T TW TN TWF TNF TF HF
TF12 TF8
TF12 TF8 HF HF HF HF TF8
TF12 HF
HK
HK H
The policy outperforms
the schedule
Brute Force
Brute Force vs. GAMELAB schedule (14 Games)
Discussion
• Potential for saving time and money
– Brute force schedule would need to run more games
for same results
• In this setup, many games did not contribute
useful samples
– Removing these -> brute force is just as good
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 43
Discussion
• Caveats
– Scheduling design not worth it when it’s simpler to
just run more games
– Scheduling not worth it when all game configurations
provide samples for all metrics equally
• Prototype shows potential for
– reducing cost and collection time through scheduling
– manipulating game configurations to target data
needs
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 44
Discussion
Scheduler automatically reasons about
differences between game configurations
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 45
Final Thoughts
• Modeling relationship between player metrics and game
configurations can by itself yield insights into player
behaviors
• Using Second Life
– Interesting mix of people
– Varied behaviors
• open ended
• noisy data
– Not a game, expectations of participants differ
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 46
Future Work
• More sophisticated examples
• More analysis
• Apply method outside of playtesting contexts
– Decide where/when to record
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 47
Game-based Data
Capture for Player
Metrics
Aline Normoyle,
John Drake
University of Pennsylvania
Maxim Likhachev
Carnegie Mellon University
Alla Safonova
Disney Research, Pittsburgh
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 48
Questions?
Special thanks to Benedict Brown, Ben Sunshine-Hill, Alex Shoulson, the
anonymous reviewers, and NSF Grant IIS-1018486.
Proof of Concept –
Blocking rooms example
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 49
Experiments –
Data Collection
Aline Normoyle Game-based Capture of Player Metrics October 12, 2012 50