Embed Size (px)
DESCRIPTION
This is a deep-dive straight into the guts of running a low-latency multiplayer game, such as a first-person shooter, on a global scale. We dive into architectures that enable you to split apart your back-end APIs from your game servers, and Auto Scale them independently. See how to run game servers in multiple AWS regions such as China and Frankfurt, and integrate them with your central game stack. We'll even demo this in action, using AWS CloudFormation and Chef to deploy Unreal Engine game servers. In the second half, hear from PlayFab, who built the backend for the Top-10 free-to-play PC shooter Loadout. PlayFab reveals details about their architecture, including AWS Elastic Beanstalk setup, Amazon DynamoDB and Amazon RDS patterns, data sharding, and use of multiple Availability Zones. Finally, PlayFab highlights challenges they faced when deploying to AWS China, and how they solved them.
Citation preview
November 14, 2014 | Las Vegas, NV
Nate Wiger, Principal Gaming Solutions Architect, AWS
James Gwertzman, CEO, PlayFab
[Game] Online Problems Continue; Features
Disabled as [Game Studio] Works on Fixes
“We are working hard to get everyone online
as soon as possible,” [Game Studio] says.
Days later, [Game Studio] leaving
disconnected players stranded
[Game] has a great online mode – if you
can get online, that is
“This is total bull**** I paid for a preorder and
I can’t even connect WTF [Game Studio]??”
http://bit.ly/verizon-latencyhttp://bit.ly/superdata-latency
Tokyo
Oregon
Frankfurt
Virginia
VPC Subnet
VPC Subnet
Availability Zone
A
Availability Zone B
VPC Subnet
VPC Subnet
Auto Scaling group
WEB
VPC Subnet
WEB
JOBS
Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
EC2
EC2
EC2
Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
④ Connect to Server
⑤ Hack Apart Your Friends
⑥ Game Over EC2
EC2
Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
④ Connect to Server
⑤ Hack Apart Your Friends
⑥ Game Over
⑦ Write via HTTP APIEC2
EC2
VPC Private Subnet
VPC Public Subnet
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
http://bit.ly/ec2-enhanced-net
http://bit.ly/linux-
tuning
VPC Private Subnet VPC Private Subnet
Availability Zone A Availability Zone B
10.1.0.13 10.2.0.16
TCP 5672
rabbitmq-node1 rabbitmq-node2
TCP 4369
& 25672
https://www.rabbitmq.com/clustering.html
https://www.rabbitmq.com/ha.html
5672
rabbitmq-node1
rabbitmq-node2
VPC Private Subnet
VPC Public Subnet
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
Auto Scaling group
VPC Private Subnet
VPC Public Subnet
Availability Zone A
GAME GAME GAME
Auto Scaling group
UnrealServer
unreal_linux.template.json
true
my-bucket-name
Unreal-123.zip
sshkeyname
1.2.3.4/24
MyPass123
Tokyo
Oregon
Frankfurt
Virginia
?
?
VPC Subnet
Availability Zone A Availability Zone B
VPC Subnet
Auto Scaling group
WEB WEB
Oregon
Tokyo
VPC SubnetJOBS
VPC Private Subnet
VPC Public Subnet
Availability Zone A
S3 Bucket
Amazon CloudWatch Logs
GAME GAME GAME
VPC Private Subnet
VPC Public Subnet
Availability Zone A
Amazon
Kinesis
S3 Bucket
Amazon Redshift
GAME GAME GAME
China
Oregon
Frankfurt
AuthChina
Leaderboard
+
Tools for operating &
monetizing live games
Complete set of cloud-based
backend services
Eco-system of pre-integrated
third-party services
.. and
more
+
AWS cloud: PlayFab Web Services
US EU
Hardware
Game Servers
Hardware
Game ServersGame
Client
Amazon Route 53
(3.playfabapi.com)
Amazon EC2
(API handling)Matchmaker
Instance Instance
Game Server
Monitor
DynamoDB Amazon RDS Amazon S3
Elastic Load Balancing
Amazon RedshiftReports
service
Instance
Logs
Matchmaker
(Secondary)
InstanceUS EU
SA AUS
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Client ServerAuthentication Title-Wide Data Management Shared Group Data Authentication
AddUsernamePassword GetCatalogItems AddSharedGroupMembers AuthenticateSessionTicket
LoginWithAndroidDeviceID GetStoreItems CreateSharedGroup Account Management
LoginWithFacebook GetTitleData GetSharedGroupData GetUserAccountInfo
LoginWithGameCenter GetTitleNews RemoveSharedGroupMembers SendPushNotification
LoginWithGoogleAccount AddUserVirtualCurrency UpdateSharedGroupData Player Data Management
LoginWithIOSDeviceID Player Item Management Server-Side Game Logic GetLeaderboard
LoginWithPlayFab ConsumeItem GetLogicServerUrl GetLeaderboardAroundUser
LoginWithSteam GetUserInventory GetUserData
RegisterPlayFabUser RedeemCoupon GetUserInternalData
SendAccountRecoveryEmail SubtractUserVirtualCurrency GetUserReadOnlyData
Account Management UnlockContainerItem GetUserStatistics
GetAccountInfo StartPurchase UpdateUserData
GetPlayFabIDsFromFacebookIDs PayForPurchase UpdateUserInternalData
GetUserCombinedInfo ConfirmPurchase UpdateUserReadOnlyData
LinkFacebookAccount PurchaseItem UpdateUserStatistics
LinkGameCenterAccount Friend List Management Title-Wide Data Management
LinkSteamAccount AddFriend GetCatalogItems
UnlinkFacebookAccount GetFriendsList GetTitleData
UnlinkGameCenterAccount RemoveFriend SetTitleData
UnlinkSteamAccount SetFriendTags Player Item Management
UpdateEmailAddress IOS-Specific APIs AddUserVirtualCurrency
UpdatePassword RegisterForIOSPushNotification GetUserInventory
UpdateUserTitleDisplayName ValidateIOSReceipt GrantItemsToUsers
Player Data Management Matchmaking APIs SubtractUserVirtualCurrency
GetFriendLeaderboard GetCurrentGames Matchmaking APIs
GetLeaderboard GetGameServerRegions NotifyMatchmakerPlayerLeft
GetLeaderboardAroundCurrentUser Matchmake RedeemMatchmakerTicket
GetUserData StartGame Steam-Specific APIs
GetUserReadOnlyData Android-Specific APIs AwardSteamAchievement
GetUserStatistics AndroidDevicePushNotificationRegistration
UpdateUserData ValidateGooglePlayPurchase
UpdateUserStatistics Analytics
LogEvent
API Request: Response from Web Services:
Game ClientAmazon Route 53
(3.playfabapi.com)
API Request:
Elastic Load
Balancing
Logs
Amazon RedshiftAmazon S3
bucket
Reports
service
LogsInstance
Logs include title-specific
endpoint for analytics
AWS wildcard DNS
(*.playfabapi.com)
Log event:
API NAMETOTAL
CALLS
AVERAGE
BACKEND
TIME (S)
AVERAGE
RESPONSE
TIME (MS)
AVERAGE
REQUEST
TIME (MS)
TOTAL
REQUEST
TIME (M)
AVERAGE
RECIEVED
(BYTES)
AVERAGE
SENT
(BYTES)
TOTAL
ERRORS
adminapi_userinfo 93,081,685 0.03 0.04 0.04 59.83 0 452 1979
gameserver_usercustomdatareadonly 1,582,348 0.06 0.04 0.04 1.06 42 902 290
matchmaker_userinfo 1,138,842 0.20 0.04 0.04 0.81 0 10046 319
matchmaker_playerjoined 1,138,247 0.01 0.04 0.04 0.79 64 2 93
matchmaker_playerleft 1,136,797 0.01 0.04 0.04 0.80 64 2 400
matchmaker_authuser 736,607 0.01 0.03 0.04 0.52 0 48 107
gameserver_usercustomdata 297,735 0.19 0.04 0.04 0.20 0 4080 0
gameserver_usercustomdatainternal 297,469 0.08 0.04 0.04 0.20 0 237 2
InstanceHealth
Check
Auto Scaling group
Auto Scaling
service
Instance
Matchmaker
(Primary)
Instance
Amazon Route 53
(matchmaker.playfab
api.com)
Elastic Load
Balancing
Matchmaker
(Secondary)
1.Auto Scaling is set to keep
two instances up.
2.Elastic Beanstalk Health
Check terminates bad
instance.
3.Amazon Route 53 fails
over all traffic to secondary
server.
4.Auto Scaling detects
failure and starts new
instance.
Amazon EC2
(API handling)
Elastic Load
Balancing CloudWatch
Health
Check
Email AlertsOn-call Engineer
Mobile Alerts
Amazon EC2 API Server
Server SSD
Amazon Redshift
Local Reports
service Local event
logsInstance
Events:
logins and
purchasesReports
service
Logs
Customer’s game
management dashboard
Analytics
Server
Logs
RabbitMQ
message
queue
LogsInstance
Logstash
service
Logstash
Instance
ElasticSearch
Instance
Instance
DynamoDB Amazon RDS Amazon S3
Player data:
• Accounts
• Profile
• Inventory
• Save data
• Player stats
• Player currency
values
• Auditing
• Log files
• Game replays
Amazon Redshift
• Analytics data
Amazon
DynamoDB
• Player data: accounts, profile, inventory, save data
User:
UserID: Hash
TitleID: range key
- DisplayName
- Linked accounts
- …
Custom user data:
UserID: Hash
TitleID: range key
- Key/value 1
- Key/value 2
- …
Inventory:
UserID: Hash
ItemID: range key
- titleID
- Catalog Item
- …
…
AZ 1
AZ 2
PlayFab Web
Services
PlayFab Web
Services
N. California
AZ 1
AZ 2
PlayFab Web
Services
PlayFab Web
Services
Oregon
AZ 1
PlayFab Web
Services
Beijing
• Original deployment: us-west-1
• Expanded to us-west-2 in July
• Coming soon: cn-north-1
Amazon EC2
Game Server
Singapore
Game Client
Instance
Player accounts are shared
across all regions:• Cross-regional wrapper for
user.get sets order of
databases and regions to
check
• user.get then checks for
player account data
• High availability, transparent
to customers and players
// Returns the first matching player record, searching DynamoDB and SimpleDB
// in both regions in the specified order.
// Updates the player record in the store and region from which it was loaded.
// Creates a player record in the default store and region.
AWS cloud: PlayFab Web Services
US EU
Hardware
Game Servers
Hardware
Game ServersGame
Client
Amazon Route 53
(3.playfabapi.com)
Amazon EC2
(API handling)Matchmaker
Instance Instance
Game Server
Monitor
DynamoDB Amazon RDS Amazon S3
Elastic Load Balancing
Amazon RedshiftReports
service
Instance
Logs
Matchmaker
(Secondary)
InstanceUS EU
SA AUS
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers
