Embed Size (px)
Citation preview
Behind the Scenes:Deploying a Low-Latency Multiplayer
Game Globally
James Gwertzman, PlayFab CEO
[Game] Online Problems Continue; Features Disabled as [Game Studio] Works on Fixes
“We are working hard to get everyone online as soon as possible,” [Game Studio] says.
Days later, [Game Studio] leaving disconnected players stranded
[Game] has a great online mode – if you can get online, that is
“This is total bull**** I paid for a preorder and I can’t even connect WTF [Game Studio]??”
“The single greatest predictor of success and of sustainable competitive advantage in this business is live game operations.”
-- Owen Mahoney, CEO Nexon
Build game
Launch
The old days…
Build game
Build backend
Launch
Build tools for ops team
Segment & target customers
Deploy servers
Business intelligence
Offers & Promotions
Update content
Host in-game events
Customer service
Nowadays…
Business intelligence
User Acquisition
+
Live ops tools and dashboards(mission control for the whole team)
Back-end services(cross-platform, one-stop shop)
+
Integration with other partners(building out the full ecosystem)
All-in-one platform to build, launch, and grow
and more ...
Some Loadout Stats
• PC launched 1/31/14
• PS4 launched 12/17/14
• Over 1 billion API Calls
• At Peak (Feb 2014):
– 300k DAU, 35K CCU
– 120 fixed game servers
– 110 EC2 game servers
Basic tenets of PlayFab architecture
• Put latency-sensitive game servers near players.• Everything else as shared services via RESTful Web APIs.• Data replication = bad.• Local caches = good.• No scheduled downtime.• Automatic scaling for load.• Backend changes should be invisible to developers.• Use buffering to smooth over spikes or failures.
Availability Zone B
Oregon
AWS cloud: PlayFab Web Services
Amazon Route 53
(3.playfabapi.com)
Amazon EC2 (API
handling)
Matchmaker
Instance
Instance
Game Server Monitor
DynamoDB Amazon RDS Amazon S3
Amazon Redshift
Reports service
Instance
Logs
Architecture Overview
Matchmaker (Secondary)
Instance
Virginia
Availability Zone A
Amazon EC2 (API
handling)
Elastic Load Balancing
Cross-zone storage
Game ClientGame Manager(Dashboards)
Amazon EC2(Virtual game
servers) Tokyo
Amazon EC2(Virtual game
servers) Sydney
Physical game
servers
Physical game
servers
Amazon EC2(Virtual game
servers)
PlayFab Client API(HTTPS)
PlayFabAdmin API
(HTTPS)Game Protocol
(TCP / UDP)
PlayFab Server API(HTTPS)
Local disk Local disk Local disk
Physical game
servers
13
PlayFab backend services
• Player Accounts– Authentication– Profile Management– Account Linking
• Data Storage– Per player– Per title– Per character (under one player)– Files / CDN delivery
• Commerce– Catalog Management– Virtual Currencies– Player Inventory– In-game Purchasing– Receipt validation (Apple/Google/Amazon)– Marketing and Promotion
• In-game Marketing– Push Notifications– In-game Messaging
• Product Management– Analytics and Reporting– Customer Segmentation– Customer Support Tools– Abuse Reporting and Banning
• Social– Friends Lists– Player Chat– Leaderboards– Game forum integration– Trading / gifting
• Multiplayer– Photon integration (real-time / turn-based)– Matchmaking– Custom game server hosting– Server monitoring
• Game logic– Server-hosted JavaScript
The last API you’ll have to integrate into your game…
Example of Web API Call
API Request: Response from Web Services:POST https://3.playfabapi.com/Client/MatchmakeContent-Type: application/json;X-Authentication: 7BC920BC255F7E60-0-0-5F4{ "BuildVersion": "5.01", "Region": "USCentral", "GameMode": "0", "LobbyId": "Lobby 32", "EnableQueue": false}
HTTP/1.1 200 OKContent-Type: application/json; charset=utf-8{ "LobbyID": "Lobby 32", "ServerHostname": "192.168.0.1", "ServerPort": 7777, "Ticket": "e98yf289f248902f4904f0924f9pj", "Status": "Waiting", "Queue": [ "User1", "User2" ]}
Oregon
AWS cloud: PlayFab Web Services
Matchmaker
Instance Instance
Game Server Monitor
Basic Gameplay (1)
1. Login via Web API to PlayFab2. Retrieve session key3. Download content updates4. Call backend services as needed
(inventory, catalog, player state)5. Use matchmaker to find server;
retrieve game ticket
Virginia
API servers
Game Client
Game servers
Storage Local diskAmazon Redshift
Reports service
Instance
Basic Gameplay (2)
6. Connect to game server7. Present ticket to authenticate8. Game server validates ticket9. Game server loads player profile10.Play w/ other players via server11.Game ends12.Server updates player profile13.Log files get written to local disk
Virginia
Game Client(other players)
Local disk
Oregon
AWS cloud: PlayFab Web Services
Matchmaker
Instance Instance
Game Server Monitor
API servers
Storage Amazon Redshift
Reports service
Instance
Game servers
Basic Gameplay (2)
14.Log files get transferred to S3 storage15.Server monitor cleans up old games;
can create new servers if needed16.Matchmaker prepares for new game17.Events get imported into Redshift
Virginia
Game Client(other players)
Local disk
Oregon
AWS cloud: PlayFab Web Services
Matchmaker
Instance Instance
Game Server Monitor
API servers
Storage Amazon Redshift
Reports service
Instance
Game servers
ELB logging for title-specific tracking
Game Client Amazon Route 53(3.playfabapi.com)
API Request:POST https://3.playfabapi.com/Client/LoginWithPlayFabContent-Type: application/json;{ "TitleId": “3", "Username": “gwertzman", "Password": “jamespassword"}
Elastic LoadBalancing
Logs
Amazon Redshift
Amazon S3 bucket
Reports service
LogsInstance
Logs include title-specific endpoint for analytics
AWS wildcard DNS (*.playfabapi.com)
Sample Loadout API usage report
API NAMETOTAL CALLS
AVERAGE BACKEND TIME (S)
AVERAGE RESPONSE TIME (MS)
AVERAGE REQUEST TIME (MS)
TOTAL REQUEST TIME (M)
AVERAGE RECIEVED (BYTES)
AVERAGE SENT
(BYTES)TOTAL
ERRORS
adminapi_userinfo 93,081,685 0.03 0.04 0.04 59.83 0 452 1979
gameserver_usercustomdatareadonly 1,582,348 0.06 0.04 0.04 1.06 42 902 290
matchmaker_userinfo 1,138,842 0.20 0.04 0.04 0.81 0 10046 319
matchmaker_playerjoined 1,138,247 0.01 0.04 0.04 0.79 64 2 93
matchmaker_playerleft 1,136,797 0.01 0.04 0.04 0.80 64 2 400
matchmaker_authuser 736,607 0.01 0.03 0.04 0.52 0 48 107
gameserver_usercustomdata 297,735 0.19 0.04 0.04 0.20 0 4080 0
gameserver_usercustomdatainternal 297,469 0.08 0.04 0.04 0.20 0 237 2
Instance
Making a Singleton Service Highly Available
Health Check
Auto Scaling group
Autoscaling service
Instance
Matchmaker (Primary)
Instance
Amazon Route 53(matchmaker.playf
abapi.com)
Elastic LoadBalancing
Matchmaker (Secondary)
1. Autoscaling is set to keep one instance up in each of 2 AZ’s.
2. Client locates matchmaker through Route 53
3. All matchmaker state written through to Dynamo DB
Storage
Game Client
AZ A AZ B
Instance
Making a Singleton Service Highly Available
Health Check
Auto Scaling group
Autoscaling service
Instance
Matchmaker (Primary)
Instance
Amazon Route 53(matchmaker.playf
abapi.com)
Elastic LoadBalancing
Matchmaker (Secondary)
4. Route53 Health Check detects bad instance and fails over all traffic to secondary.
5. Secondary loads latest state from storage
6. Autoscaling group detects bad instance and starts new one.
Storage
Game Client
AZ A AZ B
Monitoring Service Health
Amazon EC2 (API
handling)
Elastic Load Balancing CloudWatch
Health Check
Pingdom VictorOps
Email AlertsOn-call Engineer
Mobile Alerts
Datadog visualization
23
Trusted by hundreds of developers
• More than 500 game studios using the PlayFab SDK• 16 titles now live (w/ more than 10 million total registered players)• Integrate at any point of your game’s life cycle
PlayFab GameManager demo
25
Easy adoption by any game developer
Full set of support resources makes it easy for any developer to quickly start using the PlayFab platform.
Getting Started Guides SDKs for popular tools
Full documentation
+ +
+
Full-time developer relations team
PlayFab benefits
26Better Faster Cheaper
Developers
• Best-in-class technology• Proven solution• Fully documented• Back-end engineering talent• Continual improvements• Baked-in best practices• Learn from other games
• Available immediately• Less testing needed• No need to hire a new team
• No up-front engineering cost• Volume of scale• Amortized costs
Publishers
• Share tools across games• Share players across games• Less risk
• Training for new developers• Share lessons across games
• All funding goes to game• No tech support costs
26PlayFab confidential
PlayFab confidential 27
Pricing modelTest Free Standard Premium Enterprise
Test all PlayFab features risk-free
Build, test, and launch a game risk-free
Backend services for casual or mobile
games
Meet needs of demanding core or multiplayer games
Custom agreements for multi-title
publishers
FREE(max 1,000 DAU)
FREE(UNLIMITED DAU)
$2(per 1,000 users / day)
$6(per 1,000 users / day)
negotiable
All featuresare available with API throttling limits
• Live ops game manager• Login with Device ID• Receipt validation• Catalog (RM items only)• User inventory (RM
items only)• Basic reports
All of free tier plus:• Player accounts• Virtual goods• Standard analytics• Friends lists• Leaderboards • In-game messaging
All of standard tier plus:• Cloud scripting• Photon Realtime /
Turnbased / Unity Networking / Chat / Server
• Custom development• Game server hosting*
All of premium tier plus:• Private server clusters• Custom payment
provider• Custom authentication• Custom SLA
*Actual server cost (pass through) + 10% management fee.
