(GAM407) Quiplash: Multiscreen, Multiplayer Game for 10,000

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Mike Bilder & Evan Jacover, Jackbox Games

Peter Heinrich, Amazon

October 2015

GAM407

Quiplash: The Multiscreen, Multiplayer

Game for 3 to 10,000 Players

The New Fan Base

Foosball

Everyone loves

Popular worldwide, BUT

- Who’s your favorite player?

- Do you have a jersey? A hat?

- Do you watch on TV or radio?

It’s a pastime for players,

not fans

A fan base

Compare to a sport with

More Fun for the Players, PLUS

Diverse Revenue Opportunities

• Merchandise

• Broadcasting

• Events

Lower Costs

• User acquisition

• Content creation

It's just the start

Minimum viable product

What takes your game

to the next level?

How do you reach the

“new fan base”?

User

acquisition

Monetization

loop

Game

loop

Early adopters

The new fan base requires

Influential

Show the fun

Start the conversation

Influencers

Monetization

loop

User

acquisitionGame

loop

Influencer types

Content creators

The new fan base includes

Greater variety

Longer lasting

Content

creation

Influencers

Monetization

loop

User

acquisitionGame

loop

Content types

• Entertainment

• Education

• eSports

• Levels and add-ons

• Fan art, fiction

Alternative revenue

The new fan base enables

Physical goods

Experiences outside the

game

Alternative

revenue

Content

creation

Influencers

Monetization

loop

User

acquisitionGame

LOOP

Additional revenue and

builds your fan base

External engagement

A major benefit of generating

revenue outside of your app is the

opportunity to expand your fan

base

Alternative

revenue

Content

creation

Influencers

Monetization

loop

Support the influencers

Encourage content

creation

Create revenue

opportunities outside

the game

Game

loop

User

acquisition

A Brief History of Jackbox Games

Walter Schirra snuck it on the first

Gemini space mission in 1965

STREAMERS!

Our first game with an

audience feature!

Technology

Technology

• Controllers

• Server architecture

• Tuning and optimization

• Audience

• Scaling

Technology: Controllers

JackPad

JackPad problems

JackPad solutions

Technology: Servers

Architecture

Game host

Console application

instance

Architecture

Game Host

Console Application

Game Controller

Mobile-optimized

Website

Architecture

Game host

Console application

Game controller

Mobile-optimized

website

SetRoomBlob({json})SetCustomerBlob(userid, {json})SendMessageToRoomOwner({json})

Architecture

twizzler hoho churro

ABCDWDJOEVAN

AAAAQWOP

MMQEZOZB

RESTful

PORT 80

PORT 443

Socket.io

PORT 38202

HTTPS/WSS PORT 38203

Each Blobcast instance runs two servers

twizzler hoho churro

ABCDWDJOEVAN

AAAAQWOP

MMQEZOZB

blobcast.jackboxgames.com

Sort Value Score

1 hoho-blobcast-production.jackboxgames.com 308

2 twizzler-blobcast-production.jackboxgames.com 344

3 churro-blobcast-production.jackboxgames.com 391

Technology: Tuning and Optimizing

Choosing instances

• How many games and players can run on a single

instance?

• What is the most efficient instance type to use for our

application? (CPU bound? I/O bound?)

• Are four large servers more cost-efficient than one XXL

server?

Linux tuning

# /etc/sysctl.confnet.core.rmem_max = 67108864 net.core.wmem_max = 67108864 net.ipv4.tcp_rmem = 4096 87380 33554432net.ipv4.tcp_wmem = 4096 65536 33554432net.core.netdev_max_backlog = 30000

...

https://fasterdata.es.net/host-tuning/linux/

Real world simulation

1. Write a script that does exactly what a player does

a. Look for specific game states

b. Respond with appropriate messages

2. Run that script thousands of times

3. Monitor the servers

Bees with machine guns

“...[The Bees] are, more-or-less a distributed denial-of-

service attack in a fancy package…”

https://github.com/newsapps/beeswithmachineguns

https://github.com/jugglinmike/beeswithmachineguns (fork for shell commands)

./bees up -s 10 -g sec-group -k keypair --instance ami-xxxxxxx

./bees exec - "git pull && npm install“

./bees exec – "node server.js play quiplash -p 1000 -r QWOP"

Tuning: Instance types

0

20

40

60

80

100

120

1000 2000 3000 4000 5000 6000 7000

Worst case CPU

c3.large c4.large m4.large r3.large c4.xlarge

Tuning: Instance types

0

20

40

60

80

100

120

1000 2000 3000 4000 5000 6000 7000

Worst case CPU

c3.large c4.large c4.xlarge

Technology: Audience

Big audiences

• Big streamers get upwards of 30,000 live viewers

• We want to be able to have more people participating in

a game than a single server can handle

twizzler hoho churro

ABCD(Host)

ABCD ABCD

PUBLISHchan:room:ABCD

SUBSCRIBEchan:room:ABCD

Multinode rooms: Redis PUB/SUB

twizzler hoho churro

ABCD(Host)

ABCD ABCD

RPUSHchan:hoho:ABCDchan:churro:ABCD

100ms Loop

LPOPchan:hoho:ABCD

100ms Loop

LPOPchan:churro:ABCD

Multinode rooms: Redis queue polling

twizzler hoho churro

ABCD(Host)

ABCD ABCD

100 ms Loop

LPOPchan:ABCD

RPUSHchan:ABCD

RPUSHchan:ABCD

Multinode rooms: Redis queue polling

Big audiences

• The vast majority of our games are still eight or fewer

players

• Don’t add latency for regular players• Wait for over 100 players to join a room before they can join on other

servers

• No single bottleneck

twizzler hoho churro

ABCD(Host)

ABCD ABCD

100 ms Loop

LPOPchan:ABCD

RPUSHchan:ABCD

RPUSHchan:ABCD

Multinode rooms: No single bottleneck

Efficient voting

Create a new vote “session”• Redis set with a list of user IDs that have voted

• Redis hash with a list of voting options

key count

Cropdust my heart 0

Smells like love 0

vote:QWOP:useridsSET

vote:QWOP:votesHASH

Efficient voting

Create a new vote “session”• Redis set with a list of user IDs that have voted

• Redis hash with a list of voting options

Validate new voters• Check for user ID SISMEMBER O(1)

key count

Cropdust my heart 0

Smells like love 0

vote:QWOP:useridsSET

vote:QWOP:votesHASH

Efficient voting

Create a new vote “session”• Redis set with a list of user IDs that have voted

• Redis hash with a list of voting options

Validate new voters• Check for user ID SISMEMBER O(1)

Accept vote• Add user ID to voters list SADD O(1)

• Increment vote HINCRBY O(1)

56cdb7d5-9c7f-4723-a562-b07659e04e40

key count

Cropdust my heart 1

Smells like love 0

vote:QWOP:useridsSET

vote:QWOP:votesHASH

Efficient voting

Create a new vote “session”• Redis set with a list of user IDs that have voted

• Redis hash with a list of voting options

Validate new voters• Check for user ID SISMEMBER O(1)

Accept vote• Add user ID to voters list SADD O(1)

• Increment vote HINCRBY O(1)

Query vote • Get all elements of vote hash HGETALL O(N)

56cdb7d5-9c7f-4723-a562-b07659e04e40

key count

Cropdust my heart 1

Smells like love 0

vote:QWOP:useridsSET

vote:QWOP:votesHASH

Technology: Scaling

One-Click Scaling

AWS OpsWorks

Why not use OpsWorks autoscaling?

Blobcast autoscaling Cron job running every minute

Scale audience and games separately

• Keep 10,000 connection

capacity available

• Scaling can proceed slower

as needed

Audience Games

What’s next

• Leveraging regions for geographic speed

• Multicore support (Cluster Node.js)

• Use AWS Lambda for autoscaling

• Better analytics

We love Fire TV

Why Fire TV is our development platform

• They’re cheap

• They’re portable

• Tooling is easy

• It has a USB port

• We’re developing for it anyway

It’s a “dream product”

It’s been successful

All 4 of our products are in the Top

Grossing Games list

Twitch TV

Making a game for streamers

• Our marketing budget: $0

• Every stream is an ad for our game

• Building online multiplayer is hard, leveraging Twitch is

easy

• Chat is a secondary way to play

• It’s a new way of interacting with games

Add additional ways for the audience to

interact beyond voting

• Audience scoring and scoreboards

• Promote audience members to players

• Track and manipulate statistical information

• Audience member get their name on screen

Provide more tools for the host to

manage the game

• The ability to boot/ban agitators

• An option to filter terrible, racist text

• Private codes to join the game

Technical improvements

• More than 10,000 people

• Reduce latency

Crazy stuff

• What if you could instantly

get a T-shirt of a drawing

from Drawful?

Burger King’s Skeleton drawing by @chilliards

T-Shirt mock-up via Custom Ink

Next Steps

Thank you!For more information, connect with us:

http://jackboxgames.com

http://bit.ly/AmznAppDev

@PeterDotGames@JackboxGames

Drawing winners announced:

http://bit.ly/appstoregiveaway

Remember to complete

your evaluations!