ChinaNetCloud - Cloud Operations for Gaming - Tencent July 2014

Running the World’s Internet Servers www.ChinaNetCloud.com

By Steve Mushero

July, 2014


Introduction

Clouds

Game Operations

The Future


About Games

Games bigger and faster

Launch faster

Update faster

Grow faster

Flexibility is key

Performance important

Cloud technology key

DevOps also key


History Ancient Times

Big Metal – Big mainframes

OXO on EDSAC in 1952

SpaceWar! on PDP1 in 1961


History Ancient Times

How to update ?

Punch Tape

Punch Cards

Magnetic Tape


History Recent Events

Modern Games on Linux Servers

Mostly on physical servers

Buy them in boxes

Put in racks

Plug in cables

But in DVD, boot

Configure by hand

Install code

Run game


Hard to manage


History Recent Events

ISO & KickStart

PXE boot

Server cloning

Bash scripts

Version control

Check out to deploy

Slow & Inflexible


Current Situation

Gradually Automated

Simple Private Clouds

Simple Public Clouds

Version Control

Some Orchestration Servers

Limited Config Management

Diverse environments

Lots of scripts & glue


Still hard to manage


Today’s Ops Problems

Getting servers

Getting servers installed

Getting services installed

Configuring everything

Sync’ing Dev/Test/Prod

Scaling, even manually

Monitoring & Troubleshooting

Security, Performance Reliability


View of the Future

We build cloud systems that combines all

State-of-the-art fully-automated DevOps

A dozen different services Web, logic, security, database, and moreAll connected with queues

One button full system creation

One button full deployment, updates

Any time, any environment


View of the Future

How do we do this ?

Systems we build now

-State-of-the-Art Tech

-Cutting-Edge DevOps


View of the Future

Most of you have heard of DevOpsYou may be doing some of that

A lot of you know about Config ManagementPuppet, Chef, and Ansible

Some of you may have heard of DockerHottest mixed DevOps & Cloud technology

We use them all


View of the Future

How do you build and operate this ?

Carefully - Uses every state-of-the-art tech

Lots of moving parts

Lots of documentation to learn

Lots of rules to follow


Where we just did this

On a big advanced public cloud

Using a variety of tools

But can be any cloudPublic or Private

Or even physical servers

BUT, many cloud features helpAPI, Images, etc.


Phases

Two separate phases:

#1 – Provisioning VMs & InfrastructureBuilding the ‘hardware’ & ‘OS’

#2 – Deployment of code, configs, etc.Pushing code, config, making it run

Each has different thinking, tools, processes


Phases






Provisioning & Infrastructure

Goal is to automate all of this:

Server creation

OS install & config

Network & Firewall setup

Maybe include servicesNginx, MySQL, Tomcat, etc.

Lots of small detailsIP, routes, security, etc.



Parts:

Cloud System

Environment

System Definition

Config Management

Docker



Cloud

Production Environment

Web Server

Game Server

Database

Docker

Containers

Recipe

Chef


Cloud Part

Need to dynamically create VMs

Public or Private is okay

Cloud APICreate, start, stop, config VMs

VM Images helpful for cloning, speedBut can update during auto setup

Some things can be manuallySetup networks, firewalls, load balancers

IP address management very importantStatic, hosts files, DNS, JSON


Environment Definition

Real systems have Dev/Test/Prod…

Need way to define environmentCode must be standardized

To support Dev/Test/Stage/ProdConfigs, IPs, user/pw, other differences

This is NOT easy

Dev Test Stage Prod


System & Layer Definitions

Define the system architecture

Ideally in LayersHelps structure thinkingAll VMs same, e.g. Web, Game, DB

Define layer propertiesEspecially provisioning processAlso recipes, packages

Define events & lifecycle in the layerHooks to connect into everything

Load Balance

Web Server

Game Engine

Database


Configuration Management System

To do the workAll the hard parts

Such as Puppet, Chef, AnsibleThis system in Chef, we also use Ansible

Need management serverHas config, recipes, packages

VM/Nodes have agentsDo the work, pull packages, run scripts

All non-code must be packagesRPMs, etc. - Not from source

Re-use standards where possible


Configuration Management System

Think about securityThis system can build or destroy your gameEasy to make mistakes

Carefully structuredLayers, phases, lifecycle eventsProvisioning, deploying, etc.System setup, config/update, deploy,

shutdown life cycles


Management Console

Place to control it all

Tools like Jenkins

CLI tools via ssh

Fully automatic

27


Docker

Hottest DevOps technologyLinux LXC, like a mini-VM, fast, lightweight

Everything in a containerAll paths and dependencies standardized

Also a packaging systemOS, services, configs, code, dataAll in a containerIncludes Repository

Includes update systemSends diffs via layers

28


Docker

Complex to setup for real useEspecially networking

People starting to deploy on cloudsVia DevOps

Decide abstraction levelOf Cloud, Config System, Code, DockerNot obvious


Phases






Code / Deployment Level

Prepare for this

Standardized with dependenciesvery carefully managed

Especially across environments

Paths, libraries, versionAll standardized (especially paths)

Testing dependenciesLike DNS, 3rd party, prod servicesThink how to do this, by environ/layer


Other Elements

Configurations are ImportantIn DevOps, configs also code

Including OS, Nginx, DB, LB, etc.

Version control requiredDeployment pulls from SVN/Git

Dependency injectionConfigure per environment, custom

Often JSON/config-driven

Deployment process criticalDeployment process and recipes

Ensure partial deploy and undeploy, revert versions

Rolling - Deploy without downtimeWatch dependencies like DB, services, API


How it Deploys Code

Push New

Code

To SVN

Trigger

Deploy

Deploy

Event

VM

Recipe

Gets New

Container

Stops Old

Starts New

Container


All the pieces together

Docker

Repos

Container

Builder

Version

ControlCloud VMs

Load Bal

Container

Web Server

Container

Cloud VMs

User Auth

Container

Game

Container

Cloud VMs

MySQL

Container

MongoDB

Container

Cloud VMs

Game

Container

Msg Queue

Container

Chef

Engine

Agent

Agent

Agent

Agent

Developer

Code

Check-in



Benefits

Fast and Flexible

Auto-provision

Auto-deploy

Auto-scaling

Self-healing

Multi-environment Dev/Test/Stage/ProdAll exactly the same


Pioneers in OaaS – Operations as a Service

Thanks from ChinaNetCloud


Contact Us

[email protected]

+86 (21) 6422-1946

www.ChinaNetCloud.com


ChinaNetCloud Technology

(Shanghai) Company Limited

[email protected]

www.ChinaNetCloud.com

Shanghai Headquarters:

X2 Space 1-601, 1238 Xietu Lu

Shanghai, 200032 China

T: +86-21-6422-1946 F: +86-21-6422-4911

Beijing Office:

Lee World Business Building #305

57 Happiness Village Road, Chaoyang District

Beijing, 100027 China