Gamers do REST - EuroPython 2014

GAMERS DO RESTNOT REALLY

(these are real graphs, trust me)

(these are real graphs, trust me)

ABOUT MEAngel Ramboi

programmer, gamer, geek @demonware

ABOUT DEMONWAREACTIVISION-BLIZZARD

Dublin

Vancouver

Shanghai

WHAT DO WE DO?We enable gamers to find one another

and then shoot each other in the face.

WHAT WE ACTUALLY DOleaderboardsmatchmakinganti-cheataccounts managementand more …

70+ SERVICES

WE HIRE SUPERHEROES

demonware.net/jobs

TALK OVERVIEWAPI design

Process and tools

App configuration

Handling errors/logging/metrics

Authentication/authorization

WHY RESTinteroperability

scalability

API DESIGNfollowing the REST principles outlined in Roy Fielding's thesis

GET, POST, PUT, DELETE verbs for API CRUD

HTTP as the communication protocol

JSON representation

pragmatic approach: “good enough” > perfect

API DESIGNGET /v1.0/users/1/ HTTP/1.1Accept: application/json

{ "userName": "cmac1", "email": "cmacleod@highlander.org", "firstName" : "Connor", "lastName" : "MacLeod", "dateOfBirth": "1518-03-18", "country": "GB", "gender": "male", "created": "1986-03-07T13:37:00Z", "quotes": { "href": "/v1.0/users/1/quotes/", "summary": { "quotes": [ "There can be only one.", "I am Connor MacLeod of the Clan MacLeod." ] } }}

PROCESS AND TOOLS

DEVELOPING WITH AGILITYScrum or Kanban

Continuous integration

TECH STACKPython 2.7

Django 1.6

MySQL 5.6

CentOS 6

Apache + mod_wsgi

CODE AND DEPLOYMENT

SCHEMA MIGRATIONS

SCHEMA MIGRATIONSPercona Toolkit to the rescue with “online-shema-change”

www.percona.com/software/percona-toolkit

creates an already altered tablesetup triggers for insert/update/deletecopies the datarenames the tables

Downside: uses a lot of space as it duplicates all the data

APP CONFIGURATIONYAML 1.2---django: DEBUG: False ALLOWED_HOSTS: ["*"] TIME_ZONE: UTC LANGUAGE_CODE: en-us USE_I18N: True SECRET_KEY: "secret key :)" TEMPLATE_LOADERS: - django.template.loaders.filesystem.Loader INSTALLED_APPS: - django.contrib.contenttypes ...

Cross project & Validation

APP CONFIGURATIONminimum_age = Option( type={ 'type': 'integer', 'valid': [['>=', 0]] }, default=13, description='The minimum age of a user.')

JSON VALIDATION{ "title": "Example Schema", "type": "object", "properties": { "username": { "type": "string", "pattern": "̂[a-z0-9_-]{3,15}$" }, "age": { "description": "Age in years", "type": "integer", "minimum": 0 } }}

json-schema.orgpypi.python.org/pypi/jsonschema

JSON VALIDATION{ "title": "Example Schema", "type": "object", "properties": { "firstName": { "type": "string", "minLength": 2, "maxLength": 100 }, "lastName": { "type": "string", "minLength": 2, "maxLength": 100 } }, "required": ["firstName", "lastName"]}

json-schema.orgpypi.python.org/pypi/jsonschema

JSON VALIDATION{ "title": "Example Schema", "type": "object", "properties": { "gender": { "type": "string", "enum": ["male", "female", "other"], "exceptions": { "required": errors.GenderMissingError, "type": errors.InvalidGenderError, "enum": errors.InvalidGenderError } } }, "required": ["gender"]}

json-schema.orgpypi.python.org/pypi/jsonschema

ERROR HANDLINGclass ErrorHandlingMiddleware(object): def process_exception(self, request, exception): return format_and_render_error( request, exception )

ERROR HANDLING{ "error": { "msg": "Request data validation failed, see context for more details.", "code": 227000, "name": "Error:ClientError:InvalidRequest:DataInvalid", "context": [ { "msg": "Email cmacleod@highlander.org already exists", "code": 288000, "name": "Error:ClientError:Conflict:EmailExists" }, { "msg": "Username cmac1 already exists", "code": 289000, "name": "Error:ClientError:Conflict:UsernameExists" } ] }}

LOGGING

LOGGING

LOGGING// Bad message - not suitable/useful for production.logger.debug("Variable x={}".format(var))

// Good message - suitable for production.logger.error( "Request {request} failed unexpectedly for reason {reason} " "resulting in client error {error}" .format({ "request": req, "reason": expl, "error": client_error_code }))

LOGGING2014-05-10T22:58:56.394565+00:00 level=error project=highlander app=usersview=get_UsersView client=127.0.0.1 method=GET path=/v1.0/users/2msg=Error:NotFound(No user with user_id 2 could be found.There can be only one!)

METRICS

METRICSclass MetricsMiddleware(object):

def process_request(self, request): request.metrics_start_time = time.time()

def process_response(self, request, response): if hasattr(request, 'metrics_start_time'): request_time = (time.time() - request.metrics_start_time) * 1000 metrics.write( name='request_time', value=request_time ) return response

AUTHWe use JSON Web Tokens:

JSON Web Signature (JWS) objectsJSON Web Encryption (JWE) objects

JOSE is a framework intended to provide a method to securelytransfer claims:

github.com/Demonware/josepypi.python.org/pypi/jose

% pip install jose

JSON WEB SIGNATURE (JWS) OBJECTSimport josefrom time import time

claims = { 'iss': 'http://www.example.com', 'exp': int(time()) + 3600, 'sub': 42}jwk = {'k': 'password'}

jws = jose.sign(claims, jwk, alg='HS256')

JWS(header='eyJhbGciOiAiSFMyNTYifQ', payload='eyJpc3MiOiAiaHR0cDovL3d3dy5leGFtcGxlLmNvbSIsICJzdWIiOiA0MiwgImV4cCI6IDEzOTU2NzQ0Mjd9', signature='WYApAiwiKd-eDClA1fg7XFrnfHzUTgrmdRQY4M19Vr8')

jwt = jose.serialize_compact(jws)

# on the client sidejose.verify(jose.deserialize_compact(jwt), jwk)

JSON WEB ENCRYPTION (JWE) OBJECTSimport josefrom time import timefrom Crypto.PublicKey import RSA

claims = { 'iss': 'http://www.example.com', 'exp': int(time()) + 3600, 'sub': 42}key = RSA.generate(2048)pub_jwk = {'k': key.publickey().exportKey('PEM')}

jwe = jose.encrypt(claims, pub_jwk)

JWE(header='eyJhbGciOiAiUlNBLU9BRVAiLCAiZW5jIjogIkExMjhDQkMtSFMyNTYifQ', cek='SsLgP2bNKYDYGzHvLYY7rsVEBHSms6_jW-WfglHqD9giJhWwrOwqLZOaoOycsf_EBJCkHq9-vbxRb7WiNdy_C9J0_RnRRBGII6z_G4bVb18bkbJMeZMV6vpUut_iuRWoct_weg_VZ3iR2xMbl-yE8Hnc63pAGJcIwngfZ3sMX8rBeni_koxCc88LhioP8zRQxNkoNpvw-kTCz0xv6SU_zL8p79_-_2zilVyMt76Pc7WV46iI3EWIvP6SG04sguaTzrDXCLp6ykLGaXB7NRFJ5PJ9Lmh5yinAJzCdWQ-4XKKkNPorSiVmRiRSQ4z0S2eo2LtvqJhXCrghKpBNgbtnJQ', iv='Awelp3ryBVpdFhRckQ-KKw', ciphertext='1MyZ-3nky1EFO4UgTB-9C2EHpYh1Z-ij0RbiuuMez70nIH7uqL9hlhskutO0oPjqdpmNc9glSmO9pheMH2DVag', tag='Xccck85XZMvG-fAJ6oDnAw')

jwt = jose.serialize_compact(jwe)

# on the client sidepriv_jwk = {'k': key.exportKey('PEM')}jwt = jose.decrypt(jose.deserialize_compact(jwt), priv_jwk)

SUMMARYRest is awesome

Be Pragmatic

Monitor Everything

We are hiring!!!

QUESTIONS?

aramboi@demonware.net@angelramboi