Art & music vs Google App Engine

  • View
    458

  • Download
    1

Embed Size (px)

Text of Art & music vs Google App Engine

  1. 1. PyCon6 Apr2015, Florence, Italy Art & Music VS AppEngine Thomas Alisi, Solution Architect @grudelsud @stinkdigital photo courtesy of https://www.flickr.com/photos/8064990@N08/
  2. 2. See our 2015 showreel Stinkdigital is an interactive production company, working with clients and advertising agencies worldwide. Our services include creative concepting, design and high-end execution. We create everything from live-action films and websites, through to mobile apps and installations.
  3. 3. Does this number look familiar to you? 86,400
  4. 4. Does this number look familiar to you? 86,400 = 60 x 60 x 24h[number of seconds in 1 day]
  5. 5. What about this one? 31,536,000
  6. 6. What about this one? 31,536,000 = 86,400 x 365d[number of seconds in 1 year]
  7. 7. OK, now try and guess the last one 252,000,000
  8. 8. OK, now try and guess the last one 252,000,000 = 31.5M x 8y[number of seconds in 8 years]
  9. 9. but also 252,000,000 = 36M x 7[7 seconds saved for each of the 36M visits we had during the first 6 months on DevArt]
  10. 10. How did we do it? 1. Google App Engine (GAE) and AngularJS 2. data structures 3. assets management 4. load testing 5. GAE benchmarking tool and task inspector 6. GAE asynchronous API
  11. 11. Revolutions in Sound - RedBull, Google+ http://archive.revolutionsinsound.com/ DevArt - Google CL https://devart.withgoogle.com/ Inside Abbey Road - Google CL https://insideabbeyroad.withgoogle.com/
  12. 12. 1. Google App Engine (GAE) and AngularJS 2. data structures 3. assets management 4. load testing 5. GAE benchmarking tool and task inspector 6. GAE asynchronous API
  13. 13. - Grunt, Gulp, Browserify, Webpack AAARGH! - we like vanilla JS and have been fan of Gulp (at least over the past 10 minutes) - we (I) tend to use a super simple Gulp-Browserify- AngularJS boilerplate https://github.com/grudelsud/angularjs-gulp-browserify- boilerplate - but there are other good examples too e.g. https://github.com/unit9/coee-bone divide et impera
  14. 14. - GAE is not opinionated (which is good) - default webapp2 shipped with GAE is really super simple (a bit too simple) - Flask is OK and does not have preferences for a specific ORM (which is great) - dont use Django on GAE unless you really want to (e.g. use legacy modules or deploy something really quick)
  15. 15. class Jsonifiable(ndb.Model): def from_dict(cls, dict): pass def to_dict(self, async=False): pass def resolve_future_blobs(cls, async_blobs): pass class JsonRestHandler(webapp2.RequestHandler): JSON_MIMETYPE = "application/json" def write(self, data): self.response.out.write(data) def send_success(self, obj=None, cache_expiry=None): self.response.headers["Content-Type"] = self.JSON_MIMETYPE self.write(json.dumps(obj, cls=JsonifiableEncoder)) webapp2 bespoke REST micro-framework asynchronous conversion bespoke encoder basic permission check (skipped here) extend ndb.Model extend webapp2.RequestHandler
  16. 16. kibble_authenticator = ModelAuthenticatior() admin = kibble.Kibble( 'admin', __name__, kibble_authenticator, label='Admin', static_url_path='/kibble/static', default_gcs_bucket=app_identity.get_default_gcs_bucket_name(), ) security_headers.apply(app) class CustomJsonEncoder(flask.json.JSONEncoder): def default(self, obj): try: if isinstance(obj, blobstore.BlobKey): return str(obj) if isinstance(obj, fields.Vertex): return [obj.x, obj.y, obj.z] if isinstance(obj, GCSFile): return obj.cloudstore_url if isinstance(obj, ndb.Key): return obj.flat() except TypeError: pass return super(CustomJsonEncoder, self).default(obj) app = flask.Flask(__name__, template_folder='../templates') app.json_encoder = CustomJsonEncoder admin.autodiscover(paths=['admin']) app.register_blueprint(admin, url_prefix='/admin') Flask Blueprints and GAE https://github.com/xlevus/ask-kibble
  17. 17. 1. Google App Engine (GAE) and AngularJS 2. data structures 3. assets management 4. load testing 5. GAE benchmarking tool and task inspector 6. GAE asynchronous API
  18. 18. this always seems a good way to start
  19. 19. class ClubNight(BaseModel): name = ndb.StringProperty() content = ContentProperty(ndb.TextProperty) blob_key_logo = ContentProperty(ndb.BlobKeyProperty) genre = ndb.KeyProperty(kind=Genre) website = ContentProperty(ndb.StringProperty) address = ContentProperty(ndb.TextProperty) location = ContentProperty(ndb.GeoPtProperty) class Connection(ndb.Model): from_key = ndb.KeyProperty() to_key = ndb.KeyProperty() abstraction
  20. 20. class ClubNight(BaseModel): name = ndb.StringProperty() content = ContentProperty(ndb.TextProperty) blob_key_logo = ContentProperty(ndb.BlobKeyProperty) genre = ndb.KeyProperty(kind=Genre) website = ContentProperty(ndb.StringProperty) address = ContentProperty(ndb.TextProperty) location = ContentProperty(ndb.GeoPtProperty) class Connection(ndb.Model): from_key = ndb.KeyProperty() to_key = ndb.KeyProperty() to_name = ndb.StringProperty() to_kind = ndb.StringProperty() to_slug = ndb.StringProperty() to_genre_colour = ndb.StringProperty() to_image = ndb.BlobKeyProperty() to_popularity = ndb.IntegerProperty() is_published = ndb.BooleanProperty() is_public = ndb.BooleanProperty(default=False) is_featured = ndb.BooleanProperty(default=False) reality
  21. 21. class Vertex(namedtuple('Vertex', ['x', 'y', 'z'])): def __json__(self): return json.dumps(self) def __str__(self): return 'x={0.x:.2f}, y={0.y:.2f}, z={0.z:.2f}'.format(self) class VertexProperty(ndb.StringProperty): # Dummy XYZ field, used to quickly add validation through # admin.base.ModelConverter def _validate(self, value): if not isinstance(value, (tuple, Vertex)): raise TypeError("Expected Vertex got %r" % repr(value)) if len(value) != 3: raise TypeError("Expected 3-tuple/Vertex, got %r" % repr(value)) def _to_base_type(self, value): return "%s,%s,%s" % value def _from_base_type(self, value): if value: return Vertex(*[float(x) for x in value.split(",")]) return None finding the sweet spot
  22. 22. 1. Google App Engine (GAE) and AngularJS 2. data structures 3. assets management 4. load testing 5. GAE benchmarking tool and task inspector 6. GAE asynchronous API
  23. 23. The get_serving_url() method allows you to generate a stable, dedicated URL for serving web-suitable image thumbnails. You simply store a single copy of your original image in Blobstore, and then request a high-performance per-image URL. [https://cloud.google.com/appengine/docs/python/images/] ex. // Resize the image to 32 pixels (aspect-ratio preserved) http://your_app_id.appspot.com/randomStringImageId=s32 // Crop the image to 32 pixels http://your_app_id.appspot.com/randomStringImageId=s32-c Due diligence =s500 =s100
  24. 24. - success == sleepless_nights (not for me! :P) - PR scheduled on all major funnels: - Google homepage - Youtube homepage - all major blogs (the verge, wired, etc.) - London silicon roundabout - 1.2M unique visits, 10M views during the FIRST 48H si vis pacem, para bellum
  25. 25. from admin.publish_pipeline import Publish rec = urlmap.PublishRecord.new() p = Publish('admin.web.app', version=rec.key.id()) p.start() [] class Publish(Pipeline): """ Root Pipeline. Creates a :py:class:`common.models.urlmap.PublishRecord` for the pipeline, and spawns children. On completion, sets publish record to completed. """ def run(self, app, locales=None, version=None): rec = urlmap.PublishRecord.get_by_id(version) rec.pipeline_id = self.root_pipeline_id rec.put() urlmaps = yield URLMaps(version) with After(urlmaps): static = yield WriteStaticFiles.pipeline(app, version) sitemap = yield PublishSitemap(app, version) short_urls = yield CreateShortUrls.pipeline(app, version) with After(static, sitemap, short_urls): json = yield PublishJSON(app, version) with After(json): yield CleanShortUrls(app, version) yield Cleanup.pipeline() static publishing pipeline https://github.com/GoogleCloudPlatform/appengine-mapreduce/ create the pipeline and run process extend mapreduce pipeline spawn separate generators
  26. 26. 1. Google App Engine (GAE) and AngularJS 2. data structures 3. assets management 4. load testing 5. GAE benchmarking tool and task inspector 6. GAE asynchronous API
  27. 27. GAE docs are rubbish! :) i.e. read it, then forget it: https://cloud.google.com/appengine/articles/load_test 3rd party services are OK, but run your own if you can create a meaningful simulation of users behaviour hit it as hard as you can, but dont forget your wallet!
  28. 28. class ApiNav(TaskSet): @task(1) def api_global(self): self.client.get('/api/global?locale=%s' % langs[random.randint(0, len(langs)-1)], **kwargs) @task(1) def api_user(self): self.client.get('/api/user', **kwargs) @task(4) def api_gallery(self): self.client.get('/api/gallery?i=0&l=15', **kwargs) @task(8) def api_search(self): self.client.get('/api/gallery?i=0&l=15&q=%s' % terms[random.randint(0, len(terms)-1)], **kwargs) @task(6) def api_feeling_lucky(self): self.client.get('/api/page/feeling_lucky', **kwargs) @task(2) def api_big_gallery(self): self.client.get('/api/gallery?i=0&l=30', **kwargs) @task(2) def api_featured(self): self.client.get('/api/gallery?i=0&l=15&t=featured', **kwargs) class MyLocust(HttpLocust): host = 'https://sd-goog-devart.appspot.com' task_set = ApiNav min_wait = 5000 max_wait = 15000 locust init data gallery random search random project page categorized views
  29. 29. - approximately 5000 concurrent user hitting the backend API with a "casual navigation" simulation from dierent location (London, New York, AWS data centre in Ireland) - 85 running instances (class F2) at peak - no errors reported other than random https sockets timeout - average response times - < 2s for gallery content navigation - < 1s for singe project page navitation - < 3s for static contend