NetflixOSS meetup lightning talks and roadmap

Lightning talksSeason 1, Episode 2

March 13, 2013

● Lightning talks: 7:00 - 7:20

● NeflixOSS Roadmap: 7:20 - 7:30

● Announcement: 7:30 - 7:45

● Demo Stations and Q+A: 8:00 - 9:30

Overview

● The nucleus of a Composable Web Service.

● Container Agnostic with extensions for frameworks/containers.

Karyon https://github.com/Netflix/karyon

● Bootstrapping○ Dependency & Lifecycle management via

Governator.○ Service registry as Eureka.○ Property management via Archaius.

Karyon https://github.com/Netflix/karyon

● Administration (Auto-Discovered Embedded console)○ Environment○ Eureka○ JMX

Karyon https://github.com/Netflix/karyon

Runtime (Coming Soon)● SLA● Throttling● Request tracing and more ....

Karyon https://github.com/Netflix/karyon

● Sample Service using Karyon available as "Hello-netflix-oss" on github

● Recipes ...

Karyon https://github.com/Netflix/karyon

Netflix OSS Recipe: RSS ReaderFirst sample application that demonstrates how to build scalable edge and middle tier application using the Netflix OSS components.

Accelerate your learning curve !

Karyon

Hystrix

Servo Turbine

Eureka

Ribbon

Archaius

First sample application that demonstrates how to build scalable edge and middle tier application using the Netflix OSS components.

Accelerate your learning curve !

Netflix OSS Recipe: RSS Reader

https://github.com/Netflix/recipes-rss

Netflix OSS Recipe: RSS Reader

● Ephemeral Volatile Cache

○ Based on memcached/spymemcached

● Highly Scalable ○ Over 2 MM RPS at peak within Netflix○ Over 100 K RPS / instance

● Fast○ Average latency ~ 1 ms

● Elastic○ Dynamically add/remove clusters, zones and/or instances

● Heavily Used @Netflix○ Fronts services & stores (Cassandra, S3, SimpleDB, RDS)

● AWS Zone Aware○ Clusters by zone○ Replication by zones ○ Eliminate cross zone traffic

● Connection Pool○ Separate pools for Read and Write Operations

● Zone fall through

● Resilient to zone outage

● Upcoming OSS○ EVCache Server, Admin and Monitoring

https://github.com/Netflix/EVCache

Denominator

● Portable Control of DNS clouds○ DynECT, Route53, UltraDNS and mock providers

● Java Api and CLI○ idempotent commands against record sets

● GSLB Functionality in progress○ Load Balanced and Directional (GEO) record sets

github.com/Netflix/denominator

Aminator

Aminator

● Evolved from Bakery

● AMI creation tool○ CLI○ EBS AMIs○ Redhat/Debian OSs○ Python 2.6 & 2.7○ boto

● What● Why● How● When● Who

Aminator

● Experiment with other distros at scale, like Ubuntu

● Remove embedded customizations

● You asked for it!

● What● Why● How● When● Who

Aminator

● Customize existing AMI○ Create volume from base AMI

snapshot○ Attach volume○ provision package○ snapshot volume○ register snapshot

● What● Why● How● When● Who

Aminator

● Coming soon, really.● What● Why● How● When● Who

github.com/Netflix/aminator

Aminator

● Brian Moyles○ @bmoyles

● Karate Vick○ @kvick

● Michael Tripoli○ @mtripoli

● What● Why● How● When● Who

NetflixGraph

● Compact in-memory data structure used to represent generic data sets.

● Reduce the size of your application's memory footprint○ Potentially by an order of magnitude

● Remove I/O bottlenecks by holding data in RAM

● Simple, easy to use API● Built-in serialization helps to transmit your

data sets across your infrastructure

NetflixGraph - Memory Reduction

● Drastically reduce the size of your data sets in memory○ Eliminate Access Latency - Pull more into RAM○ Reduce Cost - Use Less Hardware○ Stabilize your application

NetflixGraph - Memory/CPU tradeoff

● Easily select and test your memory-CPU tradeoff for different parts of your data sets.

● Choose between tighter memory or faster access times on a per-type basis.○ Mix and match, or just use the defaults!

NetflixGraph - Memory/CPU tradeoff

● Tighter memory requirements? Use compact sets.○ Variable-byte integers encode connections between

nodes

Wikipedia: Variable-Length Quantity

NetflixGraph - Memory/CPU tradeoff

● Faster access times required? Use hashed sets!○ Still RAM-efficient, provides O(1) access for contains() operations.

NetflixGraph - Built-in Serialization

● Transmit data sets across your infrastructure● Build the data set on one server, then distribute.

http://netflix.github.com/netflix-graph

Netflix OSS Continuous Integration

● Make current state of code visible to all● Build & test on public Jenkins, hosted by our

friends at CloudBees ● Each push or merge to master is built and

status shown on the GitHub project page

Netflix OSS Continuous Integration

Netflix OSS Continuous Integration

● Build & test all pull requests● Isolated environment to protect against

malicious code● Jenkins adds a comment to the pull request

Netflix OSS Continuous Integration● One-click creation of build jobs for new

projects

https://netflixoss.ci.cloudbees.comhttp://bit.ly/jobdsl

…