View
296
Download
3
Category
Tags:
Preview:
DESCRIPTION
In this talk I’d like to introduce the Scala-based API stack at Partner Innovation Group at Netflix. After seeing a massive growth in the business model and the device ecosystem, we needed a system that could scale and be flexible at the same time. Scala provided the answer and we started with a basic set of APIs which, since then, has evolved towards complex but flexible business flows. Supporting metadata for over hundreds of brands and thousands of devices, the API development has followed a well thought-out, test-driven approach, git-flow, and what most API developers dread – documentation. I will talk about the architecture of the RESTful APIs, and the development + deployment process. We use Netflix-OSS components heavily in the architecture and cloud deployment, so I will cover them as well. Swagger is what we used for type-safe documentation, which is really easy to use and integrate. I will briefly talk about customizations we’ve done to Swagger in order to make it far more usable at Netflix. Throughout this effort there were lessons to be learnt, and plenty of best practices and recommendations for anyone starting out to build RESTful APIs, regardless of the platform or stack of choice. It’d be a great opportunity for me to walk through the architecture, and talk about the various components, technologies, and practices that are seeing increasing adoption in the modern, API driven landscape.
Citation preview
API ecosystem with Scala, Scalatra, and Swagger
at Netflix
Manish Pandit @lobster1234
Manish Pandit
Engineering Manager, Streaming Platforms
Netflix
@lobster1234
Thank you, OSCON!
50 M+ Streaming Members
40 countries and counting
Thousands of device types
Power the Consumer Electronics Partner Portal
Enable Certification of Netflix Ready Devices
Source of truth for all Device Data at Netflix
Correlate Streaming Quality Metrics
Our APIs
Devices
Smart devices +
Certification =
Lots of Device Metadata!
Model Firmware
Screen Resolution Subtitle Support
3D DRM
Remote Control Netflix SDK
4K HD …
Everything matters..
58 Resources
~10 methods per resource
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Architecture
HTTP Layer, and Manager Layer
Cassandra
EVCache
Crowd/SSO
RDS
Astyanax
Netflix OSS Cloud Components
Manager Layer = Business Code
Protocol Independent
Package-able as an artifact
HTTP Layer = Protocol Wrapper
Map HTTP methods to Manager functions
Transform JSON <=> Case Classes
The road trip from your IDE to a Production Node
Yes, it can be fun!
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Open Source Infrastructure Components
Libraries
Tools
Frameworks
PaaS with AWS IaaS
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Scala
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Immutable by default
Concise, yet expressive
Type inference
Multi paradigm
Java ecosystem
Functional
/** * Map results from a java.sql.ResultSet to a list of objects of another type * @param rs The result set * @param f Function that takes the result set and returns an instance of T * @tparam A The type * @return Sequence of instances of type A */ def mapResults[A](rs: ResultSet, f: ResultSet => A) : Seq[A] = { val array = ArrayBuffer[A]() while (rs.next) { array += f(rs) } array.toSeq }
/** * Function to return a cached value, or fetch it per the parameter f * @param key The key to look up in cache, or set against this key if not found * @param f The function that fetches the instance from some persistent store * @tparam A The type * @return None if not found in cache and persistent store, else Some[A] */ def withCache[A](key: String)(f: => Option[A]): Option[A] = {
EVCacheGateway.get[A](key) match { case None => val data = f if (!data.isEmpty) EVCacheGateway.set(key, data.get) data
case Some(x) => Some(x) } }
/** * Get a user given an ID * @param id The User ID * @return None if user is not found, else the user */ def getById(id: Long): Option[User] = {
withCache[User](s"User$id") { val query = "select * from users where user_id = ?" DBHelper.queryWith(query, extract, id).headOption
} }
Scalatra
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Lightweight HTTP wrapper
.war based deployment
Swagger support
ScalatraSpec
/** * Get all the chipsets ordered by name. */get("/providers", operation(getChipsetProviders)) {
Ok( ChipsetManager.getProviders(0,20) ) }
ScalaTest
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Simple
Intuitive
English-like
Rich
Promotes BDD
it should "return the chipset provider partner orgs" in { val data = ChipsetManager.getProviders(0,20) data.records.isEmpty should be (false) data.records.size should be (20) data.total should be >= 21l }
ScalatraSpec
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Traits to take ScalaTest to the next level
Helpers for JSON parsing
Plenty of wrappers (body, headers..)
addServlet(new ChipsetService, "/*") it should "get a list of chipset providers" in { getWithToken("/providers")(TestUserTokens.testToken) { status should equal (200) body should include (""""start":0,"end":19,"count":20""") }}
Swagger
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Excellent support within Scalatra
Type-safe documentation
Contract First API Sandbox
Machine Readable
get("/providers", operation(getAllChipsetProviders)) {//code
}case class PartnerList(data: Seq[PartnerOrg], start: Int, end: Int, count: Long, total: Long) def getAllChipsetProviders = authApiOperation[PartnerList]("getChipsetProviders", "Get all chipset providers") .parameter(queryParam[Option[Int]]("start").description("Starting record count, defaults to 0")) .parameter(queryParam[Option[Int]]("count").description("The number of records requested, defaults to 20"))
Swagger
Deployment
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Push to dev
Jenkins runs dev build, tests, merges to
master
Jenkins runs master build, makes a .deb
Aminator bakes an AMI from
the .deb
Asgard deploys the AMI in
staging cloud
Production?
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
lgml-mpandit:nrd-portal-api mpandit$ git statusOn branch devYour branch is up-to-date with 'origin/dev'.Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory)
modified: api/src/main/scala/com/netflix/nrdportal/http/ChipsetService.scalamodified: api/src/test/scala/com/netflix/nrdportal/http/ChipsetServiceSpec.scalamodified: api/src/main/scala/com/netflix/nrdportal/manager/ChipsetManager.scalamodified: api/src/test/scala/com/netflix/nrdportal/manager/ChipsetManagerSpec.scala
no changes added to commit (use "git add" and/or "git commit -a")
Check in
[info] Passed: Total 1386, Failed 0, Errors 0, Passed 1386[success] Total time: 1273 s, completed Jul 19, 2014 8:39:54 PMBuild step 'Build using sbt' changed build result to SUCCESSPushing HEAD to branch master of origin repositoryReturning node parameter for ssh-dynaslave-1291624cTriggering a new build of PPD-NRD-PORTAL-API-MASTER #1172Notifying upstream projects of job completionFinished: SUCCESS
Dev Build
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Integration Tests
Hitting Real DB
Refreshed every night with Production
Build Artifacts:nrd-portal-api_1.0-h1172.880e187_all.deb 127.71 MB[fingerprint]Changes:Added an endpoint to return a list of chipset provider partner orgs. NFLX-5514 (detail)
Started by upstream project PPD-NRD-PORTAL-API-DEV build number 1923originally caused by:Started by an SCM change
Revision: 880e1873fef63d278b3180b49af7434e234dee40origin/master
Master Build
Finally!
“A node, once deployed, cannot be changed.”
Immutable Deployments
Easy rollbacks
Consistency
Predictability
Immutable Deployments
Summary
Background Netflix OSS Components
Development Deployment/Delivery
Open Floor
Bridge the gap between dev and deploy
No such thing as too many tests
Automate everything
Document your APIs
Best solutions are implementation agnostic
Netflix OSS - http://netflix.github.io/#repo
Scalatra - http://www.scalatra.org/
Swagger - https://helloreverb.com/developers/swagger
ScalaTest - http://www.scalatest.org/
Session Feedback and Rating
slideshare.net/lobster1234
speakerdeck.com/mpandit
Recommended