Go at UberPrashant Varanasi, Senior Engineer, RPCApril 26, 2016

● Previously at Google — my first production experience with Go

● At Uber, I work on the RPC team

○ Support the Go client library for TChannel, our open source RPC framework

○ Building out the new fleetwide routing and load balancing sidecar

● One of the core reviewers for internal Go libraries

● Maintainer of the internal Go build infrastructure, go-build

About meJava-free for 2 years

● Traditionally used Python and Node.js

○ Node.js is popular in our marketplace and dispatch services

○ Python is popular for business logic as well as data analysis

● Almost all the business logic was

behind a monolithic Python API server

About UberFrom a large monolithic service...

The Uber Monolith

Users

Products

BackgroundChecks

Trips

Cities

Vehicles

Payments

Documents

Promos

● As Uber grew (in features and

engineers), continuous integration

turned into a liability as deploying the

codebase meant deploying everything

● We followed the lead of other hyper-

growth companies (i.e., Twitter, Netflix)

and broke up the monolith into many

microservices.

● This allowed more flexibility in

languages

About Uber...to over one thousand microservices

Users

ProductsTrips

Cities

Communication between services

uses HTTP/JSON or TChannel/Thrift

● In 2014, Aiden Scandella, one of our early

engineers, experimented with Go

○ Deployed a sandbox marketplace for the Uber External API

● By early 2015, Go was used across the

company,

○ Data Engineering: A cross-cluster query cache○ Marketplace: The Geofences service

● Go was selected for the following characteristics:

○ Simple language, high developer productivity○ Strong concurrency and parallelism built in

Introduction of GoCross-language microservices

Building up the ecosystem

● No standardized support for Go (build, lint, coverage) in our internal

infrastructure

○ Custom Makefiles integrated with go-junit-report and gocov

● Dependency management, used godep

○ Builds passed locally but not on build machines due to missing deps

○ Vendored dependencies hard to manage and review

● Lack of libraries for internal infrastructure

○ E.g., Kafka logging, dynamic configuration, background tasks

Making Go a first-class citizen

go-buildOld Way

Build InfrastructureFrom copy+pasted Makefiles and shell scripts to a common submodule

● Before the GO15VENDOREXPERIMENT, we started with godep

○ Vendored dependencies in every repo

○ Even internal dependencies were vendored as godep restore was flaky

○ Noisy code reviews, hard to update dependencies,

● The glide migration

○ Users can choose to check in their dependencies in the vendor folder

○ Or check in glide.yaml and glide.lock and have dependencies installed before build

○ Use an internal Github mirror to avoid outages affecting internal builds

Dependency ManagementFrom godep to glide

● Base libraries used by almost all services: config, metrics, logging, tracing,

RPC

● Utility and extension libraries:

○ Transport helpers, JSON + HTTP, TChannel + Thrift

○ Backoff, LRU cache, flags, worker pools

● Libraries for internal infrastructure built up over time

○ Storage (schemaless -- internal key-value store built on MySQL)

○ Translations, AVRO-encoded logging, dynamic config, experiments

Libraries for internal infrastructureMonorepo of Go libraries

● Now have over a hundred services written in Go

● Two years of production experience

○ Integrated runtime metrics (#goroutines, GC stats)

○ Easy profiling of services running in production

○ Investigated and fixed file descriptor leaks in open source libraries

○ Even found a compiler bug that led to stack corruption

● Strong library and infrastructure support

● Lots of resources: documentation, classes, mailing list

● Now the recommended language for new services

Experience with GoWriting code is fun again

What’s built in Go?

Goal: given a lat/lng pair, find the list of geofences this location falls in

● HTTP/JSON interface

● High throughput, low latency (P99 < 100ms)

● No infrastructure support, so project handled everything:

○ Config: Used the standard “encoding/json” package○ Logging: wrapped the standard “log” package, and reported errors to

Sentry○ Metrics: Wrapped open source StatsD client

Blog Post

GeofencesOne of the earliest Go services at Uber

Goal: match riders to drivers, sharding the matching across machines

● Ringpop for sharding the matching by location

● TChannel / Thrift for RPC interface

● Uses our internal libraries for:

○ Config (both static, and dynamic configuration)

○ Logging (includes logging to disk, Kafka, and Sentry)

○ Metrics (reported to M3, our internal Metrics infrastructure)

● Much more infrastructure support for Go services

GeobaseOne of the more recent Go services at Uber

● Embeddable application level sharding:

○ Fault detection: provides membership list of alive nodes using a variation of SWIM (similar to Serf)

○ Consistent hashing: keys are hashed to a node, and failures will not create load imbalances or redistribute every key

○ Forwarding: provides forward of HTTP or TChannel requests

● Used for sharding work, sharded cache, serializing

requests to a resource (in-order)

RingpopScalable, fault-tolerant application-layer sharding. Open source!

● Strong forwarding performance, as it was built for Ringpop forwarding

● Built on top of TCP, provides multiplexing and framing

○ Supports out of order responses, no head of line blocking

○ Very simple protocol, easy to implement in multiple languages

● Provides high-level RPC features:

○ Timeouts, consistent retry semantics

○ Connection management and smart peer selection

● Integrates with Thrift as a first-class citizen

TChannelIntra-datacenter RPC protocol. Open source, available for Go, Java, Node, and Python

● Strong integration with Thrift

○ Uses net/context

○ Integrates with error valuesval, err := client.Get(ctx, key)

if err != nil {

switch err := err.(type) {

case *keyvalue.KeyNotFound:

// Handle Thrift exception

default:

// Unknown error

}

}

TChannel + ThriftIntra-datacenter RPC protocol. Open source, available for Go, Java, Node, and Python

● Often used internally with Python and Node.js

● Uses pprof output to generate the flamegraph

go-torchVisualization of profiling output. Open source!

● Powers the majority of high QPS services at Uber

● Already the most popular language for new services

● Open source our core libraries and tools

■ TChannel, Ringpop, go-torch, yab (beta), zap (beta)

■ Working with open source community on opentracing

● Check out our engineering blog and Github page for more information

Go at Uber

Thanks