Jose Nino@junr03

Next Generation Client APIs with Envoy MobileQCon London - March 2020

Some slides built in collaboration with: @goaway & @rebello95

Server

Networking

Networking

Client

@rebello95

Server

Networking

Networking

Client

@junr03

API

Server

Networking

Networking

Server

@junr03

Server

Networking

Server

Networking

Server

Networking

Server

Networking Server

NetworkingServer

NetworkingServer

Networking

Server

Networking

Service

Networking

Networking

Server

@junr03

API

Server

Networking

Service

Networking

Server

Networking

Service

Networking Service

NetworkingService

NetworkingServer

Networking

Server

Networking

Service

Networking

Server

Networking Server

Networking

Server

Networking

Server

Networking

Server

Networking Server

Networking

Server

Networking

Server

Networking

Server

Networking Service

Networking

Server

Networking

Service

Networking

Server

Networking Server

Networking

Server

Networking

Service

Networking

Networking

Server

@junr03

API

Server

Networking

Service

Networking

Server

Networking

Service

Networking Service

NetworkingService

NetworkingServer

Networking

Server

Networking

Service

Networking

Server

Networking Server

Networking

Server

Networking

Server

Networking

Server

Networking Server

Networking

Server

Networking

Server

Networking

Server

Networking Service

Networking

Server

Networking

Service

Networking

Server

Networking Server

Networking

Server

Networking

����

������

��

Jose Nino@junr03

Infrastructure Engineer

~4 years @

Service

Networking

Networking

Server

@junr03

API

Server

Networking

Service

Networking

Server

Networking

Service

Networking Service

NetworkingService

NetworkingServer

Networking

Server

Networking

Service

Networking

Server

Networking Server

Networking

Server

Networking

Server

Networking

Server

Networking Server

Networking

Server

Networking

Server

Networking

Server

Networking Service

Networking

Server

Networking

Service

Networking

Server

Networking Server

Networking

Server

Networking

����

������

��

Server

Networking

Networking

Client

@junr03

API

We recognized two important dimensions in API management: what data we send (shape), and how we send it (transport).

@junr03

~3 years in Server Networking

Server-side Interface Definition Language (IDL)

Service A“server”

@junr03

#service_a.proto

service ServiceA {

rpc UpdateA(ARequest) returns (AResponse) {

option (http.http_options).path = "/service_a";

option (http.http_options).method = "POST";

}

}

Service B “client”

● Network Proxy driven by Open-source

(OSS) community

● Common substrate for network behavior

What is ?

@junr03

Sidecar

Service B “client”

Sidecar

Service A“server”

Storage

EdgeMobile Client

Sidecar

Service B

Sidecar

Service A

3rd-party

Strongly-typed APIs and a Universal Network Primitive

@junr03

(circa early 2019)

The End

@junr03

Storage

EdgeMobile Client

Sidecar

Service B

Sidecar

Service A

3rd-party��

Strongly-typed APIs and a Universal Network Primitive

@junr03

Three 9s of reliability at the server-side is meaningless if the user of a mobile client is only

able to complete the desired product flows a fraction of the time.

What do we want from our Client APIs?

@junr03

Consistency ? ✓

Performance ? ✓

Extensibility ? ✓

Resilience ? ✓

Security ? ✓

Observability ? ✓

Three 9s of reliability at the server-side is meaningless if the user of a mobile client is only able to complete the desired product flows a fraction of the time.

What do we want from our Client APIs?

@junr03

Storage

Edge

Sidecar

Service B

Sidecar

Service A

3rd-party

Clients as another node in the mesh

Mobile Client

@goaway @junr03

API

Next generation Client APIs with Envoy Mobile

Mobile Client

@junr03

API

��We recognized two important dimensions in API management: what data we send (shape), and how we send it (transport).

~1 year in Client Networking

We recognized two important dimensions in API management: what data we send (shape), and how we send it (transport).

@junr03

API Shape

Early API workflow at Lyft

Tech spec

handwritten.swift

handwritten.kt

handwritten.py/.go

@rebello95

iOS

Server

{ “key_a”: “hello”}

Android

{ “kye_a”: “hello”}

Programming errors

@rebello95

iOS

Server

{ “key_a”: “hello”}

Android

{ “kye_a”: “hello”}

Programming errors

@rebello95

iOS

Server

{ “key_a”: “hello”}

Android

{ “kye_a”: “hello”}

Programming errors

@rebello95 @junr03

��

iOS

Server

{ “key_a”: “hello”}

Android

{ “kye_a”: “hello”}

Programming errors

@rebello95 @junr03

iOS

Server

{ “key_a”: “hello”}

Android

{ “kye_a”: “hello”}

if (android):else:

Programming errors

@rebello95

● Handwritten code led to mistakes/typos

● No visibility into deserialization failures at compile time

Problems

@rebello95 @junr03

YAML schema integration

@rebello95

paths:

/foo:

post:

parameters:

- in: body

schema:

$ref: '#/definitions/FooRequest'

responses:

200:

schema:

$ref: '#/definitions/FooResponse'

YAML definitions

@rebello95

● Manual code generation for Android

● No integration on iOS

● No single source of truth

YAML results

@rebello95

Consistency

@rebello95 @junr03

We needed a source of truth that provided guarantees about the shape of

any given API.

Envisioned workflow

Tech spec IDL repo

foo_api.swift

foo_api.kt

foo_api.py/.go

foo_api.proto

@rebello95

● Platform-neutral IDL for defining and serializing strongly-typed APIs.

● JSON support (migration)

● Already used server-side (consistency)

Why protobuf?

@rebello95 @junr03

IDL pipeline

/api/v1/foo.proto

Generators

+7 others

foo.go

foo.py

foo.kt

foo.swift

@rebello95

IDL pipeline

/api/v1/foo.proto

Generators

+7 others

foo.go

foo.py

foo.kt

foo.swift FooV1API library

foo-v1-api library

@rebello95

Using generated APIs in Swift

@rebello95

// Modules/FooV1API/FooAPI.swift

public struct FooRequest: Codable, Equatable { … }

public struct FooResponse: Codable, Equatable {

public let id: Int64

public let firstName: String

}

Generated models

@rebello95

// Modules/FooV1API/FooAPI.swift

import RxSwift

public protocol FooAPI {

func updateFoo(_ request: FooRequest)

-> Single<Result<FooResponse, FooAPIError>>

}

public struct FooClientAPI: FooAPI {

// Abstracted implementation

}

Generated APIs

@rebello95

// Modules/FooV1API/FooAPI.swift

import RxSwift

public protocol FooAPI {

func updateFoo(_ request: FooRequest)

-> Single<Result<FooResponse, FooAPIError>>

}

public struct FooClientAPI: FooAPI {

🌟 // Abstracted implementation 🌟 (hint hint wink wink!)

}

Generated APIs

@rebello95 @junr03

// Modules/FooV1APIMock/FooAPI.swift

open class FooMockAPI: FooAPI {

open lazy var mockUpdateFoo: (FooRequest)

-> Result<FooResponse, FooAPIError> = { … }

open func updateFoo(_ request: FooRequest)

-> Single<Result<FooResponse, FooAPIError>> { … }

}

Generated mocks

@rebello95

● Single source of truth for all APIs

● Consistent, generated code on all platforms

● Highly testable

● Underlying transport fully abstracted 🌟 565958

Results

@rebello95

Performance

@rebello95 @junr03

Experimenting with payload encoding

Client

Service

JSON request

Existing system

JSON response

@rebello95

Client

Service

JSON requestContent-Type: json

Accept: x-protobuf,json

Content negotiation

MiddlewareJSON > Protobuf

Protobuf requestContent-Type: x-protobuf

Protobuf responseContent-Type: x-protobuf

MiddlewareNo change

Protobuf responseContent-Type: x-protobuf

@rebello95

JSON

Protobuf

@rebello95

● 40%+ reduction in payload sizes

● Faster response times

● Higher success rates

● Transparent to engineers

Wins from protobuf

@rebello95

Three 9s of reliability at the server-side is meaningless if the user of a mobile client is only able to complete the desired product flows a fraction of the time.

What do we want from our Client APIs?

@junr03

Consistency ✓ ✓

Performance ✓ ✓

Extensibility ? ✓

Resilience ? ✓

Security ? ✓

Observability ? ✓

Extensibility

@junr03

Using protobuf annotations to declare API behavior

service Foo {

rpc GetFoo(FooRequest) returns (FooResponse) {

option (http.http_options).path = "/foo/receive";

option (http.http_options).method = "GET";

option (http.http_options).client_poll_ms = 5000;

option (http.http_options).gzip_enabled = true;

}

}

Declarative APIs

@rebello95 @junr03

Service

Envoy

URLSession

iOS

OkHTTP

Android

@rebello95

Three 9s of reliability at the server-side is meaningless if the user of a mobile client is only able to complete the desired product flows a fraction of the time.

What do we want from our Client APIs?

@junr03

Consistency ✓ ✓

Performance ✓ ✓

Extensibility 🤔 ? ✓

Resilience ? ✓

Security ? ✓

Observability ? ✓

We recognized two important dimensions in API management: what data we send (shape), and how we send it (transport).

@junr03

API Transport

Service

Envoy

Library X

Client

@rebello95

Clients as another node in the mesh

Storage

Edge

Sidecar

Service B

Sidecar

Service A

3rd-partyClient

@goaway @junr03

Standardizing infrastructure

@goaway @junr03

Why is world domination transport standardization useful?

● Write once, deploy everywhere

● Common tooling for common problems

● Reduce cognitive load

@goaway @junr03

Envoy Mobile to the rescue!

● v0.1: proof-of-concept

● v0.2: laying the foundation

● v0.3: first production-ready release

@goaway @junr03

Envoy as a Library

@junr03

How did we turn a network proxy into a networking library?

Build System: bazel

@goaway @junr03

● Cross-platform support

● Used by Envoy

//library/common:c_types//library/kotlin:android_framework

//library/swift:ios_framework

Layered Architecture

//:ios_dist

//:android_dist

//library/common:main_interface

@goaway @junr03

//library/common/...

@envoy//source/...

Platform (iOS/Android) Bridge (C) Native (C++/Envoy)

How to run a process in an app?

picture of an engine (a very fast one)

@goaway @junr03

Threading contexts

Application Threads

Envoy Main Thread

Callback Threads

@goaway @junr03

Library Matrix

Platform (iOS/Android) Bridge (C types/bindings) Native (C++/Envoy)

Application Threads

Envoy Main Thread

Callback Threads

@goaway @junr03

Library Lifecycle - Running Envoy

Application Threads

Envoy Main Thread

Callback Threads

EnvoyEngine

@goaway @junr03

Platform (iOS/Android) Bridge (C types/bindings) Native (C++/Envoy)

Library Lifecycle - Running Envoy

Application Threads

Envoy Main Thread

Callback Threads

EnvoyEngine

EnvoyRunner Engine

@goaway @junr03

Platform (iOS/Android) Bridge (C types/bindings) Native (C++/Envoy)

Server Envoy

Envoy Main Thread

Worker Threads

Envoy

Dispatcher

Http Manager

Requests

@goaway @junr03

Platform (iOS/Android)

Application Threads

Envoy Main Thread

Callback Threads

EnvoyEngine

EnvoyRunner

Dispatcher

Http Manager

@goaway @junr03

Bridge (C types/bindings) Native (C++/Envoy)

Library Lifecycle - using Envoy Constructs

Platform (iOS/Android)

Application Threads

Envoy Main Thread

Callback Threads

EnvoyEngine

EnvoyRunner

Dispatcher

Http Manager

HttpStream

@goaway @junr03

Bridge (C types/bindings) Native (C++/Envoy)

Library Lifecycle - starting a stream

Library Lifecycle - dispatching a stream

Application Threads

Envoy Main Thread

Callback Threads

EnvoyEngine

EnvoyRunner

Dispatcher

Http Manager

HttpStream

@goaway @junr03

Platform (iOS/Android) Bridge (C types/bindings) Native (C++/Envoy)

Http Manager

Http Filters

@junr03

L7 FilterL7 FilterHttp Filter

Dispatcher ● Foundational extension point

● Enact behavior

● Provide the guarantees we want!

Callbacks

Library Lifecycle - callbacks

Application Threads

Envoy Main Thread

Callback Threads

EnvoyEngine

EnvoyRunner

Dispatcher

Http Manager

HttpStream

@goaway @junr03

Platform (iOS/Android) Bridge (C types/bindings) Native (C++/Envoy)

Library Lifecycle - platform callbacks

Application Threads

Envoy Main Thread

Callback Threads

EnvoyEngine

EnvoyRunner

Dispatcher

Http Manager

HttpStream

Callbacks

Lambdas

Dispatcher

@goaway @junr03

Platform (iOS/Android) Bridge (C types/bindings) Native (C++/Envoy)

Platform (iOS/Android)

Overall View

Application Threads

Envoy Main Thread

Callback Threads

EnvoyEngine

EnvoyRunner

Dispatcher

Http Manager

HttpStream

Callbacks

Lambdas

Dispatcher

Canceled

@goaway @junr03

Bridge (C types/bindings) Native (C++/Envoy)

Canceled

Platform Code

let envoy = try EnvoyClientBuilder(domain: "api.envoyproxy.io") .addLogLevel(.warn) .addStatsFlushSeconds(60) .build()

val envoy = EnvoyClientBuilder( Domain("api.envoyproxy.io")) .addLogLevel(LogLevel.WARN) .addStatsFlushSeconds(60) ... .build()

@goaway @junr03

Build an Engine

let envoy = try EnvoyClientBuilder(domain: "api.envoyproxy.io") .addLogLevel(.warn) .addStatsFlushSeconds(60) .build()

@goaway @junr03

Build an Engine

let envoy = try EnvoyClientBuilder(domain: "api.envoyproxy.io") .addLogLevel(.warn) .addStatsFlushSeconds(60) .build()

@goaway @junr03

Build a Request

let request = RequestBuilder(path: "/pb.api.v1.Foo/GetBar") .addHeader(name: "x-custom-header", value: "foobar") .build()

@goaway @junr03

Build a Request

let request = RequestBuilder(path: "/pb.api.v1.Foo/GetBar") .addHeader(name: "x-custom-header", value: "foobar") .build()

@goaway @junr03

Build a Response Handler

let handler = ResponseHandler() .onHeaders { headers, status, _ -> ... } .onData { data -> // Deserialize message data here } ...

@goaway @junr03

Build a Response Handler

let handler = ResponseHandler() .onHeaders { headers, status, _ -> // Product logic } .onData { data -> // Product logic } ...

@goaway @junr03

Three 9s of reliability at the server-side is meaningless if the user of a mobile client is only able to complete the desired product flows a fraction of the time.

What do we want from our Client APIs?

@junr03

Consistency ✓ ✓

Performance ✓ ✓

Extensibility 🤔 ? ✓

Resilience ? ✓

Security ? ✓

Observability ? ✓

IDL pipeline

/api/v1/foo.proto

Generators

+7 others

foo.go

foo.py

foo.kt

foo.swift FooV1API library

foo-v1-api library

@goaway @junr03

NSUrl

OkHttp

IDL pipeline

/api/v1/foo.proto

Generators

+7 others

foo.go

foo.py

foo.kt

foo.swift FooV1API library

foo-v1-api library

Client

@goaway @junr03

Ecosystem

@junr03

Shape + Transport = Ecosystem!

Three 9s of reliability at the server-side is meaningless if the user of a mobile client is only able to complete the desired product flows a fraction of the time.

What do we want from our Client APIs?

Consistency ✓ ✓

Performance ✓ ✓

Extensibility ? ✓

Resilience ? ✓

Security ? ✓

Observability ? ✓@junr03

Extensibility

@junr03

Using protobuf annotations to declare API behavior and Envoy Mobile to enact consistent behavior.

service Foo {

rpc GetFoo(FooRequest) returns (FooResponse) {

option (http.http_options).path = "/foo/receive";

option (http.http_options).method = "GET";

option (http.http_options).client_poll_ms = 5000;

option (http.http_options).gzip_enabled = true;

}

}

Declarative APIs

@rebello95

Service

Envoy

URLSession

iOS

OkHTTP

Android

@rebello95 @junr03

�� ����

Extensibility

@junr03

Using protobuf annotations to declare API behavior and Envoy Mobile to enact consistent behavior.

Http Manager

Http Filters

@junr03

L7 FilterL7 FilterHttp Filter

Dispatcher ● Foundational extension point

● Enact behavior

● Provide the guarantees we want!

Envoy filters

Compression Filtergzip request

Mobile

Proxy

Compression FilterUnzip payload

🥳

EdgeClient

@junr03

Advanced transport

@junr03

● Dynamic Forward Proxy ✅

● Compression 🚧

● Deferred Requests 🕒 (Resilience)

● OAuth 🕒 (Security)

● Request Signing 🕒 (Security)

● Network condition 🕒 (Resilience)

Deferred Requests

@junr03

Deferred Request Filter

Mobile

Proxy

Client❌ Service ❌

Deferred Request Filter

Mobile

Proxy

Client✅ Service ✅

🥳

��

��

Request Signing

@junr03

Client

Request Signature Filter

sign request

Mobile

Proxy

Request Signature Filter

Verify signature

Edge

🕵‍♀

Advanced transport

@junr03

● Dynamic Forward Proxy ✅

● Compression 🚧

● Deferred Requests 🕒 (Resilience)

● OAuth 🕒 (Security)

● Request Signing 🕒 (Security)

● Network condition 🕒 (Resilience)

Observability

@junr03

True end-to-end insight

Observability

ts(envoy_mobile.cluster.api.upstream_rq.count)

ts(envoy_edge.cluster.*.upstream_rq.count)

@goaway @junr03

Time-series Metrics

MetricsService

@goaway @junr03

Dashboards!

ts(envoy_edge...)

@goaway @junr03

ts(envoy_mobile...)

Three 9s of reliability at the server-side is meaningless if the user of a mobile client is only able to complete the desired product flows a fraction of the time.

What do we want from our Client APIs?

@junr03

Consistency ✓ ✓

Extensibility ✓ ✓

Performance ✓ ✓

Resilience ✓ ✓

Security ✓ ✓

Observability ✓ ✓

Onwards!

● Additional functionality in filters

● Protocol Experimentation with QUIC

● OSS Code Generators: complete OSS ecosystem for model-based

APIs.

@junr03

Next Generation Client APIs

@junr03

Model-based APIs defined in a strongly-typed IDL so platform engineers can iterate on behavior using

a common transport layer.

Alpha Beta Production Experiment @Lyft!

@goaway @junr03

envoy-mobile.github.io@junr03

Join us!