Containers & Kubernetespeople.cs.umu.se/cklein/wasp-slides/Grzadkowski...Google Cloud Platform logo Containers & Kubernetes Umea University 2017/05/11 Filip Grzadkowski

Google Cloud Platform

logo

Containers & KubernetesUmea University2017/05/11

Filip Grzadkowski <[email protected]>Technical Leader & Engineering Manager@fgrzadkowski

So I have my app...

Old Way: Shared machines

kernel

libs

app

app app

app

app

libskernel

libs

app app

kernel

app

libs

libskernel

kernel

Old Way: Virtual machines

libs

app

kernel

libs

app

libs

app

libs

app

New Way: Containers

How?

Implemented by a number of (unrelated) Linux APIs:

• cgroups: Restrict resources a process can consume• CPU, memory, disk IO, ...

• namespaces: Change a process’s view of the system• Network interfaces, PIDs, users, mounts, ...

• capabilities: Limits what a user can do• mount, kill, chown, ...

• chroots: Determines what parts of the filesystem a user can see


Google has been developing and using containers to manage our applications for over 12 years.

Images by Connie Zhou

Everything at Google runs in containers:• Gmail, Web Search, Maps, ...• MapReduce, batch, ...• GFS, Colossus, ...• Even GCE itself: VMs in

containers

Everything at Google runs in containers:• Gmail, Web Search, Maps, ...• MapReduce, batch, ...• GFS, Colossus, ...• Even GCE itself: VMs in

containers

We launch over 2 billion containers per week.

Docker

Source: Google Trends

Now that we have containers...

Isolation: Keep jobs from interfering with each other

Scheduling: Where should my job be run?

Lifecycle: Keep my job running

Discovery: Where is my job now?

Constituency: Who is part of my job?

Scale-up: Making my jobs bigger or smaller

Auth{n,z}: Who can do things to my job?

Monitoring: What’s happening with my job?

Health: How is my job feeling?

Enter Kubernetes

Manage applications,not machines

Google confidential │ Do not distribute

Part 1 - Basics


App deployment

Pods

Example: data puller & web serverPod

File Puller Web Server

Volume

ConsumersContent Manager

Run my application.


App: MyApp

Phase: prod

Role: FE

App: MyApp

Phase: test

Role: FE

App: MyApp

Phase: prod

Role: BE

App: MyApp

Phase: test

Role: BE

Labels


App: MyApp

Phase: prod

Role: FE

App: MyApp

Phase: test

Role: FE

App: MyApp

Phase: prod

Role: BE

App: MyApp

Phase: test

Role: BE

App = MyApp

Selectors


App: MyApp

Phase: prod

Role: FE

App: MyApp

Phase: test

Role: FE

App: MyApp

Phase: prod

Role: BE

App: MyApp

Phase: test

Role: BE

App = MyApp, Role = FE

Selectors


App: MyApp

Phase: prod

Role: FE

App: MyApp

Phase: test

Role: FE

App: MyApp

Phase: prod

Role: BE

App: MyApp

Phase: test

Role: BE

App = MyApp, Role = BE

Selectors


Selectors

App: MyApp

Phase: prod

Role: FE

App: MyApp

Phase: test

Role: FE

App: MyApp

Phase: prod

Role: BE

App: MyApp

Phase: test

Role: BE

App = MyApp, Phase = prod


App: MyApp

Phase: prod

Role: FE

App: MyApp

Phase: test

Role: FE

App: MyApp

Phase: prod

Role: BE

App: MyApp

Phase: test

Role: BE

App = MyApp, Phase = test

Selectors


ReplicaSet ReplicaSet- name = “my-rc”- selector = {“App”: “MyApp”}- podTemplate = { ... }- replicas = 4

API Server

How many?

3

Start 1 more

OK

How many?

4

Run N copies of my application.

node 1

f0118

node 3

node 4node 2

d9376

b0111

a1209

Replication Controller- Desired = 4- Current = 4

ReplicaSet

node 1

f0118

node 3

node 4node 2


d9376

b0111

a1209

ReplicaSet

node 1

f0118

node 3

node 4


b0111

a1209

ReplicaSet

node 1

f0118

node 3

node 4


b0111

a1209

c9bad

ReplicaSet


Networking


10.1.1.0/24

172.16.1.1

172.16.1.2

Docker networking

10.1.2.0/24172.16.1.1

10.1.3.0/24172.16.1.1


10.1.1.0/24

172.16.1.1

172.16.1.2

Docker networking

10.1.2.0/24172.16.1.1

10.1.3.0/24172.16.1.1

NAT

NAT

NAT

NAT

NAT


Kubernetes networking

IPs are routable• vs docker default private IP

Pods can reach each other without NAT• even across nodes

No brokering of port numbers• too complex, why bother?

This is a fundamental requirement• can be L3 routed• can be underlayed (cloud)• can be overlayed (SDN)


10.1.1.0/24

172.16.1.1

172.16.1.2

Kubernetes networking

10.1.2.0/24172.16.1.1

10.1.3.0/24172.16.1.1

Services

IP / DNS

Client

Expose my application.

selector = {“App”: “MyApp”}


...and more!

More

• Resource isolation• L7 Load Balancing• ConfigMap• Secret• Deployment• Job• DaemonSet• PersistentVolumes• Rolling updates• DNS• Autoscaling

• Monitoring• Logging• Rkt, Hyper• Authorization• Dashboard• ...


Some takeaways


Modularity

Loose coupling is a goal everywhere• simpler• composable• extensible

Code-level plugins where possible

Isolate risk by interchangeable parts

Example: ReplicaSetExample: Scheduler


Control loops

Drive current state -> desired state

Act independently

APIs - no shortcuts or back doors

Example: ReplicaSet

observe

diff

act


Part 2 - auto-recovery


Autoscaling


Autoscaling

Pod Autoscaling Cluster Autoscaling


Autoscaling

Pod Autoscaling Cluster Autoscaling


HorizontalPodAutoscaler


Configuration spec: scaleTargetRef: kind: ReplicationController name: my-rc minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 70

$ kubectl autoscale rc my-rc --min=2 --max=10 --cpu-percent=70


How does it work?


More users

observes


More pods

resizes


Scaling algorithm

sum(utilization)

target utilizationceil( )

min <= target # instances <= max

target # instances =


Cluster Autoscaling


Overview

Node A Node B

?


Overview

Node A Node B Node C


Overview



Overview



Overview



Overview

Node A Node B


• Scale UP when pod is pending• Scale DOWN when node not needed for

long time• Simulations

Status: beta since v1.3

Overview


Machine failures


Machine failures

Node A Node B


Machine failures

Node A Node B


Machine failures

Node A Node B

Node Controller● Unresponsive

node


Machine failures

Node A Node B


node


Machine failures

Node A Node B


node

ReplicaSet Controller● desired: 4● actual: 3


Machine failures

Node A Node B


node

ReplicaSet Controller● desired: 4● actual: 4


Machine failures

Node ATaint=BrokenDisk

Node B

Node Controller● BrokenDisk


Machine failures


Node B

?


Machine failures


Node B


Machine failures


Node B

?Toleration=BrokenDisk


Machine failures


Node B

Toleration=BrokenDisk


Is it useful?


AWS S3 outage

Realistically this kind of failure on a day like Tuesday could have resulted in hours of downtime. With Kubernetes, there was never a moment of panic, just a sense of awe watching the automatic mitigation as it happened.

Rob Scott,VP of Software Architecture @Spire

source

https://medium.com/spire-labs/mitigating-an-aws-instance-failure-with-the-magic-of-kubernetes-128a44d44c14

https://medium.com/spire-labs/mitigating-an-aws-instance-failure-with-the-magic-of-kubernetes-128a44d44c14


Kubernetes is Open

https://kubernetes.io

Code: github.com/kubernetes/kubernetesChat: slack.k8s.ioTwitter: @kubernetesio

open community

open design

open source

open to ideas

Documents

Containers & Kubernetespeople.cs.umu.se/cklein/wasp-slides/Grzadkowski...Google Cloud Platform logo Containers & Kubernetes Umea University 2017/05/11 Filip Grzadkowski