Delivering a production Cloud Foundry Environment with Bosh

Delivering a produc0on CF env with Bosh

Agenda

• Few words about Bosh • Deploying a produc0on run0me • Data services - The missing part

Bosh

Bosh = ❤

Comprehensive lifecycle management of distributed systems

with a single tool.

• Infrastructure independence • Opera0ng system independence • Release vs. deployment

• Rolling updates • Monitoring / Self-Healing • Network configura0on • VM provisioning • Storage management / Persistent disk • Packaging, templa0ng, soOware-

configura0on

Produc0on CF

Produc0on?

Everything fails

Infrastructure fails first

A system is produc0on ready, if nobody has to get up when

ordinary failures occur.

Ordinary failures are failures happening within a single AZ.

Design to Fail

Infrastructure AZs

Availability Zone 1 Availability Zone 2 Availability Zone 3

3 AZs enable quorum basedleader elec0on

Low network latency required

AZ configura0on with Bosh

Before Bosh cloud_config

How to apply AZs to Boshresource_pools: - name: small_z1 network: default cloud_properties: instance_type: m1.small availability_zone: de_sky_1 stemcell: name: bosh-openstack-kvm-ubuntu-trusty-go_agent version: 3137

...

jobs: - name: psqldb1 instances: 1 resource_pool: small_z1 templates: - name: postgresql-ha release: postgresql94 - name: consul release: consul

With Bosh cloud_config

How to apply AZs to Boshcloud_config.yml

azs: - name: z1 cloud_properties: availability_zone: de_sky_1 - name: z2 cloud_properties: availability_zone: de_sky_2 - name: z3 cloud_properties: availability_zone: de_sky_3

vm_types: - name: small network: default cloud_properties: instance_type: m1.small stemcell: name: bosh-openstack-kvm-ubuntu-trusty-go_agent version: 3137

How to apply AZs to Boshdeployment manifest:

...

jobs: - name: psqldb instances: 3 templates: - name: postgresql-ha release: postgresql94 - name: consul release: consul azs: [z1, z2, z3] vm_type: small

Produc0on Cloud Foundry

Cloud Foundry Run0me

Elimina0ng SPOFs

• Create a list of system components • Check every component whether it is a

SPOF • Check every SPOF if it can be clustered • Cluster it SPOF if possible • Prepare for night-shiOs, if not

CF Run0me \w SPOFs

Cloud Foundry \w SPOFs

Can be deployed, redundantly.

Cloud Controller

HMDEA

Blob Store UAA

CC DB UAA DB

NATS

Blob Store NFS

Needs addi0on effort.

Cloud Foundry \wo SPOFs


Cloud Controller

HMDEA

Blob Store UAA

UAA & CC PostgreSQL Cluster

NATS

Blob Store SwiO/S3

Eliminated SPOF.

Wanted: 3 node PostgreSQL Cluster

for UAA and CC DB

Alterna*ve: MySQL Galera Cluster

3 VMs

PostgreSQL Cluster

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

CC-DB

UAA-DB

CC-DB

UAA-DB

CC-DB

UAA-DB

PostgreSQL Cluster

Deployed and Monitored by Bosh

Master DB Failure

PostgreSQL Cluster #1

Health Monitor

Bosh

Director …

Postgresql VM#110.0.0.5

DB Master

Consul Agent

repmgrd

Bosh Agent


DB Slave

Consul Agent

repmgrd

Bosh Agent


DB Slave

Consul Agent

repmgrd

Bosh Agent

Failover Challenges

• Provide a set of creden0als to access the PostgreSQL database

• Creden0als must always point to the DB master &

• IP addresses may change during failover when promo0ng a new master db server >

• Can’t use IP addresses in service bindings • Can’t use Bosh DNS names

Consul > HA DNS

Master DB Failure




DB Slave DB Slave

Consul Agent Consul Agent


DB Master

Consul Agent

repmgrd repmgrd repmgrd

Consul DNS Name: mst.s1-1.consul

Consul Node

Consul Cluster

Consul Node

Consul Node

Consul Node

Consul Node


Health Monitor

Bosh

Director …

Bosh Agent Bosh Agent Bosh Agent

Automa0c Failover




DB Slave DB Slave


repmgrd repmgrd

DB Master

Consul Node

Consul Cluster

Consul Node

Consul Node

Consul Node

Consul Node



Health Monitor

Bosh

Director …

Bosh Agent Bosh Agent

Bosh Self-Healing

Recovery from degraded Mode




DB Slave DB Slave


repmgrd repmgrd

DB Master

Consul Node

Consul Cluster

Consul Node

Consul Node

Consul Node

Consul Node


Health Monitor

Bosh

Director …


DB Slave

Consul Agent

repmgrd

Bosh Agent Bosh AgentBosh Agent

Checkpoint reached:

SPOFless Cloud Foundry Run0me

Cloud Foundry SPOFs


Cloud Controller

HMDEA

Blob Store UAA

UAA & CC PostgreSQL Cluster

NATS

Blob Store SwiO/S3

Eliminated SPOF.

A Cloud Foundry is not produc0on ready

without produc0on ready data services.

Data Services

Apps oOen strongly depend on data services


3 VMs

PostgreSQL Cluster

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App App App App App App App











3 VMs

PostgreSQL Cluster

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App

App App App App

App App App App App

App App App App App App

App App

App App

App App

App App App

App

App App App App App App App



App App App App App

App App App App

App App App App App App App App




App App App App App App App App App

App App App

Service Instance

Service Instance

Service Instance


3 VMs

PostgreSQL Cluster

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App

App App App App

App App App App App


App App

App App

App App

App App App

App


Service Instance

Service Instance

Service Instance


3 VMs

PostgreSQL Cluster

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

Service Instance

App App App App

App App App App

App App App App App


App App

App App

App App

App App App

App


Service Instance

Service Instance

Service Instance

Shared data services are not an op0on!

Use on-demand provisioned, dedicated service instances, instead.












Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance












Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance












Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Service instance failures will be contained.

Let Bosh do the dirty work.

a9s Deployer

Templates Deployments

Bosh

a9s Service Broker

my-3node-postgres-cluster-2

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

my-single-postgres-1

Postgresql VM#1

Middleware Adapter

Cloud Foundry Adapter

a9s PostgreSQL SPI

Service InstanceService Instance

my-3node-postgres-cluster-3

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

…

Cloud ControllerCF Clientcreate service

create service

create deployment from template xy with anributes {…}

deploy release abc & deployment manifest xyz

Execute deployments

create service specific

creden0als

create binding

a9s PostgreSQL Bosh Release

System Overview


anynines PostgreSQL Service

PostgreSQL Bosh

Run0me Bosh

Cluster 1 VM#1

CC #1 UAA #1 HM #1

…DEA #1

CC #2 UAA #2 HM #2

…DEA #2

CC #3 UAA #3 HM #3

…DEA #3

Cluster 1 VM#2

Cluster 1 VM#3

Cluster 2 VM#1

Cluster 2 VM#2

Cluster 2 VM#3

Cluster n VM#1

Cluster n VM#2

Cluster n VM#3

IaaS AZ 1 IaaS AZ 2 IaaS AZ 3

Summary

Bosh is a great companion for all Cloud Foundry related automa0on challenges.

Including CF run0me and data services.

Ques0ons?

@fischerjulian @anynines

Thank you.

@fischerjulian @anynines

TiteltextTitle small Lato Normal 64pt

Textebene Lato Leicht 42pt

Title Lato Normal 84pt

BeschriOung dunkel Lato leicht 18ptBeschriOung hell Lato Leicht 18ptTextebene Code Lato Leicht 18pt

<— only on white background!!!

Software

Delivering a production Cloud Foundry Environment with Bosh