Local testing of Containerized Distributed Systems

Strange Group Meetup, Justin Carter (@bodymindarts)

Berlin, Oct. 27. 2016 Stark & Wayne

Delmo

Delmo is a tool to test systems running within multiple docker containers

https://github.com/bodymindarts/delmo

30 staff, 7 countries

PaaS experts

Cloud Foundry experts

Plattforms

Dingo PostgreSQL

Delmo

http://www.slideshare.net/KarlIsenberg/cloud-surfing-kubernetes-on-mesos

The sophistication of our platform bounds the granularity of decomposition in our

systems

(also means more complexity to setup locally)

How much of the cloud stack do we want to run locally (if any)?

Platform on dev workstation? Dev env in the cloud?

Apps decoupled from platform?

Contract Driven Tests / Mocks add a lot of code to maintain.

Dealing with deps?

Do we need them?

Cloud dev workflowHow much of the stack do we want to run?

How do we want to run our deps?

How do we automate this and integrate into an efficient workflow?

When you can deploy 1000s of apps, you’d better be able to deploy 1000s

of dbs!

Platform consideration

PostgreSQL in Docker

Clustered with streaming replication

Continuously backed up

Automatic failover

Dingo PostgreSQL

https://github.com/dingotiles/dingo-postgresql-release

Dingo PostgreSQL Architecture

H A P r o x y

Deployed by BOSH

Client 1

Client 2

Critical FeaturesRe-configuration of PostgreSQL

Restart PostgreSQL

Streaming backups to S3

Changing Docker Image

Test independently of BOSH

Test smaller system boundary

Build and test containers locally

Goals

Postgres Leader

Postgres Replica

Example TestWill failover work?

Dingo PG

HAProxy

Query

Leader

etcd

AssertDid data get replicated?

Did reconfiguration / promotion work?

Are connections being accepted?

Given: System definition

When: Some failure

Then: Assert expected Behaviour

Test Case

Delmo

Declare system with docker-compose

Declare test cases

Run tests (locally or remotely)

patroni1

Docker Compose

patroni2

etcd

System Tasks

Tasks are packaged bash scripts that execute commands against the running system

System

Tasks

Cache

Delmo Config

Meta

Tests

Tasks

Demo!

https://github.com/bodymindarts/delmo-demo

Initialise

Start SystemRun TasksCleanup

Summary

MoreRun on a remote docker-machine

Run tasks in parallel

delmo-concourse image (for executing in CI/CD)

Next Steps

Better error output / validation

Network failure / partitioning

Support for other backends (bosh)?

Questions?

Justin Carter @bodymindarts github.com/bodymindarts github.com/bodymindarts/delmo