Let’s talk CI/CD and DevOpswith Cloudbees & ClusterHQ

Ryan WallnerDeveloper Evangelist, ClusterHQ

What’s in here?

How to get CI and Staging

to use realistic data

for each branch and commit from every developer

Docker and Jenkins today● Jenkins is great for automating builds … but it’s not so great at

managing data, this is left to external systems

● Docker and Docker Compose are great for defining and running multi or single container environments … not so true for managing data.

● How do you pull point-in-time references of data to anywhere in your CI/CD pipeline?○ Consider Dev, Test, Staging, QA and Production.

Example Jenkins Workflows

Jenkins Master

Jenkins Slave + Docker

Jenkins Slave + Docker

Jenkins Slave + Docker

Docker Staging

$ git push origin branch

Notify

Build Build

Success?

Dev Test Staging

*or maybe only when you merge into master or specific staging branches

But.. staging and testing without realistic data isn’t great● Does it represent what a user would see?

● Is there “enough” data to see how the application works under load?

● Is the data isolated so I can test or see how staging looks with different types or amounts of data?

● Do I know where this data came from, who created it?

FlockerHub and FliFlockerHub is GitHub for data

a. Hub & Catalog for Docker data volumesb. Fine-grained access controls & user mgmtc. Available initially as ClusterHQ hosted

service

Fli is Git for data

a. cli interface for FlockerHubb. Snapshot Docker volumesc. Push & pull from FlockerHubd. Available as Apache 2.0 download

User View

Data Servers

https://flockerhub.clusterhq.com

https://data.flockerhub.clusterhq.com

Github/Google

Data operations

Session

AuthTokentoken.txt

Staging / test with realistic data is great - w/ FlockerHub● Does it represent what a user would see?

Yes, I can have prod ops give us obfuscated production snapshots so we can get a realistic view of data.

● Is there “enough” data to see how the application?Yes, we can get a fresh, up-to-date snapshot from Ops teams.

● Is the data isolated so I can see how staging looks with different types or amounts of data?

Yes, we can get fresh clones of a snapshot we choose each and every commit.

● Do I know where this data came from, who created it?Yes, We can track who the snapshot was created by and who has access to it.

Example Jenkins Workflow with Fli and FlockerHub

Jenkins Master

Jenkins Slave + Docker

Jenkins Slave + Docker

Jenkins Slave + Docker

Docker Staging

$ git push my-branch

Notify

Build Build

Success?

Dev Test Staging

*Jenkinsfile has a reference of what Snapshot to use in staging.

$ fli pull

*every developer, every branch. Isolated snapshots

pushpull

Jenkins Master

Jenkins Slave + Docker

Docker Staging

$ git push my-branch

Notify

Build test_123With snapshot 321

Success?

Dev Test Staging

*Jenkinsfile has a reference to use snapshot 321 for test 123

$ fli pull

1. pull (only changes)2. clone snapshot3. give clone to docker4. docker run && test5. snapshot volume if failure6. docker stop/kill

Manage snapshots, orgs and teams https://flockerhub.clusterhq.com

Demo

Jenkins workflow with Fli and FlockerHub

Thank YouSign up

https://clusterhq.com/register/flockerhub-beta/ @ryanwallner

Walkthrough❏ Checkout new branch to work with, push and watch build in Jenkins❏ Show Staging❏ Change branch to use a different snapshot that has a bad phone number or vin number

❏ Show output from error in CI❏ Use manifest produced by failure to reproduce locally.❏ vi fli-docker-manifest.yml (in inventory-app)❏ fli-docker run -f fli-docker-manifest.yml -c -t /root/fh.token❏ docker ps to show locally running re-created env with snapshot❏ run the test that failed locally.

❏ docker run --net=flicompose_net -e FRONTEND_HOST="frontend" -e DATABASE_HOST="db" -e FRONTEND_PORT=8000 --rm -v ${PWD}/frontend/:/frontend clusterhq/mochatest "cd /frontend && npm install && mocha --debug test/test_db_vehicle_vins.js"

❏ cleanup with fli-docker destroy -f fli-docker-manifest.yml -clean

Backup/Example Slides

Test failures within Jenkins

Test logs show failed test and what snapshot was taken after the failure

Jenkins Build/Test with Snapshots/ClonesJenkins first pulls the necessary snapshots and creates a volume for the Docker app to use.[parallel tests 9] Syncing Volumeset inventory-app[parallel tests 9] Getting ID of the Snapshot 750k-records-snap[parallel tests 9] ID of snapshot is 6234bb23-15c9[parallel tests 9] Pulling inventory-app:6234bb23-15c9[parallel tests 9] Creating volume from Snapshot[parallel tests 9] Volume /chq/e41408a1-9e41/bd6f5bc0-f4ad created[parallel tests 9] Loading the path into the application.

The volume /chq/e41408a1-9e41/bd6f5bc0-f4ad/ will be mapped into the docker-compose.yml

The application boots up with a point in time snapshot of data in staging for the particular branch

* http://jenkinsdemo.clusterhq.com/job/inventory-pipeline-multi/job/master/

Staging is available per commit with its own isolated data.

fli-dockerhttps://github.com/ClusterHQ/fli-docker

Stateful-Apps as Code?fli-docker

The fli-docker utility is designed to simplify the deployment of stateful applications inside Docker containers using a Flocker Hub Stateful Application Manifest (SAM) used side by side with the Docker Compose file.

In a local development environment$ cat docker-compose.yml

services: db: networks: - net image: rethinkdb volumes: - rethink-data:/data ports: - 28015 - 29015 - 8080 command: rethinkdb

$ cat fli-manifest.yml

docker_app: docker-compose.yml

flocker_hub: endpoint: data.flockerhub.clusterhq.com

volumes: - name: rethink-data snapshot: someSnapshot-of-DB volumeset: inventory-app