Openshift for Continuous Integration Alex Lossent – IT/PES/IS AI meeting 1-Oct-2015 Openshift for Continuous Integration1

Openshift for Continuous Integration 1

Openshift for Continuous Integration

Alex Lossent – IT/PES/IS

AI meeting 1-Oct-2015


Outline• Continuous Integration: context and history• Platform-as-a-Service concept• Docker/Kubernetes/Openshift• How Openshift compares to Puppet, Heat• Plans and outlook



Why a Continuous Integration platform?

• Concept: improve code quality by integrating code frequently and applying automatic quality control (tests etc.)

• Awareness of multiple CI setups for all large software projects at CERN• In IT, PH, GS, EN…• Using Bamboo, Jenkins, TeamCity…

• First experience with Jenkins in PES: AIQA• Project to provide a Jenkins “platform as a

service” • to make it easier for developers to deploy CI



CI: current offer• Since end 2014, we provide a CI “platform”

• Using well-established, open-source Jenkins• A Java web application

• Request an instance from http://cern.ch/forge • Documentation http://cern.ch/jenkinsdoc

• Clear interest from developers• >40 instances as of September 2015• Developers@CERN forum on 28-29 September

• A “platform” rather than a “service”• We provide the infrastructure (Puppet/Openstack)• Out of scope: support for Jenkins configuration itself, integration

of existing CI setups

• Full details presented at ITTF 5-Dec-2014


http://cern.ch/forge

http://cern.ch/jenkinsdoc

https://indico.cern.ch/event/344225/


Significant challenges• Need more flexibility

• Every project has specific needs• No standard development platform at CERN

• Frequent requests for custom RPMs, libraries, OSes, plugins, integration scenarios etc.

• Need more automation• Many changes trigger a support ticket• Too much manual work for both users and ourselves

• Need better scaling• 1..n VM per project with CI• Security Team project to provide static code analysis



Towards a PaaS solution• We want to provide the infrastructure for CI• While giving maximum freedom to Jenkins

users• With minimum cost of entry and

maintenance effort for users• I.e. self-service, fully automated, self-repairing

• A good candidate for a PaaS solution



Options to deploy a web app…


Less effortLess flexibilitySpecific help

More effortBestflexibilityUnspecific help

Web hosting:AFS/DFS websiteMiddleware-on-demand

SaaS:DrupalSharePoint

IaaS:OpenstackPuppetDNSLB

PaaS:Good flexibilityLimited effort

“to develop, run, and manage Web applications without the complexity of building and maintaining the infrastructure typically associated with developing and launching an app” [Wikipedia]

https://en.wikipedia.org/wiki/Platform_as_a_service


PaaS: useful technologies• Containers

• Operating System-level virtualization:• A host system runs several isolated user spaces on top of a single kernel• Provides app isolation with less overhead than VMs• Containers can be provisioned and started in seconds

• Allow to separate maintenance of the host OS from that of the application itself

• Like VMs can stop worrying about physical hardware

• Docker• A popular container solution• An application can be packaged with all its environment into an image

• Includes all libraries, app framework, code, configuration…• Portable: can be deployed and instantiated with little constraints on a

variety of hosts



PaaS: useful technologies• Kubernetes

• Docker container orchestration system by Google• Software-defined network (SDN) to provide

advanced services• Recent but gets a lot of traction

• RedHat Openshift• Builds a PaaS solution on top of Kubernetes• Adds user management, UI, web proxy, image

registry, application templates etc.• And an application lifecycle (Continuous

Deployment)



What can we expect from using Openshift?

• Flexibility• Delegate full control on the Jenkins app to users

• Self-service and automation (by design)• Scaling

• Better resource-efficiency with containers

• Advanced level of service (from Kubernetes)• Persistent storage via Ceph (RBD)• Fast recovery (virtual IPs, replication controller)• Good availability (container migration)



How to apply this to CI?• We provide and maintain a reference Jenkins

Docker image• With relevant pre-configuration and integration• And an Openshift application template for it

• Users create instances of Jenkins on our Openshift cluster (self-service via CLI/GUI)

• Allow user-provided Docker images for slaves• Each project can have its own build environment• No need to deploy and maintain dedicated VMs• Fresh container for each job run



PaaS general use case• Openshift interesting for general web application hosting

• ~1400 HTTP-related openings in the external firewall• Applications currently hosted outside CERN

• More flexible than web hosting/MWoD• Wide choice of frameworks, languages…

• Significantly less effort than IaaS• No host OS to manage• Configuration management tailored for web apps

• Scalability, redundancy, good availability are almost free• Fills a gap in providing tools to facilitate web app

deployment and upgrade



Case study: GitLab• IaaS-based deployment

• Many VMs (dev/prod, redundancy with DNSLB)• NFS filer, DBoD (single points of failure)

• Complex configuration management with Puppet• Puppet module + tricks, custom Lemon sensors• Significant work with each new version

• Complex, manual upgrade procedure• 30 minute downtime (even for possibly online upgrades)• Not doable with Puppet (lack of synchronization), even less so

with migration of Puppet repos to GitLab

• Slow recovery in case of failure• DNS update delay• Can be much longer if DBoD or NFS filer fail (if possible)



Case study: what Openshift could improve

• Better resource efficiency: a lot less VMs• Direct use of Ceph (RBD) volume for storage• Single GitLab instance (“pod”) at any given time

• Faster recovery• Takes seconds to reschedule a container, reconnect to storage,

reassign virtual IP• SPoF (and dependency loop) removal: NFS filer

• possibility to host database in Openshift as well

• Considerably simplified management and procedures• Apps are described in terms of how to generate a Docker image from

source code and a deployment configuration• Openshift designed to facilitate Continuous Delivery

• Automation of building Docker images, deploying new app versions (and rollback)

• Can leverage standard RoR migration/upgrade procedure



Caveat: security updates• App owners responsible for maintaining their

apps… and all their dependencies• Security fixes!• Openshift provides an (optional) workflow

whereby a base image can be updated and app rebuilt on top of it

• Mostly appropriate for apps with a Continuous Deployment model• Frequent integration, updates and redeployment



An interesting complement to existing web app hosting offer


Less effortLess flexibilitySpecific help

More effortBestflexibilityUnspecific help

Web hosting:AFS/DFS websiteMiddleware-on-demand

SaaS:DrupalSharePoint

IaaS:OpenstackPuppetDNSLB

PaaS:Good flexibilityLimited effort


What about Heat?• Openstack Heat was also considered as a way to address

current challenges with CI• Heat = orchestration for Openstack• Can be used to provide a Jenkins “application template”• Improvements regarding automation, self-service, flexibility

• This option would have been fantastic if available end 2014 when CI project started

• But now suffers from availability of a solution like Openshift• Non-trivial work required:

• integration of Heat with Puppet to maintain long-lived VMs• complete refactor of how Jenkins is managed by Puppet

• Resource efficiency• “level of service” (recovery etc.)



Where are we now?• Prototype Openshift cluster was relatively

easy to deploy• Hosts managed with Puppet (+Ansible)• Polishing to do but no blocker in sight

• Work on a preconfigured Jenkins image in progress• With SSO, GitLab integration, etc.

• Production status before end of the year is still realistic



Collaboration opportunities?• Docker has been lurking around for some

time, but no well-defined strategy yet• In particular, no official SLC5/6 or CC7 images

• A new type of resource to manage• Reassign when users leave CERN, etc.• Integration with existing services?

• New technologies to master• Docker, Kubernetes are definitely technologies

that will play a major role in the coming years



Do you want to know more?• Openshift 3 Roadshow:

free workshop in Geneva on October 22nd


https://meeting-reg.com/redhat/openshift-roadshow-2015/registration/index.php?project_name=geneva_october22&sc_cid=

https://meeting-reg.com/redhat/openshift-roadshow-2015/registration/index.php?project_name=geneva_october22&sc_cid=

Documents

Openshift for Continuous Integration Alex Lossent – IT/PES/IS AI meeting 1-Oct-2015 Openshift for Continuous Integration1