CORONA Dan Kelleher | Scientific Systems Administrator | Harvard FAS Research Computing

Overview •  Node Types •  Docker Containers

–  Base, Libvirt, nfs, ONE server, ONE hypervisor, Sunstone –  Docker network, privilege and PID –  Systemd vs Supervisord –  Compose

•  Challenges –  Configuration –  Oneadmin token and SSH keys

Motivation •  Binary roll out and roll back

–  Simple deployment –  Automated testing –  Upgrades

•  Elasticity? –  Deploy a new node using cluster scheduler –  Tear it down when it’s not longer needed

•  Kolla –  https://github.com/openstack/kolla

Node Types Controller

Hybrid

ONE Server •  Host Net •  Host PID

Hybrid Hybrid

Hypervisor NFS •  Host Net •  Host PID •  Privileged

Sunstone •  Host Net •  Host PID •  Privileged

Libvirt •  Host Net •  Host PID •  Privileged •  Cgroups •  /dev •  /run •  /lib/

modules

NFS •  Host Net •  Host PID •  Privileged

ONE Hyper •  Host Net •  Host PID •  Privileged •  Cgroups •  /dev •  /run

ONE Server •  Host Net •  Host PID

NFS •  Host Net •  Host PID •  Privileged

Libvirt •  Host Net •  Host PID •  Privileged •  Cgroups •  /dev •  /run •  /lib/

modules

ONE Hyper •  Host Net •  Host PID •  Privileged •  Cgroups •  /dev •  /run

Sunstone •  Host Net •  Host PID •  Privileged

Overview •  Node Types •  Docker Containers

–  Base, Libvirt, nfs, ONE server, ONE hypervisor, Sunstone –  Docker network, privilege and PID –  Systemd vs Supervisord –  Compose

•  Challenges –  Configuration –  Oneadmin token and SSH keys

Corona-Base

Libvirt The Libvirt container is straight forward build but:

–  In order to setup the virtualization limitations the libvirt container needs access to cgroups

–  It also needs access to the device directory and kernel modules directory to control the hardware

Libvirt is also a self contained daemon which makes process management easier

Libvirt

NFS

NFS supervisor

NFS •  NFS on the other hand is easy to build and doesn’t

require access to a lot of resources but is complex to configure

•  NFS requires 5 intertwined daemons which each need to be configured

•  In addition a reload condition needs to be defined •  This is one of the cases where systemd would have

excelled. (More on that in a bit)

ONE Server

ONE Hypervisor

ONE •  By comparison the OpenNebula software was both easy

to build and simple to configure in supervisord •  The challenge in the ONE software lies in configuring the

software it self without jumping into the container.

Sunstone

This container needs a lot more work

CORONA Un-Contained •  Host PID is mostly for convenience of management •  Host Networking was the best solution to create the

bridges for the trunked vlans •  Privilege is required by anything which needs to bind to

the host networking stack in the protected port range –  Some of this could likely be reduced using docker port binding –  Libvirt needs to be able to inject kernel modules –  ONE hypervisor needs to create the bridges

MySQL •  There’s a MySQL container straight from docker hub

–  https://hub.docker.com/_/mysql/

•  Also there is a MariaDB official container on docker hub –  https://hub.docker.com/_/mariadb/

•  In the future docker should allow the controller container to pick up the database name, password, and username if they are on the same host

How to Supervise

Systemd •  Pros:

–  Dependencies are handled –  Reloads are simple –  Configured by the

packages

•  Cons: –  Running multiple instances

proved challenging –  Require a lot of packages

Supervisord •  Pros:

–  Running multiple instances is simple

–  Doesn’t run as root

•  Cons: –  Has to be hand configured

Docker Compose

Overview •  Node Types •  Docker Containers

–  Base, Libvirt, nfs, ONE server, ONE hypervisor, Sunstone –  Docker network, privilege and PID –  Systemd vs Supervisord –  Compose

•  Challenges –  Configuration –  Oneadmin token and SSH keys

Configuration •  The Docker standard is to use environment variable

–  At the moment we are using flat files mounted as read only docker volumes.

•  A Script needs to be created to parse environment variables and put them in the correct config files –  Building this script has been challenging –  A lot of configuration variables add to complexity

Token and SSH •  In our current setup these are laid down by puppet •  Mounting these out in production is not very secure •  Hypervisors can’t auto check-in without these

–  This is critical for elasticity

Demo?

Questions?