Embed Size (px)
DESCRIPTION
Throughout the last year, we have been using and developing tools that allow us to have an IaaS where our data center is configured by Puppet and our virtualization and authentication needs are catered by Openstack. RedHat's foreman is our lifecycle management tool which we configured to support both bare metal and Openstack virtual machines. We use git to manage environments and hostgroup configurations and we will tell you how we deal with its security implications, how to store Hieradata secrets. Switching from a homebrew toolchain to open source tools like Facter, Foreman, Openstack has turned out into many contributions to these teams. Nearly everyone at CERN has started to wear the devops hat which brings new challenges in terms of development workflows and scalability. Daniel Lobato Garcia Software Engineer, CERN Daniel Lobato is a developer who has worked in very different environmentst, from data centers and mainframes to startups. Nowadays he has dived into the Agile Infrastructure team at CERN where the design and implementation of the new computing infrastructure is done. As for Puppet, he currently helps RedHat to develop Foreman, a lifecycle management tool for physical and virtual machines. One of his goals at CERN is to knot this tool to all the relevant parts of the infrastructure, which includes Puppet for configuration management, OpenStack for virtualization and authentication, Puppetdb and others. He is sure the source of all computer problems is between the chair and the keyboard.
Citation preview
A one stop solution
for Puppet and Openstack
Daniel Lobato Garcia
daniel.lobato.garcia@cern,ch
@eLobatoss
What is CERN
Between Geneva and the Jura mountains, straddling the Swiss-French border
Mission: learn what is the universe made of and how does it work?
3
Fundamental
questions in
physics
Why do particles have mass?
What is 96% of the universe made of?
Why isn’t there anti-matter in the universe?
What was the state of matter after the Big Bang?
4
8/12/2013 Document reference 5
8/12/2013 Document reference 6
8/12/2013 Document reference 7
8/12/2013 Document reference 8
Current status
• 270 Openstack hypervisors
• 2900 virtual machines
• 300 users
• 14 Puppet masters
• 6 Foreman backend nodes
• Some production services migrating to our
cloud – early birds
9
Goals
• Ramp up to 15K hypervisors – 150-200K
vms in 2015
• Multi-site (Hungary)
10
8/12/2013 Document reference 11
8/12/2013 Document reference 12
Why?
• Unnecessary homebrew stack of tools
• Shift to cloud standards with minimal
customizations
• High turnover – can’t teach new tools
13
Why?
• Symbiotic relationship with the community
14
Openstack?
• Modular IaaS free open source project
• APIs ~compatible with those of Amazon
15
Openstack Nova
(compute)
Cloud fabric controller
16
Openstack
Keystone (Identity)
RBAC
Integrated with LDAP
Multiple auth* methods
17
Openstack Glance
(Images)
Discovery, registration,
delivery of images
18
Openstack Horizon
(Dashboard)
19
Modules
• Puppet definitions for every use case you
can imagine.
• Dynamic environments
• Hadoop node
• Openstack hypervisor
• … you name it
20
21
Workflow..?
Modules and Git
• Manifests and hieradata are version
controlled
22
23
Git workflow
Puppet masters
24
Easy cherry pick
25
Git workflow
26
Git workflow
Jens
‘Puppetfiles’
Separate repositories
Makes environments and
creates them on the masters
Foreman
• Lifecycle management tool for VMs and
physical servers
• External Node Classifier – tells the puppet
master what a node should look like
27
28
29
Power operations & Foreman
8/12/2013 Document reference 30
Foreman Proxy
Physical
box IPMI
Physical
box IPMI
Physical
box IPMI
VM VM VM
Openstack
Nova API
Openstack VM creation
8/12/2013 Document reference 31
Openstack VM creation
8/12/2013 Document reference 32
Openstack VM creation
8/12/2013 Document reference 33
Scalability experiences
• Split up services
• Puppet – critical vs non critical
34
12 backend nodes
Batch
4 backend nodes
Interactive
Scalability experiences
• Foreman – split into different services
35
ENC Reports
processing UI/API
Load balancer
9443 – UI/API
9444 – Reports
9445 – ENC
…
Scalability experiences
• Autoscale via alarms (Heat)
• Define situations (i.e: load threshold..)
• Spin up VMs as needed
36
Scalability guidelines
37
github.com
/
cernops
38
39
Secrets provisioning (naïve)
• Use case: provision a db password
41
Secrets provisioning (hiera-gpg)
• Use case: provision a db password
42
Secrets provisioning (hack)
• Use case: provision a db password
43
Secrets provisioning
•Masters need not read secrets
44
