A one stop solution
for Puppet and Openstack
Daniel Lobato Garcia
daniel.lobato.garcia@cern,ch
@eLobatoss
What is CERN
Between Geneva and the Jura mountains, straddling the Swiss-French border
Mission: learn what is the universe made of and how does it work?
3
Fundamental
questions in
physics
Why do particles have mass?
What is 96% of the universe made of?
Why isn’t there anti-matter in the universe?
What was the state of matter after the Big Bang?
4
8/12/2013 Document reference 5
8/12/2013 Document reference 6
8/12/2013 Document reference 7
8/12/2013 Document reference 8
Current status
• 270 Openstack hypervisors
• 2900 virtual machines
• 300 users
• 14 Puppet masters
• 6 Foreman backend nodes
• Some production services migrating to our
cloud – early birds
9
Goals
• Ramp up to 15K hypervisors – 150-200K
vms in 2015
• Multi-site (Hungary)
10
8/12/2013 Document reference 11
8/12/2013 Document reference 12
Why?
• Unnecessary homebrew stack of tools
• Shift to cloud standards with minimal
customizations
• High turnover – can’t teach new tools
13
Why?
• Symbiotic relationship with the community
14
Openstack?
• Modular IaaS free open source project
• APIs ~compatible with those of Amazon
15
Openstack Nova
(compute)
Cloud fabric controller
16
Openstack
Keystone (Identity)
RBAC
Integrated with LDAP
Multiple auth* methods
17
Openstack Glance
(Images)
Discovery, registration,
delivery of images
18
Openstack Horizon
(Dashboard)
19
Modules
• Puppet definitions for every use case you
can imagine.
• Dynamic environments
• Hadoop node
• Openstack hypervisor
• … you name it
20
21
Workflow..?
Modules and Git
• Manifests and hieradata are version
controlled
22
23
Git workflow
Puppet masters
24
Easy cherry pick
25
Git workflow
26
Git workflow
Jens
‘Puppetfiles’
Separate repositories
Makes environments and
creates them on the masters
Foreman
• Lifecycle management tool for VMs and
physical servers
• External Node Classifier – tells the puppet
master what a node should look like
27
28
29
Power operations & Foreman
8/12/2013 Document reference 30
Foreman Proxy
Physical
box IPMI
Physical
box IPMI
Physical
box IPMI
VM VM VM
Openstack
Nova API
Openstack VM creation
8/12/2013 Document reference 31
Openstack VM creation
8/12/2013 Document reference 32
Openstack VM creation
8/12/2013 Document reference 33
Scalability experiences
• Split up services
• Puppet – critical vs non critical
34
12 backend nodes
Batch
4 backend nodes
Interactive
Scalability experiences
• Foreman – split into different services
35
ENC Reports
processing UI/API
Load balancer
9443 – UI/API
9444 – Reports
9445 – ENC
…
Scalability experiences
• Autoscale via alarms (Heat)
• Define situations (i.e: load threshold..)
• Spin up VMs as needed
36
Scalability guidelines
37
github.com
/
cernops
38
39
Secrets provisioning (naïve)
• Use case: provision a db password
41
Secrets provisioning (hiera-gpg)
• Use case: provision a db password
42
Secrets provisioning (hack)
• Use case: provision a db password
43
Secrets provisioning
•Masters need not read secrets
44
Recommended
