Rackspace Private Cloud Installation Guideebook.konfigurasi.net/Openstack/Openstack Rackspace Private Cloud... · Rackspace Private Cloud Installa-tion Guide January 20, 2016 RPC

rackspace.com/cloud/private

http://rackspace.com/cloud/private

Rackspace Private Cloud Installa-tion Guide

January 20, 2016 RPC v10

ii

Rackspace Private Cloud Installation GuideRPC v10 (2016-01-20)Copyright © 2015 Rackspace All rights reserved.

This documentation is intended for Rackspace customers who are interested in installing an Open-Stack-powered private cloud according to the recommendations of Rackspace.



iii

Table of Contents1. Preface ........................................................................................................................ 1

1.1. About Rackspace Private Cloud ........................................................................ 11.2. Rackspace Private Cloud configuration .............................................................. 11.3. Rackspace Private Cloud support ...................................................................... 1

2. Overview ..................................................................................................................... 32.1. Ansible ............................................................................................................. 32.2. Linux Containers (LXC) ..................................................................................... 32.3. Host layout ...................................................................................................... 42.4. Host networking .............................................................................................. 62.5. OpenStack Networking ................................................................................... 102.6. Installation requirements ................................................................................ 132.7. Installation workflow ...................................................................................... 13

3. Deployment host ....................................................................................................... 153.1. Installing the operating system ....................................................................... 153.2. Configuring the operating system ................................................................... 153.3. Installing source and dependencies ................................................................. 153.4. Configuring Secure Shell (SSH) keys ................................................................ 16

4. Target hosts .............................................................................................................. 174.1. Installing the operating system ....................................................................... 174.2. Configuring Secure Shell (SSH) keys ................................................................ 174.3. Configuring the operating system ................................................................... 184.4. Configuring LVM ............................................................................................ 184.5. Configuring the network ................................................................................ 18

4.5.1. Reference architecture ......................................................................... 194.5.2. Configuring the network on a target host ............................................ 21

5. Deployment configuration ......................................................................................... 275.1. Prerequisites ................................................................................................... 275.2. Configuring target host networking ................................................................ 275.3. Configuring target hosts ................................................................................. 305.4. Configuring service passwords ........................................................................ 325.5. Specifying additional groups for the system user (optional) ............................. 335.6. Configuring the is_metal flag (optional) ...................................................... 335.7. Configuring the hypervisor (optional) ............................................................. 335.8. Configuring the Image Service (optional) ........................................................ 335.9. Configuring the Block Storage service (optional) ............................................. 35

5.9.1. Configuring the Block Storage service on bare metal ............................ 355.9.2. Configuring the Block Storage service in a container ............................. 375.9.3. Configure the Block Storage service with NFS protocols ........................ 395.9.4. Configure Block Storage backups to Object Storage .............................. 395.9.5. Creating Block Storage availability zones .............................................. 40

5.10. Configuring Active Directory or LDAP ........................................................... 405.10.1. Adding LDAP/Active Directory ........................................................... 415.10.2. LDAP Directory configuration ............................................................. 425.10.3. Active Directory configuration ............................................................ 43

6. Foundation playbooks ............................................................................................... 446.1. Running the foundation playbook .................................................................. 446.2. Troubleshooting ............................................................................................. 44

7. Infrastructure playbooks ............................................................................................ 46



iv

7.1. Running the infrastructure playbook ............................................................... 467.2. Verifying infrastructure operation ................................................................... 47

8. OpenStack playbooks ................................................................................................ 488.1. Utility Container Overview .............................................................................. 488.2. Running the OpenStack playbook ................................................................... 498.3. Set up Block Storage backups (optional) ......................................................... 508.4. Verifying OpenStack operation ....................................................................... 50

9. Rackspace RPC Solutions tab ..................................................................................... 529.1. Launch a solution ........................................................................................... 52

10. Rackspace Private Cloud monitoring ........................................................................ 5310.1. Service and response .................................................................................... 5310.2. Hardware monitoring ................................................................................... 5310.3. Software monitoring ..................................................................................... 5310.4. CDM monitoring ........................................................................................... 5410.5. Running monitoring playbooks ..................................................................... 54

11. Operations .............................................................................................................. 5611.1. Configuring an NFS back end to store Glance images .................................... 5611.2. Galera cluster maintenance ........................................................................... 56

11.2.1. Removing nodes ................................................................................ 5711.2.2. Starting a cluster ................................................................................ 57

11.3. Galera cluster recovery ................................................................................. 5911.3.1. Single-node failure ............................................................................. 5911.3.2. Multi-node failure .............................................................................. 5911.3.3. Complete failure ................................................................................ 6111.3.4. Rebuilding a container ....................................................................... 61

12. Additional resources ................................................................................................ 6412.1. Document Change History ............................................................................ 64



v

List of Figures2.1. Host Layout Overview .............................................................................................. 62.2. Network components ............................................................................................... 72.3. Container network architecture ................................................................................ 92.4. Bare/Metal network architecture ............................................................................ 102.5. Networking agents containers ................................................................................ 112.6. Compute hosts ....................................................................................................... 122.7. Installation workflow .............................................................................................. 143.1. Installation workflow .............................................................................................. 154.1. Installation workflow .............................................................................................. 174.2. Target hosts for infrastructure, networking, and storage services ............................. 234.3. Target hosts for Compute service ........................................................................... 245.1. Installation workflow .............................................................................................. 276.1. Installation workflow .............................................................................................. 447.1. Installation workflow .............................................................................................. 468.1. Installation work flow ............................................................................................. 48



1

1. PrefaceRackspace Private Cloud Software has been developed by Rackspace as a way to quicklyinstall an OpenStack private cloud, configured as recommended by Rackspace OpenStackspecialists.

1.1. About Rackspace Private CloudRackspace Private Cloud software uses Ansible to create an OpenStack cluster on UbuntuLinux. The installation process provides a familiar approach for Linux system administrators,and the environment can be updated easily without downloading and installing a new ISO.

1.2. Rackspace Private Cloud configurationRackspace Private Cloud Software uses Ansible and Linux Containers (LXC) to install andmanage OpenStack Juno with the following services:

• Compute (nova)

• Object Storage (swift)

• Block Storage (cinder)

• Networking (neutron)

• Dashboard (horizon)

• Identity (keystone)

• Image Service (glance)

• Orchestration (heat)

RPC also provides the following infrastructure, monitoring, and logging services to supportOpenStack:

• Galera with MariaDB

• RabbitMQ

• Memcached

• Rsyslog

• Logstash

• Elasticsearch with Kibana

1.3. Rackspace Private Cloud supportRackspace offers 365x24x7 support for Rackspace Private Cloud Software. If you are inter-ested in purchasing Escalation Support or Core Support for your cloud, or taking advantageof our training offerings, contact us at: <[email protected]>.

http://docs.openstack.org/developer/nova/

http://docs.openstack.org/developer/swift/

https://wiki.openstack.org/wiki/Cinder

https://wiki.openstack.org/wiki/Neutron

http://docs.openstack.org/developer/horizon/

http://docs.openstack.org/developer/keystone/

http://docs.openstack.org/developer/glance/

https://wiki.openstack.org/wiki/Heat

http://galeracluster.com/

https://mariadb.org/

http://www.rabbitmq.com/

http://memcached.org/

http://www.rsyslog.com/

http://logstash.net/

http://www.elasticsearch.org/

mailto:[email protected]



2

You can also visit the RPC community forums. The forum is open to all RPC users and ismoderated and maintained by Rackspace personnel and OpenStack specialists:

https://community.rackspace.com/products/f/45

For more information about Rackspace Private Cloud, please visit the Rackspace PrivateCloud pages:

• Software and Reference Architecture

• Support

• Resources

For any other information regarding Rackspace Private Cloud Software, refer to theRackspace Private Cloud release notes.

https://community.rackspace.com/products/f/45

http://www.rackspace.com/cloud/private/openstack/

http://www.rackspace.com/cloud/private/openstack/

http://www.rackspace.com/cloud/private/openstack/software/

http://www.rackspace.com/cloud/private/openstack/support/

http://www.rackspace.com/cloud/private/openstack/resources/



3

2. OverviewRackspace Private Cloud (RPC) v10 Software uses a combination of Ansible and Linux Con-tainers (LXC) to install and manage OpenStack Juno. This chapter discusses the followingtopics:

• The technology used by Rackspace Private Cloud Software

• The environment and network architecture

• Requirements to install Rackspace Private Cloud Software

• The installation process workflow

2.1. AnsibleRPC v10 Software uses a combination of Ansible and Linux Containers (LXC) to install andmanage OpenStack Juno. Ansible provides an automation platform to simplify systemand application deployment. Ansible manages systems using Secure Shell (SSH) instead ofunique protocols that require remote daemons or agents.

Ansible uses playbooks written in the YAML language for orchestration. For more informa-tion, see Ansible - Intro to Playbooks.

In this guide, Rackspace refers to the host running Ansible playbooks as the deploymenthost and the hosts on which Ansible installs RPC as the target hosts.

A recommended layout for installing RPC involves five target hosts in total: three infras-tructure hosts, one compute host, and one logging host. RPC software also supports oneor more optional storage hosts. All hosts require at least four 10 Gbps network interfaces.In Rackspace datacenters, hosts can use an additional 1 Gbps network interface for servicenetwork access. More information on setting up target hosts can be found in Section 2.3,“Host layout” [4].

For more information on physical, logical, and virtual network interfaces within hosts seeSection 2.4, “Host networking” [6].

2.2. Linux Containers (LXC)Containers provide operating-system level virtualization by enhancing the concept of ch-root environments, which isolate resources and file systems for a particular group of pro-cesses without the overhead and complexity of virtual machines. They access the same ker-nel, devices, and file systems on the underlying host and provide a thin operational layerbuilt around a set of rules.

The Linux Containers (LXC) project implements operating system level virtualization on Lin-ux using kernel namespaces and includes the following features:

• Resource isolation including CPU, memory, block I/O, and network using cgroups.

http://docs.ansible.com/playbooks_intro.html



4

• Selective connectivity to physical and virtual network devices on the underlying physicalhost.

• Support for a variety of backing stores including LVM.

• Built on a foundation of stable Linux technologies with an active development and sup-port community.

Useful commands:

• List containers and summary information such as operational state and network configu-ration:

# lxc-ls --fancy

• Show container details including operational state, resource utilization, and veth pairs:

# lxc-info --name container_name

• Start a container:

# lxc-start --name container_name

• Attach to a container:

# lxc-attach --name container_name

• Stop a container:

# lxc-stop --name container_name

2.3. Host layoutThe recommended layout contains a minimum of five hosts (or servers).

• Three control plane infrastructure hosts

• One logging infrastructure host

• One compute host

To use the optional Block Storage (cinder) service, a sixth host is required. Block Storagehosts require an LVM volume group named cinder-volumes. See Section 2.6, “Installation re-quirements” [13] and Section 4.4, “Configuring LVM” [18] for more information.

The hosts are called target hosts because Ansible deploys the RPC environment within thesehosts. The RPC environment also requires a deployment host from which Ansible orches-trates the deployment process. One of the target hosts can function as the deploymenthost.



5

At least one hardware load balancer must be included to manage the traffic among thetarget hosts.

Infrastructure Control Plane target hosts contain the following services:

• Infrastructure:

• Galera

• RabbitMQ

• Memcached

• Logging

• OpenStack:



• Compute management (nova)



• Dashboard (horizon)

Infrastructure Logging target hosts contain the following services:

• Rsyslog

• Logstash

• Elasticsearch with Kibana

Compute target hosts contain the following services:

• Compute virtualization

• Logging

(Optional) Storage target hosts contain the following services:

• Block Storage scheduler

• Block Storage volumes



6

Figure 2.1. Host Layout Overview

2.4. Host networkingThe combination of containers and flexible deployment options requires implementation ofadvanced Linux networking features such as bridges and namespaces.

Bridges provide layer 2 connectivity (similar to switches) among physical, logical, and virtualnetwork interfaces within a host. After creating a bridge, the network interfaces are virtual-ly "plugged in" to it.

RPC software uses bridges to connect physical and logical network interfaces on the host tovirtual network interfaces within containers on the host.

Namespaces provide logically separate layer 3 environments (similar to routers) within ahost. Namespaces use virtual interfaces to connect with other namespaces including thehost namespace. These interfaces, often called veth pairs, are virtually "plugged in" be-tween namespaces similar to patch cables connecting physical devices such as switches androuters.

Each container has a namespace that connects to the host namespace with one or moreveth pairs. Unless specified, the system generates random names for veth pairs.



7

The relationship between physical interfaces, logical interfaces, bridges, and virtual inter-faces within containers is shown in Figure 2.2, “Network components” [7].

Figure 2.2. Network components

Target hosts can contain the following network bridges:

• LXC internal lxcbr0:

• Mandatory (automatic).

• Provides external (typically internet) connectivity to containers.

• Automatically created and managed by LXC. Does not directly attach to any physical orlogical interfaces on the host because iptables handle connectivity. Attaches to eth0 ineach container.

• Container management br-mgmt:

• Mandatory.

• Provides management of and communication among infrastructure and OpenStackservices.

• Manually created and attaches to a physical or logical interface, typically a bond0VLAN subinterface. Also attaches to eth1 in each container.

• Storage br-storage:



8

• Optional.

• Provides segregated access to block storage devices between Compute and Block Stor-age hosts.

• Manually created and attaches to a physical or logical interface, typically a bond0VLAN subinterface. Also attaches to eth2 in each associated container.

• OpenStack Networking tunnel/overlay br-vxlan:

• Mandatory.

• Provides infrastructure for VXLAN tunnel/overlay networks.

• Manually created and attaches to a physical or logical interface, typically a bond1VLAN subinterface. Also attaches to eth10 in each associated container.

• OpenStack Networking provider br-vlan:

• Mandatory.

• Provides infrastructure for VLAN and flat networks.

• Manually created and attaches to a physical or logical interface, typically bond1. Al-so attaches to eth11 in each associated container. Does not contain an IP address be-cause it only handles layer 2 connectivity.

Figure 2.3, “Container network architecture” [9] provides a visual representation ofnetwork components for services in containers.



9

Figure 2.3. Container network architecture

The RPC software installs the Compute service in a bare metal environment rather thanwithin a container. Figure 2.4, “Bare/Metal network architecture” [10] provides a visualrepresentation of the unique layout of network components on a Compute host.



10

Figure 2.4. Bare/Metal network architecture

2.5. OpenStack NetworkingOpenStack Networking (neutron) is configured to use a DHCP agent, L3 Agent and LinuxBridge agent within a networking agents container. Figure 2.5, “Networking agents con-tainers” [11] shows the interaction of these agents, network components, and connec-tion to a physical network.



11

Figure 2.5. Networking agents containers



12

The Compute service uses the KVM hypervisor. Figure 2.6, “Compute hosts” [12] showsthe interaction of instances, Linux Bridge agent, network components, and connection to aphysical network.

Figure 2.6. Compute hosts



13

2.6. Installation requirementsDeployment host:

• Required items:

• Ubuntu 14.04 LTS (Trusty Tahr) or compatible operating system that meets all other re-quirements.

• Secure Shell (SSH) client supporting public key authentication.

• Synchronized network time (NTP) client.

• Python 2.7 or later.

Target hosts:

• Required items:

• Ubuntu Server 14.04 LTS (Trusty Tahr) 64-bit operating system, with Linux kernel ver-sion 3.13.0-34-generic or later.

• SSH server supporting public key authentication.

• Synchronized NTP client.

• Optional items:

• For hosts providing Block Storage (cinder) service volumes, a Logical Volume Manager(LVM) volume group named cinder-volumes.

• LVM volume group named lxc to store container file systems. If the lxc volume groupdoes not exist, containers in the root file system will automatically be installed.

Note

Each container creates a 5 GB logical volume. Plan storage accordingly tosupport the quantity of containers on each target host.

2.7. Installation workflowThis diagram shows the general workflow associated with RPC installation.



14

Figure 2.7. Installation workflow



15

3. Deployment hostFigure 3.1. Installation workflow

The RPC software installation process requires one deployment host. The deployment hostcontains Ansible and orchestrates the RPC installation on the target hosts. One of the tar-get hosts, preferably one of the infrastructure variants, can be used as the deploymenthost. To use a deployment host as a target host, follow the steps in Chapter 4, “Targethosts” [17] on the deployment host. This guide assumes separate deployment and tar-get hosts.

3.1. Installing the operating systemInstall the Ubuntu Server 14.04 (Trusty Tahr) LTS 64-bit operating system on the deploy-ment host with at least one network interface configured to access the Internet or suitablelocal repositories.

3.2. Configuring the operating systemInstall additional software packages and configure NTP.

1. Install additional software packages if not already installed during operating system in-stallation:

# apt-get install aptitude build-essential git ntp ntpdate \ openssh-server python-dev sudo

2. Configure NTP to synchronize with a suitable time source.

3.3. Installing source and dependenciesInstall the source and dependencies for the deployment host.

1. Clone the repository into the /opt directory:

http://releases.ubuntu.com/14.04/



16

# cd /opt# git clone -b TAG https://github.com/openstack/openstack-ansible.git

Replace TAG with the current stable release tag.

2. Install pip 1.5.6 and dependencies:

# curl -O https://bootstrap.pypa.io/get-pip.py# python get-pip.py \--find-links="https://mirror.rackspace.com/rackspaceprivatecloud/python_packages/TAG" \--no-index

Replace TAG with the current stable version release.

3. Install Ansible and dependencies:

# pip install -r /opt/openstack-ansible/requirements.txt

Note

The command described above will install the correct version of Ansible.Do not use the apt package manager. If Ansible is installed using the lat-ter, uninstall it before performing this step.

3.4. Configuring Secure Shell (SSH) keysAnsible uses Secure Shell (SSH) with public key authentication for connectivity between thedeployment and target hosts. To reduce user interaction during Ansible operations, keypairs should not include passphrases. However, if a passphrase is required, consider usingthe ssh-agent and ssh-add commands to temporarily store the passphrase before perform-ing Ansible operations.



17

4. Target hostsFigure 4.1. Installation workflow

The RPC software installation process requires at least five target hosts that will contain theOpenStack environment and supporting infrastructure. On each target host, perform thefollowing tasks:

• Naming target hosts.

• Install the operating system.

• Generate and set up security measures.

• Update the operating system and install additional software packages.

• Create LVM volume groups.

• Configure networking devices.

4.1. Installing the operating systemInstall the Ubuntu Server 14.04 (Trusty Tahr) LTS 64-bit operating system on the targethost with at least one network interface configured to access the Internet or suitable localrepositories.

Note

On target hosts without local (console) access, Rackspace recommends addingthe Secure Shell (SSH) server packages to the installation.

4.2. Configuring Secure Shell (SSH) keysAnsible uses Secure Shell (SSH) for connectivity between the deployment and target hosts.

1. Copy the contents of the public key file on the deployment host to the /root/.ssh/authorized_keys on each target host.

2. Test public key authentication from the deployment host to each target host. SSHshould provide a shell without asking for a password.



18

4.3. Configuring the operating systemCheck the kernel version, install additional software packages, and configure NTP.

1. Check the kernel version. It should be 3.13.0-34-generic or later.

2. Install additional software packages if not already installed during operating system in-stallation:

# apt-get install bridge-utils debootstrap ifenslave ifenslave-2.6 \ lsof lvm2 ntp ntpdate openssh-server sudo tcpdump vlan

Note

During the installation of RPC, unattended upgrades are disabled. For long-running systems, periodically check for and apply security updates.

3. Add the appropriate kernel modules to the /etc/modules file to enable VLAN andbond interfaces:

# echo 'bonding' >> /etc/modules # echo '8021q' >> /etc/modules

4. Configure NTP to synchronize with a suitable time source.

5. Reboot the host to activate the changes.

4.4. Configuring LVM1. To use the optional Block Storage (cinder) service, create an LVM volume group

named cinder-volumes on the Block Storage host. A metadata size of 2048 must bespecified during physical volume creation. For example:

# pvcreate --metadatasize 2048 physical_volume_device_path# vgcreate cinder-volumes physical_volume_device_path

2. Optionally, create an LVM volume group named lxc for container file systems. If the lxcvolume group does not exist, containers will be automatically installed into the file sys-tem under /var/lib/lxc by default.

4.5. Configuring the networkAlthough Ansible automates most deployment operations, networking on target hosts re-quires manual configuration because it can vary dramatically per environment. For demon-stration purposes, these instructions use a reference architecture with example network in-terface names, networks, and IP addresses. Modify these values as needed for the particu-lar environment.

The reference architecture for target hosts contains the following mandatory components:



19

• A bond0 interface using two physical interfaces. For redundancy purposes, avoid usingmore than one port on network interface cards containing multiple ports. The exam-ple configuration uses eth0 and eth2. Actual interface names can vary depending onhardware and drivers. Configure the bond0 interface with a static IP address on the hostmanagement network.

• A bond1 interface using two physical interfaces. For redundancy purposes, avoid usingmore than one port on network interface cards containing multiple ports. The exampleconfiguration uses eth1 and eth3. Actual interface names can vary depending on hard-ware and drivers. Configure the bond1 interface without an IP address.

• Container management network subinterface on the bond0 interface and br-mgmtbridge with a static IP address.

• The OpenStack Networking VXLAN subinterface on the bond1 interface and br-vxlanbridge with a static IP address.

• The OpenStack Networking VLAN br-vlan bridge on the bond1 interface without an IPaddress.

The reference architecture for target hosts can also contain the following optional compo-nents:

• Storage network subinterface on the bond0 interface and br-storage bridge with astatic IP address.

For more information, see OpenStack Ansible Networking.

4.5.1. Reference architectureAfter establishing initial host management network connectivity using the bond0 interface,modify the /etc/network/interfaces file as described in the following procedure.

Procedure 4.1. Modifying the network interfaces file

1. Physical interfaces:

# Physical interface 1auto eth0iface eth0 inet manual bond-master bond0 bond-primary eth0


# Physical interface 3auto eth2iface eth2 inet manual bond-master bond0

# Physical interface 4auto eth3

https://github.com/openstack/openstack-ansible/blob/10.1.0/etc/network/README.rst



20

iface eth3 inet manual bond-master bond1

2. Bonding interfaces:

# Bond interface 0 (physical interfaces 1 and 3)auto bond0iface bond0 inet static bond-slaves eth0 eth2 bond-mode active-backup bond-miimon 100 bond-downdelay 200 bond-updelay 200 address HOST_IP_ADDRESS netmask HOST_NETMASK gateway HOST_GATEWAY dns-nameservers HOST_DNS_SERVERS

# Bond interface 1 (physical interfaces 2 and 4)auto bond1iface bond1 inet manual bond-slaves eth1 eth3 bond-mode active-backup bond-miimon 100 bond-downdelay 250 bond-updelay 250

If not already complete, replace HOST_IP_ADDRESS, HOST_NETMASK,HOST_GATEWAY, and HOST_DNS_SERVERS with the appropriate configuration forthe host management network.

3. Logical (VLAN) interfaces:

# Container management VLAN interfaceiface bond0.CONTAINER_MGMT_VLAN_ID inet manual vlan-raw-device bond0

# OpenStack Networking VXLAN (tunnel/overlay) VLAN interfaceiface bond1.TUNNEL_VLAN_ID inet manual vlan-raw-device bond1

# Storage network VLAN interface (optional)iface bond0.STORAGE_VLAN_ID inet manual vlan-raw-device bond0

Replace *_VLAN_ID with the appropriate configuration for the environment.

4. Bridge devices:

# Container management bridgeauto br-mgmtiface br-mgmt inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references tagged interface bridge_ports bond0.CONTAINER_MGMT_VLAN_ID address CONTAINER_MGMT_BRIDGE_IP_ADDRESS netmask CONTAINER_MGMT_BRIDGE_NETMASK dns-nameservers CONTAINER_MGMT_BRIDGE_DNS_SERVERS



21

# OpenStack Networking VXLAN (tunnel/overlay) bridgeauto br-vxlaniface br-vxlan inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references tagged interface bridge_ports bond1.TUNNEL_VLAN_ID address TUNNEL_BRIDGE_IP_ADDRESS netmask TUNNEL_BRIDGE_NETMASK

# OpenStack Networking VLAN bridgeauto br-vlaniface br-vlan inet manual bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references untagged interface bridge_ports bond1

# Storage bridge (optional)auto br-storageiface br-storage inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port reference tagged interface bridge_ports bond0.STORAGE_VLAN_ID address STORAGE_BRIDGE_IP_ADDRESS netmask STORAGE_BRIDGE_NETMASK

Replace *_VLAN_ID, *_BRIDGE_IP_ADDRESS, and *_BRIDGE_NETMASK,*_BRIDGE_DNS_SERVERS with the appropriate configuration for the environment.

4.5.2. Configuring the network on a target host

This example uses the following parameters to configure networking on a single targethost. The sample interface configurations are intended to illustrate the scope and design ofthe network. When deploying outside of Rackspace data centers, settings should be config-ured according to the data center's network hardware.

See Figure 4.2, “Target hosts for infrastructure, networking, and storage services” [23]and Figure 4.3, “Target hosts for Compute service” [24] for a visual representation ofthese parameters in the architecture.

• VLANs:

• Host management: Untagged/Native

• Container management: 10

• Tunnels: 30

• Storage: 20

Networks:



22

• Host management: 10.240.0.0/22

• Container management: 172.29.236.0/22

• Tunnel: 172.29.240.0/22

• Storage: 172.29.244.0/22

Addresses:

• Host management: 10.240.0.11

• Host management gateway: 10.240.0.1

• DNS servers: 69.20.0.164 69.20.0.196

• Container management: 172.29.236.11

• Tunnel: 172.29.240.11

• Storage: 172.29.244.11



23

Figure 4.2. Target hosts for infrastructure, networking, and storage services



24

Figure 4.3. Target hosts for Compute service

Contents of the /etc/network/interfaces file:







25

# Bond interface 0 (physical interfaces 1 and 3)auto bond0iface bond0 inet static bond-slaves eth0 eth2 bond-mode active-backup bond-miimon 100 bond-downdelay 200 bond-updelay 200 address 10.240.0.11 netmask 255.255.252.0 gateway 10.240.0.1 dns-nameservers 69.20.0.164 69.20.0.196

# Bond interface 1 (physical interfaces 2 and 4)auto bond1iface bond1 inet manual bond-slaves eth1 eth3 bond-mode active-backup bond-miimon 100 bond-downdelay 250 bond-updelay 250

# Container management VLAN interfaceiface bond0.10 inet manual vlan-raw-device bond0

# OpenStack Networking VXLAN (tunnel/overlay) VLAN interfaceiface bond1.30 inet manual vlan-raw-device bond1

# Storage network VLAN interface (optional)iface bond0.20 inet manual vlan-raw-device bond0

# Container management bridgeauto br-mgmtiface br-mgmt inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references tagged interface bridge_ports bond0.10 address 172.29.236.11 netmask 255.255.252.0 dns-nameservers 69.20.0.164 69.20.0.196

# OpenStack Networking VXLAN (tunnel/overlay) bridgeauto br-vxlaniface br-vxlan inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references tagged interface bridge_ports bond1.30 address 172.29.240.11 netmask 255.255.252.0

# OpenStack Networking VLAN bridgeauto br-vlaniface br-vlan inet manual



26

bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references untagged interface bridge_ports bond1

# Storage bridge (optional)auto br-storageiface br-storage inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port reference tagged interface bridge_ports bond0.20 address 172.29.244.11 netmask 255.255.252.0

In non-Rackspace data centers, the service network configuration should be comment-ed out of /etc/rpc_deploy/rpc_user_config.yml. The commented-out sectionshould appear as follows:

# Cidr used in the Service network# snet: 172.29.248.0/22

#- network:# group_binds:# - glance_api# - nova_compute# - neutron_linuxbridge_agent# type: "raw"# container_bridge: "br-snet"# container_interface: "eth3"# ip_from_q: "snet"



27

5. Deployment configurationFigure 5.1. Installation workflow

Ansible references a handful of files containing mandatory and optional configuration di-rectives. These files must be modified to define the target environment before running theAnsible playbooks. Perform the following tasks:

• Configure Target host networking to define bridge interfaces and networks.

• Configure a list of target hosts on which to install the software.

• Configure virtual and physical network relationships for OpenStack Networking (neu-tron).

• Configure passwords for all services.

• (Optional) Configure the hypervisor.

• (Optional) Configure Block Storage (cinder) to use the NetApp back end.

• (Optional) Configure Block Storage (cinder) backups.

• (Optional) Create Block Storage availability zones.

5.1. PrerequisitesCopy the contents of the /opt/openstack-ansible/etc/rpc_deploy directory tothe /etc/rpc_deploy directory.

# cp -R /opt/openstack-ansible/etc/rpc_deploy /etc

5.2. Configuring target host networkingModify the /etc/rpc_deploy/rpc_user_config.yml file to configure networking.



28

1. Configure the IP address ranges associated with each network in the cidr_networkssection:

cidr_networks: # Container management network container: CONTAINER_MGMT_CIDR # Tunnel network tunnel: TUNNEL_CIDR # Storage network (optional) storage: STORAGE_CIDR

Replace *_CIDR with the appropriate IP address range in CIDR notation. For example,203.0.113.0/24.

Note

Use the same IP address ranges as the underlying physical network in-terfaces or bridges configured in Section 4.5, “Configuring the net-work” [18]. For example, if the container network uses 203.0.113.0/24, theCONTAINER_MGMT_CIDR should also use 203.0.113.0/24.

The default configuration includes the optional storage and service net-works. To remove one or both of them, comment out the appropriate net-work name.

2. Configure the existing IP addresses in the used_ips section:

used_ips: - EXISTING_IP_ADDRESSES

Replace EXISTING_IP_ADDRESSES with a list of existing IP addresses in the rangesdefined in the previous step. This list should include all IP addresses manually config-ured on target hosts in the Section 4.5, “Configuring the network” [18], internal loadbalancers, service network bridge, and any other devices to avoid conflicts during theautomatic IP address generation process.

Note

Add individual IP addresses on separate lines. For example, to prevent useof 203.0.113.101 and 201:

used_ips: - 203.0.113.101 - 203.0.113.201

Add a range of IP addresses using a comma. For example, to prevent use of203.0.113.101-201:

used_ips: - 203.0.113.101, 203.0.113.201

3. Configure load balancing in the global_overrides section:



29

global_overrides: # Internal load balancer VIP address internal_lb_vip_address: INTERNAL_LB_VIP_ADDRESS # External (DMZ) load balancer VIP address external_lb_vip_address: EXTERNAL_LB_VIP_ADDRESS # Load balancer hostname lb_name: LB_HOSTNAME # Container network bridge device management_bridge: "MGMT_BRIDGE" # Tunnel network bridge device tunnel_bridge: "TUNNEL_BRIDGE"

Replace INTERNAL_LB_VIP_ADDRESS with the internal IP address of the load bal-ancer. Infrastructure and OpenStack services use this IP address for internal communi-cation.

Replace EXTERNAL_LB_VIP_ADDRESS with the external, public, or DMZ IP addressof the load balancer. Users primarily use this IP address for external API and web inter-faces access.

Replace LB_HOSTNAME with the hostname of the load balancer that resolves to the ex-ternal, public, or DMZ IP address of the load balancer.

Replace MGMT_BRIDGE with the container bridge device name, typically br-mgmt.

Replace TUNNEL_BRIDGE with the tunnel/overlay bridge device name, typically br-vxlan.

4. Configure the management network in the provider_networks subsection:

provider_networks: - network: group_binds: - all_containers - hosts type: "raw" container_bridge: "br-mgmt" container_interface: "eth1" ip_from_q: "container"

5. Configure optional networks in the provider_networks subsection:

provider_networks: - network: group_binds: - glance_api - cinder_api - cinder_volume - nova_compute type: "raw" container_bridge: "br-storage" container_interface: "eth2" ip_from_q: "storage"



30

Note

The default configuration includes one or more optional networks. To re-move any of them, comment out the entire associated stanza beginningwith the - network: line.

6. Configure OpenStack Networking VXLAN tunnel/overlay networks in theprovider_networks subsection:

provider_networks: - network: group_binds: - neutron_linuxbridge_agent container_bridge: "br-vxlan" container_interface: "eth10" ip_from_q: "tunnel" type: "vxlan" range: "TUNNEL_ID_RANGE" net_name: "vxlan"

Replace TUNNEL_ID_RANGE with the tunnel ID range. For example, 1:1000.

7. Configure OpenStack Networking flat (untagged) and VLAN (tagged) networks in theprovider_networks subsection:

provider_networks: - network: group_binds: - neutron_linuxbridge_agent container_bridge: "br-vlan" container_interface: "eth11" host_bind_override: "eth12" type: "flat" net_name: "vlan" - network: group_binds: - neutron_linuxbridge_agent container_bridge: "br-vlan" container_interface: "eth11" type: "vlan" range: VLAN_ID_RANGE net_name: "vlan"

Replace VLAN_ID_RANGE with the VLAN ID range for each VLAN provider network.For example, 1:1000. Supports more than one range of VLANs on a particular networkusing a special construction. For example, to use VLANs 1-1000 and 2001-3000:

range: '1:1000,vlan:2001:3000'

Create a similar stanza for each additional network.

5.3. Configuring target hostsModify the /etc/rpc_deploy/rpc_user_config.yml file to configure the targethosts.



31

Warning

Do not assign the same IP address to different target hostnames. Unexpectedresults may occur. Each IP address and hostname must be a matching pair. Touse the same host in multiple roles, for example infrastructure and networking,specify the same hostname and IP in each section.

Use short hostnames rather than fully-qualified domain names (FQDN) to pre-vent length limitation issues with LXC and SSH. For example, a suitable shorthostname for a compute host might be: 123456-Compute001.

1. Configure a list containing at least three infrastructure target hosts in theinfra_hosts section:

infra_hosts: 603975-infra01: ip: INFRA01_IP_ADDRESS 603989-infra02: ip: INFRA02_IP_ADDRESS 627116-infra03: ip: INFRA03_IP_ADDRESS 628771-infra04: ...

Replace *_IP_ADDRESS with the IP address of the br-mgmt container managementbridge on each infrastructure target host. Use the same net block as bond0 on thenodes, for example:

infra_hosts: 603975-infra01: ip: 10.240.0.80 603989-infra02: ip: 10.240.0.81 627116-infra03: ip: 10.240.0.184

2. Configure a list containing at least one network target host in the network_hostssection:

network_hosts: 602117-network01: ip: NETWORK01_IP_ADDRESS 602534-network02: ...

Replace *_IP_ADDRESS with the IP address of the br-mgmt container managementbridge on each network target host.

3. Configure a list containing at least one compute target host in the compute_hostssection:

compute_hosts: 900089-compute001: ip: COMPUTE001_IP_ADDRESS 900090-compute002: ...

Replace *_IP_ADDRESS with the IP address of the br-mgmt container managementbridge on each compute target host.



32

4. Configure a list containing at least one logging target host in the log_hosts section:

log_hosts: 900088-logging01: ip: LOGGER1_IP_ADDRESS 903877-logging02: ...

Replace *_IP_ADDRESS with the IP address of the br-mgmt container managementbridge on each logging target host.

5. Configure a list containing at least one optional storage host in the storage_hostssection:

storage_hosts: 100338-storage01: ip: STORAGE01_IP_ADDRESS 100392-storage02: ...

Replace *_IP_ADDRESS with the IP address of the br-mgmt container managementbridge on each storage target host. Each storage host also requires additional configu-ration to define the back end driver.

Note

The default configuration includes an optional storage host. To installwithout storage hosts, comment out the stanza beginning with thestorage_hosts: line.

5.4. Configuring service passwordsChange the default password for all services in the /etc/rpc_deploy/user_variables.yml file. Consider using Ansible Vault to increase security by encrypt-ing this file.

Note that the following options configure passwords for the web interfaces:

• keystone_auth_admin_password configures the admin tenant password for boththe OpenStack API and dashboard access.

• kibana_password configures the password for Kibana web interface access.

The openstack-ansible repository provides a script to generate random passwords for eachservice. For example:

# cd /opt/openstack-ansible/scripts# python pw-token-gen.py --file /etc/rpc_deploy/user_variables.yml

To regenerate existing passwords, add the --regen flag.

http://docs.ansible.com/playbooks_vault.html



33

5.5. Specifying additional groups for the systemuser (optional)

You can specify additional groups that the system_user is added to in each container.The default is set to adm in the /opt/openstack-ansible/rpc_deployment/in-ventory/group_vars/all.yml file, but can be overridden.

# groups to add the system_user to in each container.additional_system_groups: - adm

For an example of overriding, refer to the /opt/openstack-ansi-ble/rpc_deployment/inventory/group_vars/nova_all.yml file. In this contain-er, the system_user is also added to the libvirtd group:

additional_system_groups: - adm - libvirtd

For more information, see github.com/openstack/openstack-ansible/blob/juno/rpc_deployment/inventory/group_vars/nova_all.yml#L52-L54 and openstack-ansible bug1442366.

5.6. Configuring the is_metal flag (optional)When the is_metal flag is set to TRUE, for cinder, the cinder-volume service is deployedon bare metal. When set to FALSE, the service is deployed to a container. All cinder back-end drivers that use iSCSI, Fibre Channel, or any other hardware-dependent protocol shouldbe running inside a cinder-volume service that is deployed on bare metal. For nova computeand swift storage containers, the default setting for the flag is TRUE.

5.7. Configuring the hypervisor (optional)By default, the KVM hypervisor is used. If you are deploying to a host that does not supportKVM hardware acceleration extensions, select a suitable hypervisor type such as qemu orlxc. To change the hypervisor type, uncomment and edit the following line in the /etc/rpc_deploy/user_variables.yml file:

# nova_virt_type: kvm

5.8. Configuring the Image Service (optional)In an all-in-one deployment with a single infrastructure node, the Image Service uses the lo-cal file system on the target host to store images. In a Rackspace deployment with three in-frastructure nodes, the Image Service must use Cloud Files or NetApp. The following proce-dure describes how to modify the /etc/rpc_deploy/user_variables.yml file to en-able Cloud Files usage.

https://github.com/openstack/openstack-ansible/blob/juno/rpc_deployment/inventory/group_vars/nova_all.yml#L52-L54

https://github.com/openstack/openstack-ansible/blob/juno/rpc_deployment/inventory/group_vars/nova_all.yml#L52-L54

https://bugs.launchpad.net/openstack-ansible/+bug/1442366

https://bugs.launchpad.net/openstack-ansible/+bug/1442366



34

1. Change the default store to use swift, the underlying architecture of Cloud Files:

glance_default_store: swift

2. Set the appropriate authentication URL:

For US Rackspace cloud accounts:

rackspace_cloud_auth_url: https://identity.api.rackspacecloud.com/v2.0

For UK Rackspace cloud accounts:

rackspace_cloud_auth_url: https://lon.identity.api.rackspacecloud.com/v2.0

3. Set Rackspace cloud account credentials by locating the RAX_CLOUD_TENANT_ID:

a. Log into mycloud.rackspace.com as the desired user.

b. Locate and click on the Account: $USERNAME link in the upper right section of thescreen.

c. Copy the Account Number shown.

4. Set the remaining Rackspace cloud account credentials with theRAX_CLOUD_TENANT_ID:

rackspace_cloud_tenant_id: RAX_CLOUD_TENANT_IDrackspace_cloud_username: RAX_CLOUD_USER_NAMErackspace_cloud_password: RAX_CLOUD_PASSWORD

5. Change the glance_swift_store_endpoint_type from the default in-ternalURL settings to public if needed. Glance services are typically backed byRackspace cloud files in the Rackspace Data Center. If the OpenStack environmentmust run outside the data center, adjust the key value:

glance_swift_store_endpoint_type: publicURL

6. Replace RAX_CLOUD_* with the appropriate Rackspace cloud account credential com-ponents.

7. Define the store name:

glance_swift_store_container: STORE_NAME

Replace STORE_NAME with the store name in Cloud Files. If the store doesn't exist, anew store will be created.

8. Define the store region:

glance_swift_store_region: STORE_REGION

Replace STORE_REGION with one of the following region codes: DFW, HKG, IAD,LON, ORD, SYD.



35

Note

UK Rackspace cloud accounts must use the LON region. US Rackspace cloudaccounts can use any region except LON.

9. (Optional) Set the paste deploy flavor:

glance_flavor: GLANCE_FLAVOR

By default, the Image service uses caching and authenticates with the Identity service.The default maximum size of the image cache is 10 GB. The default Image service con-tainer size is 12 GB. In some configurations, the Image service might attempt to cachean image which exceeds the available disk space. If necessary, you can disable caching.For example, to use Identity without caching, replace GLANCE_FLAVOR with key-stone:

glance_flavor: keystone

Or, to disable both authentication and caching, set GLANCE_FLAVOR to no value:

glance_flavor:

The possible values for GLANCE_FLAVOR are:

• (Nothing)

• caching

• cachemanagement

• keystone

• keystone+caching

• keystone+cachemanagement (default)

• trusted-auth

• trusted-auth+cachemanagement

5.9. Configuring the Block Storage service (option-al)

Block Storage (cinder) provides persistent storage to guest instances. The service managesvolumes and snapshots.

5.9.1. Configuring the Block Storage service on bare metal

Block Storage can be installed on bare metal instead of in a container. Installing Block Stor-age on bare metal can help facilitate future upgrades.



36

1. Modify the /etc/rpc_deploy/rpc_environment.yml file. Under thecontainer_skel: section, within cinder_volumes_container:, setis_metal: to true.

container_skel: # other containers cinder_volumes_container: is_metal: true belongs_to: - storage_containers contains: - cinder_scheduler - cinder_volume # other containers

2. Modify the /etc/rpc_deploy/rpc_user_config.yml file. Setenvironment_version: to the value returned by md5sum /etc/rpc_deploy/rpc_environment.yml

3. Generate an inventory.

# /opt/openstack-ansible/rpc_deployment/inventory# ./dynamic-inventory.py --file /etc/rpc_deploy/rpc_user_config.yml

This creates the /etc/rpc_deploy/rpc_inventory.json file, which contains theinventory.

4. Open the /etc/rpc_deploy/rpc_inventory.json file to find the br-storagevalues (storage_address and storage_netmask) assigned to the node on whichcinder-volumes is running. Search on "component": "cinder_volume" to findthe correct section.

"storage01": { "ansible_ssh_host": "172.24.240.8", "component": "cinder_volume", "container_address": "172.24.240.8", "container_name": "storage01", "container_netmask": "255.255.252.0", "container_network": { "container_bridge": "br-mgmt", "container_interface": "eth1", "container_netmask": "255.255.252.0" }, "container_types": "storage01_containers", "is_metal": true, "physical_host": "storage01", "storage_address": "172.24.247.73", "storage_netmask": "255.255.252.0" },

5. Edit the /etc/network/interfaces file to add the values for br-storage on thenode on which cinder-volume is installed on bare metal. Add the following stanzato the /etc/network/interfaces file:

iface br-storage inet static address STORAGE_ADDRESS netmask STORAGE_NETMASK

6. Run the following commands to bring the br-storage network down and back up:



37

ifdown br-storage ; ifup br-storage

7. After the network is up, verify that the IP address is on the bridge.

8. Repeat steps four through seven for all nodes on which cinder-volume is installedon bare metal.

9. Verify that the networks were added.

# ip a l | grep br-storage

10.

5.9.2. Configuring the Block Storage service in a container

By default, the Block Storage service uses the LVM backend. To use a NetApp storage appli-ance backend, edit the /etc/rpc_deploy/rpc_user_config.yml file and configureeach storage node that will use it:

Note

Ensure that the NAS Team enables httpd.admin.access.

1. Add the netapp stanza under the cinder_backends stanza for each storage node:

cinder_backends: netapp:

The options in subsequent steps fit under the netapp stanza.

Note

The back end name is arbitrary and becomes a volume type within theBlock Storage service.

2. Configure the storage family:

netapp_storage_family: STORAGE_FAMILY

Replace STORAGE_FAMILY with ontap_7mode for Data ONTAP operating in 7-modeor ontap_cluster for Data ONTAP operating as a cluster.

3. Configure the storage protocol:

netapp_storage_protocol: STORAGE_PROTOCOL

Replace STORAGE_PROTOCOL with iscsi for iSCSI or nfs for NFS.

For the NFS protocol, you must also specify the location of the configuration file thatlists the shares available to the Block Storage service:

nfs_shares_config: SHARE_CONFIG

Replace SHARE_CONFIG with the location of the share configuration file. For example,/etc/cinder/nfs_shares.



38

4. Configure the server:

netapp_server_hostname: SERVER_HOSTNAME

Replace SERVER_HOSTNAME with the hostnames for both netapp controllers.

5. Configure the server API port:

netapp_server_port: PORT_NUMBER

Replace PORT_NUMBER with 80 for HTTP or 443 for HTTPS.

6. Configure the server credentials:

netapp_login: USER_NAMEnetapp_password: PASSWORD

Replace USER_NAME and PASSWORD with the appropriate values.

7. Select the NetApp driver:

volume_driver: cinder.volume.drivers.netapp.common.NetAppDriver

8. Configure the volume back end name:

volume_backend_name: BACKEND_NAME

Replace BACKEND_NAME with a suitable value that provides a hint for the Block Stor-age scheduler. For example, NETAPP_iSCSI.

9. Check that the rpc_user_config.yml configuration is accurate:

storage_hosts: xxxxxx-Infra01: ip: 172.29.236.16 container_vars: cinder_backends: limit_container_types: cinder_volume netapp: netapp_storage_family: ontap_7mode netapp_storage_protocol: nfs netapp_server_hostname: 111.222.333.444 netapp_server_port: 80 netapp_login: rpc_cinder netapp_password: password volume_driver: cinder.volume.drivers.netapp.common.NetAppDriver volume_backend_name: NETAPP_NFS

For netapp_server_hostname, specify the IP address of the Data ONTAP server.Include iSCSI or NFS for the netapp_storage_family depending on the configura-tion. Add 80 if using HTTP or 443 if using HTTPS for netapp_server_port.

The cinder-volume.yml playbook will automatically install the nfs-common fileacross the hosts, transitioning from an LVM to a NetApp back end.



39

5.9.3. Configure the Block Storage service with NFS proto-cols

If the NetApp back end is configured to use an NFS storage protocol, edit /etc/rpc_deploy/rpc_user_config.yml, and configure the NFS client on each storagenode that will use it.

1. Add the nfs_client stanza under the container_vars stanza for each storagenode:

container_vars:nfs_client:

2. Configure the location of the file that lists shares available to the block storage service.This configuration file must include nfs_shares_config:

nfs_shares_config: SHARE_CONFIG

Replace SHARE_CONFIG with the location of the share configuration file. For example,/etc/cinder/nfs_shares.

3. Configure one or more NFS shares:

shares: - { ip: “NFS_HOST”, share: “NFS_SHARE” }

Replace NFS_HOST with the IP address or hostname of the NFS server, and theNFS_SHARE with the absolute path to an existing and accessible NFS share.

5.9.4. Configure Block Storage backups to Object StorageYou can configure Block Storage (cinder) to back up volumes to Object Storage (swift) bysetting configuration variables and running a specific playbook. The default configurationbacks up volumes to an Object Storage installation accessible within your environment. Al-ternatively, you can set cinder_service_backup_swift_url and other variables list-ed below to back up to an external Object Storage installation.

1. Add or edit the following line in the /etc/rpc_deploy/user_variables.ymlfile:

cinder_service_backup_program_enabled: true

2. By default, Block Storage will use the access credentials of the user initiating thebackup. You can modify configuration variables to change how Block Storage per-forms backups. Edit any of the following variables in the /etc/rpc_deploy/user_variables.yml file:

## Backups...# cinder_service_backup_swift_auth: Options include 'per_user' or 'single_user', we default to# 'per_user' so that backups are saved to a user's swift account.cinder_service_backup_swift_auth: per_user# cinder_service_backup_swift_url: This is your swift storage url when using 'per_user', or keystone



40

# endpoint when using 'single_user'. When using 'per_user', you# can leave this as empty or as None to allow cinder-backup to# obtain storage url from environment.cinder_service_backup_swift_url:cinder_service_backup_swift_user:cinder_service_backup_swift_key:cinder_service_backup_swift_tenant:cinder_service_backup_swift_auth_version: 2cinder_service_backup_swift_container: volumebackupscinder_service_backup_swift_object_size: 52428800cinder_service_backup_swift_retry_attempts: 3cinder_service_backup_swift_retry_backoff: 2cinder_service_backup_compression_algorithm: zlibcinder_service_backup_metadata_version: 1

To deploy the Block Storage backup service, follow the procedure in Section 8.3, “Set upBlock Storage backups (optional)” [50]. For more information about installing swift, re-fer to the Object Storage Deployment Guide.

5.9.5. Creating Block Storage availability zonesMultiple availability zones can be created to manage Block Storage storage hosts. Edit the/etc/rpc_deploy/rpc_user_config.yml file to set up availability zones.

1. For each cinder storage host, configure the availability zone under thecontainer_vars stanza:

cinder_storage_availability_zone: CINDERAZ

Replace CINDERAZ with a suitable name. For example cinderAZ_2

2. If more than one availability zone is created, configure the default availability zone forscheduling volume creation:

cinder_default_availability_zone: CINDERAZ_DEFAULT

Replace CINDERAZ_DEFAULT with a suitable name. For example, cinderAZ_1. Thedefault availability zone should be the same for all Cinder storage hosts.

Note

If the cinder_default_availability_zone is not defined, the de-fault variable will be used.

5.10. Configuring Active Directory or LDAPThese instructions require an understanding of OpenStack Identity (Keystone) core con-cepts, including users, roles, tenants, and tokens, and a working knowledge of Active Direc-tory and LDAP.

Active Directory prerequisites

• A working Active Directory server.



41

• The address, hostname, or URL of the Active Directory server, such asactiveserver.example.com.

• An AD/LDAP bind-user user with sufficient privileges to search all of the organizationalunits that are defined within the environment. The user should have the minimum privi-leges sufficient for binding to the directory and for searching and reading the directoryentities that are exposed to the RPC identity services, and should not have the privilegesof AD Administrator.

LDAP prerequisites

• A working LDAP server with OpenLDAP or 389 Directory Server installed. The top levelorganization should already be created, as in the following example:

dn: cn=example,cn=com dc: example objectClass: dcObject objectClass: organizationalUnit o: example.com

• The address, hostname, or URL of the LDAP server, such asldapserver.example.com.

• A configured LDAP bind-user user with sufficient privileges to bind to the directory andto read and search all of the directory entities exposed to RPC identity services, as in thefollowing example:

cn=admin,dc=example,dc=com

This user should NOT have the privileges of LDAP server administrator.

LDAP users and attributes consumed by RPC identity services must be created by means ofnative AD or LDAP tools, as RPC does not provide an itnerface to create or edit LDAP users,attributes, or groups.

For more information about OpenStack Identity integration with Active Directory andLDAP, refer to the following OpenStack documentation.

• How to integrate Keystone with Active Directory

• Integrate Identity with LDAP

5.10.1. Adding LDAP/Active Directory

The following procedure describes how to modify the /etc/rpc_deploy/user_variables.yml file to enable Active Directory or LDAP.

Procedure 5.1. Adding LDAP/Active Directory to an environment

1. On the Keystone containers, install python-ldap and ldappool.

https://wiki.openstack.org/wiki/HowtoIntegrateKeystonewithAD

http://docs.openstack.org/admin-guide-cloud/content/configuring-keystone-for-ldap-backend.html



42

$ apt-get install python-ldap$ pip install ldappool

2. In LDAP/Active Directory, create users for the following OpenStack services and assignpasswords to them.

Note

The UID or CN of these users must match the corresponding OpenStackUUID for each user. This ensures that the roles are correctly mapped.

• admin

• cinder

• cinderv2

• ec2

• glance

• heat

• keystone

• neutron

• nova

• novav3

• s3

The admin user will use the OpenStack admin user password.

3. Open /etc/rpc_deploy/user_variables.yml.

4. Locate the keystone_identity_driver variable and change it as follows:

keystone_identity_driver: "keystone.identity.backends.ldap.Identity"

5. Edit the [ldap] section as needed. Refer to the appropriate configuration section fordetailed information about the required configurations:

• Section 5.10.2, “LDAP Directory configuration” [42]

• Section 5.10.3, “Active Directory configuration” [43]

5.10.2. LDAP Directory configuration

The following variables must be set in /etc/rpc_deploy/user_variables.yml.



43

keystone_ldap: Truekeystone_ldap_identity_driver: keystone.identity.backends.ldap.Identitykeystone_ldap_domain_config_dir: /etc/keystone/domainskeystone_ldap_page_size: 2000keystone_ldap_query_scope: subkeystone_ldap_server: IP of LDAP Serverkeystone_ldap_suffix: OU=Users,DC=example,DC=comkeystone_ldap_user_allow_create: falsekeystone_ldap_user_allow_delete: falsekeystone_ldap_user_allow_update: falsekeystone_ldap_user_attribute_ignore: password,tenantId,tenantskeystone_ldap_user_bind: UID=rpc-ldap-bind,OU=Service Accounts,DC=example,DC=comkeystone_ldap_user_bind_password: Passwordkeystone_ldap_user_enabled_attribute: userAccountControlkeystone_ldap_user_enabled_default: 512keystone_ldap_user_enabled_mask: 2keystone_ldap_user_filter: ''keystone_ldap_user_id_attribute: uidkeystone_ldap_user_mail_attribute: mailkeystone_ldap_user_name_attribute: displayNamekeystone_ldap_user_objectclass: inetOrgPersonkeystone_ldap_user_tree_dn: OU=Users,DC=example,DC=comkeystone_ldap_tls_req_cert: allow

5.10.3. Active Directory configuration

The following variables must be set in /etc/rpc_deploy/user_variables.yml.

keystone_ldap: Truekeystone_ldap_identity_driver: keystone.identity.backends.ldap.Identitykeystone_ldap_domain_config_dir: /etc/keystone/domainskeystone_ldap_page_size: 2000keystone_ldap_query_scope: subkeystone_ldap_server: IP of Active Directory Serverkeystone_ldap_suffix: OU=Users,DC=example,DC=comkeystone_ldap_user_allow_create: falsekeystone_ldap_user_allow_delete: falsekeystone_ldap_user_allow_update: falsekeystone_ldap_user_attribute_ignore: password,tenantId,tenantskeystone_ldap_user_bind: CN=rpc-ldap-bind,OU=Service Accounts,DC=example,DC=comkeystone_ldap_user_bind_password: Passwordkeystone_ldap_user_enabled_attribute: userAccountControlkeystone_ldap_user_enabled_default: 512keystone_ldap_user_enabled_mask: 2keystone_ldap_user_filter: ''keystone_ldap_user_id_attribute: cnkeystone_ldap_user_mail_attribute: mailkeystone_ldap_user_name_attribute: sAMAccountNamekeystone_ldap_user_objectclass: personkeystone_ldap_user_tree_dn: OU=Users,DC=example,DC=comkeystone_ldap_tls_req_cert: allow



44

6. Foundation playbooksFigure 6.1. Installation workflow

The main Ansible foundation playbook prepares the target hosts for infrastructure andOpenStack services and performs the following operations:

• Perform deployment host initial setup

• Build containers on target hosts

• Restart containers on target hosts

• Install common components into containers on target hosts

6.1. Running the foundation playbook1. Change to the /opt/openstack-ansible/rpc_deployment directory.

2. Run the host setup playbook, which runs a series of sub-playbooks:

$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/setup/host-setup.yml

Confirm satisfactory completion with zero items unreachable or failed:

PLAY RECAP ********************************************************************...deployment_host : ok=18 changed=11 unreachable=0 failed=0

6.2. TroubleshootingQ: How do I resolve the following error after running a playbook?

failed: [target_host] => (item=target_host_horizon_container-69099e06) =>{"err": "lxc-attach: No such file or directory - failed to open



45

'/proc/12440/ns/mnt'\nlxc-attach: failed to enter the namespace\n", "failed":true, "item": "target_host_horizon_container-69099e06", "rc": 1}msg: Failed executing lxc-attach.

A: The lxc-attach sometimes fails to execute properly. This issue can be resolved by run-ning the playbook again.



46

7. Infrastructure playbooksFigure 7.1. Installation workflow

The main Ansible infrastructure playbook installs infrastructure services and performs thefollowing operations:

• Install Memcached

• Install Galera

• Install RabbitMQ

• Install Rsyslog

• Install Elasticsearch

• Install Logstash

• Install Kibana

• Install Elasticsearch command-line utilities

• Configure Rsyslog

7.1. Running the infrastructure playbook1. Change to the /opt/openstack-ansible/rpc_deployment directory.

2. Run the infrastructure setup playbook, which runs a series of sub-playbooks:

$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/infrastructure/infrastructure-setup.yml


PLAY RECAP ********************************************************************...



47

deployment_host : ok=27 changed=0 unreachable=0 failed=0

7.2. Verifying infrastructure operationVerify the database cluster and Kibana web interface operation.

Procedure 7.1. Verifying the database cluster

1. Determine the Galera container name:

$ lxc-ls | grep galerainfra1_galera_container-4ed0d84a

2. Access the Galera container:

$ lxc-attach -n infra1_galera_container-4ed0d84a

3. Run the MariaDB client, show cluster status, and exit the client:

$ mysql -u root -pMariaDB> show status like 'wsrep_cluster%';+--------------------------+--------------------------------------+| Variable_name | Value |+--------------------------+--------------------------------------+| wsrep_cluster_conf_id | 3 || wsrep_cluster_size | 3 || wsrep_cluster_state_uuid | bbe3f0f6-3a88-11e4-bd8f-f7c9e138dd07 || wsrep_cluster_status | Primary |+--------------------------+--------------------------------------+MariaDB> exit

The wsrep_cluster_size field should indicate the number of nodes in the clusterand the wsrep_cluster_status field should indicate primary.

Procedure 7.2. Verifying the Kibana web interface

1. With a web browser, access the Kibana web interface using the external load bal-ancer IP address defined by the external_lb_vip_address option in the /etc/rpc_deploy/rpc_user_config.yml file. The Kibana web interface uses HTTPS onport 8443.

2. Authenticate using the username kibana and password defined by thekibana_password option in the /etc/rpc_deploy/user_variables.yml file.



48

8. OpenStack playbooksFigure 8.1. Installation work flow

The main Ansible OpenStack playbook installs OpenStack services and performs the follow-ing operations:

• Install common components

• Create utility container that provides utilities to interact with services in other containers

• Install Identity (keystone)

• Generate service IDs for all services

• Install the Image Service (glance)

• Install Orchestration (heat)

• Install Compute (nova)

• Install Networking (neutron)

• Install Block Storage (cinder)

• Install Dashboard (horizon)

• Reconfigure Rsyslog

8.1. Utility Container OverviewThe utility container provides a space where miscellaneous tools and other software can beinstalled. Tools and objects can be placed in a utility container if they do not require a ded-icated container or if it is impractical to create a new container for a single tool or object.Utility containers can also be used when tools cannot be installed directly onto a host.

For example, the tempest playbooks are installed on the utility container since tempest test-ing does not need a container of its own. For another example of using the utility contain-er, see Section 8.4, “Verifying OpenStack operation” [50].



49

8.2. Running the OpenStack playbook1. Change to the /opt/openstack-ansible/rpc_deployment directory.

2. Run the OpenStack setup playbook, which runs a series of sub-playbooks:

$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/openstack/openstack-setup.yml

Note

The openstack-common.yml sub-playbook builds all OpenStack servicesfrom source and takes up to 30 minutes to complete. As the playbook pro-gresses, the quantity of containers in the "polling" state will approach zero.If any operations take longer than 30 minutes to complete, the playbookwill terminate with an error.

changed: [target_host_glance_container-f2ebdc06]changed: [target_host_heat_engine_container-36022446]changed: [target_host_neutron_agents_container-08ec00cd]changed: [target_host_heat_apis_container-4e170279]changed: [target_host_keystone_container-c6501516]changed: [target_host_neutron_server_container-94d370e5]changed: [target_host_nova_api_metadata_container-600fe8b3]changed: [target_host_nova_compute_container-7af962fe]changed: [target_host_cinder_api_container-df5d5929]changed: [target_host_cinder_volumes_container-ed58e14c]changed: [target_host_horizon_container-e68b4f66]<job 802849856578.7262> finished on target_host_heat_engine_container-36022446<job 802849856578.7739> finished on target_host_keystone_container-c6501516<job 802849856578.7262> finished on target_host_heat_apis_container-4e170279<job 802849856578.7359> finished on target_host_cinder_api_container-df5d5929<job 802849856578.7386> finished on target_host_cinder_volumes_container-ed58e14c<job 802849856578.7886> finished on target_host_horizon_container-e68b4f66<job 802849856578.7582> finished on target_host_nova_compute_container-7af962fe<job 802849856578.7604> finished on target_host_neutron_agents_container-08ec00cd<job 802849856578.7459> finished on target_host_neutron_server_container-94d370e5<job 802849856578.7327> finished on target_host_nova_api_metadata_container-600fe8b3<job 802849856578.7363> finished on target_host_glance_container-f2ebdc06<job 802849856578.7339> polling, 1675s remaining<job 802849856578.7338> polling, 1675s remaining<job 802849856578.7322> polling, 1675s remaining<job 802849856578.7319> polling, 1675s remaining



50

Note

Setting up the compute hosts takes up to 30 minutes to complete, particu-larly in environments with many compute hosts. As the playbook progress-es, the quantity of containers in the "polling" state will approach zero. Ifany operations take longer than 30 minutes to complete, the playbook willterminate with an error.

ok: [target_host_nova_conductor_container-2b495dc4]ok: [target_host_nova_api_metadata_container-600fe8b3]ok: [target_host_nova_api_ec2_container-6c928c30]ok: [target_host_nova_scheduler_container-c3febca2]ok: [target_host_nova_api_os_compute_container-9fa0472b]<job 409029926086.9909> finished on target_host_nova_api_os_compute_container-9fa0472b<job 409029926086.9890> finished on target_host_nova_api_ec2_container-6c928c30<job 409029926086.9910> finished on target_host_nova_conductor_container-2b495dc4<job 409029926086.9882> finished on target_host_nova_scheduler_container-c3febca2<job 409029926086.9898> finished on target_host_nova_api_metadata_container-600fe8b3<job 409029926086.8330> polling, 1775s remaining


PLAY RECAP **********************************************************************...deployment_host : ok=44 changed=11 unreachable=0 failed=0

8.3. Set up Block Storage backups (optional)If configured in Section 5.9.4, “Configure Block Storage backups to Object Storage” [39],Block Storage (cinder) backs up volumes to Object Storage (swift). For more informationabout installing swift, refer to the Object Storage Deployment Guide.

1. Change to the /opt/openstack-ansible/rpc_deployment directory.

2. Run the cinder-backup playbook to deploy the cinder-backup service in thecinder-volume container:

$ ansible-playbook -e \ @/etc/rpc_deploy/user_variables.yml \ playbooks/openstack/cinder-backup.yml

8.4. Verifying OpenStack operationVerify basic operation of the OpenStack API and dashboard.



51

Procedure 8.1. Verifying the API

The utility container provides a command line (CLI) environment for additional configura-tion and testing.

1. Determine the utility container name:

$ lxc-ls | grep utilityinfra1_utility_container-161a4084

2. Access the utility container:

$ lxc-attach -n infra1_utility_container-161a4084

3. Source the admin tenant credentials:

$ source openrc

4. Run an OpenStack command that uses one or more APIs. For example:

$ keystone user-list+----------------------------------+----------+---------+-------+| id | name | enabled | email |+----------------------------------+----------+---------+-------+| 090c1023d0184a6e8a70e26a5722710d | admin | True | || 239e04cd3f7d49929c7ead506d118e40 | cinder | True | || e1543f70e56041679c013612bccfd4ee | cinderv2 | True | || bdd2df09640e47888f819057c8e80f04 | demo | True | || 453dc7932df64cc58e36bf0ac4f64d14 | ec2 | True | || 257da50c5cfb4b7c9ca8334bc096f344 | glance | True | || 6e0bc047206f4f5585f7b700a8ed6e94 | heat | True | || 187ee2e32eec4293a3fa243fa21f6dd9 | keystone | True | || dddaca4b39194dc4bcefd0bae542c60a | neutron | True | || f1c232f9d53c4adabb54101ccefaefce | nova | True | || fdfbda23668c4980990708c697384050 | novav3 | True | || 744069c771d84f1891314388c1f23686 | s3 | True | || 4e7fdfda8d14477f902eefc8731a7fdb | swift | True | |+----------------------------------+----------+---------+-------+

Procedure 8.2. Verifying the dashboard

1. With a web browser, access the dashboard using the external load balancer IP addressdefined by the external_lb_vip_address option in the /etc/rpc_deploy/rpc_user_config.yml file. The dashboard uses HTTPS on port 443.

2. Authenticate using the username admin and password defined by thekeystone_auth_admin_password option in the /etc/rpc_deploy/user_variables.yml file.

Note

Uploading public images using the dashboard or CLI can only be performed byusers with administrator privileges.



52

9. Rackspace RPC Solutions tabRPC has developed a suite of free solution templates for widely used open source applica-tions. Launching a template creates a production-ready application stack. These templatesreside in a new Solutions tab available in the horizon Dashboard.

To install the Solutions tab, contact Rackspace Support.

9.1. Launch a solutionRPC provides templates for the following solutions:

Drupal An open source content management system.

ELK Stack A stack used for log aggregation and analytics.

Galera An open source database cluster solution for MySQL.

Gerrit CI (Continuous Integra-tion)

A code review system that provides a web user interfaceon top of the Git version control system.

Gitlab CE (Community Edition) An open source web-based Git repository manager withwiki and issue tracking features.

Hortonworks HDP (Hornton-works Data Platform)

An open source framework for storing, processing, andanalyzing large volumes of data.

Magento An open source e-commerce platform.

Mon-goDB

An open source NoSQL document-oriented database.

For more information about each solution, go to: http://rcbops.github.io/tem-plates/.

1. Log in to the horizon Dashboard.

2. Select the Rackspace Private Cloud tab in the Rackspace section of the navigationpane, and click Solutions.

3. Choose a solution and click Installation and More Information.

4. On the solution Parameters window, choose the correct parameters from the drop-down lists.

5. Click Launch Solution.

The Stacks screen of the Orchestration tab displays the solution and its deploymentstatus, as well as all currently active heat templates. The solution status updates regu-larly, showing details of each component in the stack as it is provisioned. When the so-lution is ready, click the solution to see its outputs, including the IP address that it wasassigned.

http://rcbops.github.io/templates/

http://rcbops.github.io/templates/



53

10. Rackspace Private Cloud monitoringRackspace Cloud Monitoring Service allows Rackspace Private Cloud (RPC) customers tomonitor system performance, and safeguard critical data.

10.1. Service and responseWhen a threshold is reached or functionality fails, the Rackspace Cloud Monitoring Ser-vice generates an alert, which creates a ticket in the Rackspace ticketing system. This ticketmoves into the RPC support queue. Tickets flagged as monitoring alerts are given highestpriority, and response is delivered according to the Service Level Agreement (SLA). Refer tothe SLA for detailed information about incident severity levels and corresponding responsetimes.

Specific monitoring alert guidelines can be set for the installation. These details should bearranged by a Rackspace account manager.

10.2. Hardware monitoringHardware monitoring is available only for customers whose clouds are hosted within aRackspace data center. Customers whose clouds are hosted in their own data centers areresponsible for monitoring their own hardware.

For clouds hosted within a Rackspace data center, Rackspace will provision monitoring sup-port for the customer. Rackspace Support assists in handling functionality failure, runningsystem health checks, and managing system capacity. Rackspace Cloud Monitoring Servicewill notify Support when a host is down, or when hardware fails.

10.3. Software monitoringFor software monitoring, polling time is determined by the maas_check_period settingin /etc/rpc_deploy/user_variables.yml, which defaults to 60 seconds.

Rackspace Private Cloud Monitoring Service has two kinds of checks:

• Local: These agent.plugin checks are performed against containers. The checks pollthe API and gather lists of metrics.

These checks will generate a critical alert after three consecutive failures.

Local checks are performed on the following services:

• Compute (nova)







54

• Image Service (glance): The check connects to the glance registry and tests status bycalling an arbitrary URL.

• Dashboard (horizon): The check verifies that the login page is available and uses thecredentials from openrc-maas to log in.

• Galera: The check connects to each member of a Galera cluster and verifies that themembers are fully synchronized and active.

• RabbitMQ: The check connects to each member of a RabbitMQ cluster and gathersstatistics from the API.

• Memcached: The check connects to a Memcached server.

• Global: These remote.http checks poll the load-balanced public endpoints, such as apublic nova API. If a service is marked as administratively down, the check will skip it.

These checks will generate a critical alert after one failure.

Global checks are performed on the following services:

• Compute (nova)






10.4. CDM monitoringThe maas_cdm playbook configures CDM monitoring for the following services and gener-ates alerts at the specified thresholds.

• CPU Idle: < 10%

• Memory used: > 95%

• Disk space used: > 95%

The maas_cdm playbook also configures Object Storage mount point checks. Thesechecks monitor disk space on the mount points and generate alerts if they are unmount-ed or unavailable. For clouds using OpenStack Object Storage, it is important to re-runmaas_cdm.yml play whenever the number of Object Storage nodes changes.

10.5. Running monitoring playbooksThe monitoring playbooks install hardware and software monitoring tools, and consist ofthe following operations:

http://docs.rackspace.com/cm/api/v1.0/cm-devguide/content/appendix-check-types-remote.html#section-ct-remote.http



55

• Install the monitoring CLI package and Rackspace monitoring agent.

• Install checks and alarms for services inside of containers.

• Install checks and alarms for global service monitoring.

• Install checks and alarms for CDM monitoring.

• Install checks and alarms for Object Storage monitoring.

Procedure 10.1. Running the playbooks

1. On the deployment host, change to the /opt/openstack-ansi-ble/rpc_deployment directory.

2. Run the monitoring setup playbook. This installs the CLI package and monitoringagent on all physical hosts and creates a dedicated keystone user for monitoring.

$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/monitoring/raxmon-all.yml

3. Install the monitoring checks and alarms for containerized services.

$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/monitoring/maas_local.yml

4. Install the monitoring checks and alarms for global service monitoring.

$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/monitoring/maas_remote.yml

5. Install the monitoring checks and alarms for CDM monitoring.

$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/monitoring/maas_cdm.yml

6. For clouds using OpenStack Object Storage, install the checks and alarms for swiftmonitoring.

$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/monitoring/swift_maas.yml

Note

Object Storage mount point checks are set up as part of themaas_cdm.yml play. For more information, see Section 10.4, “CDM moni-toring” [54].



56

11. OperationsThe following operations apply to environments after initial installation.

11.1. Configuring an NFS back end to store Glanceimages

The following procedure describes how to configure an NFS back end to store Glance im-ages.

1. Install the nfs-common package on the infra hosts:

$ cd /opt/openstack-ansible/rpc_deployment$ ansible infra_hosts -m command -a 'apt-get install -y nfs-common'

2. Create a mount point directory for a bare-metal mount:

$ ansible infra_hosts -m shell -a 'mkdir /mnt/glance_container_nfs'

3. Add a line with NFS details to the /etc/fstab file:

$ ansible infra_hosts -m shell -a 'echo X.X.X.X:/MNT \/mnt/glance_container_nfs nfs \nfsvers=3,rsize=32768,wsize=32768,actimeo=0,retry=120,hard 0 0 >> \/etc/fstab'

4. Mount the NFS file system:

$ ansible infra_hosts -m shell -a 'mount -a'

5. Verify the NFS mount:

$ ansible infra_hosts -m shell -a 'mount | grep nfs'

6. Add a line to configure LXC for glance containers:

$ ansible infra_hosts -m shell -a 'echo lxc.mount.entry = \/mnt/glance_container_nfs/ var/lib/glance/images \none defaults,bind,rw 0 0 >> /var/lib/lxc/$(lxc-ls | \grep glance_container)/config'

7. Stop and restart the glance containers:

$ ansible infra_hosts -m shell -a 'lxc-stop -n $(lxc-ls | \grep glance_container)'$ ansible infra_hosts -m shell -a 'lxc-start -d -n $(lxc-ls | grep glance_container)'

11.2. Galera cluster maintenanceRoutine maintenance includes gracefully adding or removing nodes from the cluster with-out impacting operation and also starting a cluster after gracefully shutting down all nodes.



57

11.2.1. Removing nodes

In the following example, all but one node was shut down gracefully:

$ ansible galera_container -m shell -a "mysql -h localhost\ -e 'show status like \"%wsrep_cluster_%\";'"node3_galera_container-3ea2cbd3 | FAILED | rc=1 >>ERROR 2002 (HY000): Can't connect to local MySQL serverthrough socket '/var/run/mysqld/mysqld.sock' (2)

node2_galera_container-49a47d25 | FAILED | rc=1 >>ERROR 2002 (HY000): Can't connect to local MySQL serverthrough socket '/var/run/mysqld/mysqld.sock' (2)

node4_galera_container-76275635 | success | rc=0 >>Variable_name Valuewsrep_cluster_conf_id 7wsrep_cluster_size 1wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1wsrep_cluster_status Primary

Compare this example output with the output from the multi-node failure scenario wherethe remaining operational node is non-primary and stops processing SQL requests. Grace-fully shutting down the MariaDB service on all but one node allows the remaining opera-tional node to continue processing SQL requests. When gracefully shutting down multiplenodes, perform the actions sequentially to retain operation.

11.2.2. Starting a cluster

Gracefully shutting down all nodes destroys the cluster. Starting or restarting a cluster fromzero nodes requires creating a new cluster on one of the nodes.

1. The new cluster should be started on the most advanced node. Run the following com-mand to check the seqno value in the grastate.dat file on all of the nodes:

$ ansible galera_container -m shell -a "cat /var/lib/mysql/grastate.dat"node2_galera_container-49a47d25 | success | rc=0 >># GALERA saved state version: 2.1uuid: 338b06b0-2948-11e4-9d06-bef42f6c52f1seqno: 31cert_index:

node3_galera_container-3ea2cbd3 | success | rc=0 >># GALERA saved state version: 2.1uuid: 338b06b0-2948-11e4-9d06-bef42f6c52f1seqno: 31cert_index:

node4_galera_container-76275635 | success | rc=0 >># GALERA saved state version: 2.1uuid: 338b06b0-2948-11e4-9d06-bef42f6c52f1seqno: 31cert_index:



58

In this example, all nodes in the cluster contain the same positive seqno values be-cause they were synchronized just prior to graceful shutdown. If all seqno values areequal, any node can start the new cluster.

$ /etc/init.d/mysql start --wsrep-new-cluster

This command results in a cluster containing a single node. Thewsrep_cluster_size value shows the number of nodes in the cluster.


node3_galera_container-3ea2cbd3 | FAILED | rc=1 >>ERROR 2002 (HY000): Can't connect to local MySQL serverthrough socket '/var/run/mysqld/mysqld.sock' (2)


2. Restart MariaDB on the other nodes and verify that they rejoin the cluster.

node2_galera_container-49a47d25 | success | rc=0 >>Variable_name Valuewsrep_cluster_conf_id 3wsrep_cluster_size 3wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1wsrep_cluster_status Primary

node3_galera_container-3ea2cbd3 | success | rc=0 >>Variable_name Valuewsrep_cluster_conf_id 3wsrep_cluster_size 3wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1wsrep_cluster_status Primary




59

11.3. Galera cluster recovery

11.3.1. Single-node failureIf a single node fails, the other nodes maintain quorum and continue to process SQL re-quests.

1. Run the following Ansible command to determine the failed node:

$ ansible galera_container -m shell -a "mysql -h localhost\ -e 'show status like \"%wsrep_cluster_%\";'"node3_galera_container-3ea2cbd3 | FAILED | rc=1 >>ERROR 2002 (HY000): Can't connect to local MySQL server throughsocket '/var/run/mysqld/mysqld.sock' (111)



In this example, node 3 has failed.

2. Restart MariaDB on the failed node and verify that it rejoins the cluster.

3. If MariaDB fails to start, run the mysqld command and perform further analysis on theoutput. As a last resort, rebuild the container for the node.

11.3.2. Multi-node failureWhen all but one node fails, the remaining node cannot achieve quorum and stops process-ing SQL requests. In this situation, failed nodes that recover cannot join the cluster becauseit no longer exists.

1. Run the following Ansible command to show the failed nodes:

$ ansible galera_container -m shell -a "mysql \-h localhost -e 'show status like \"%wsrep_cluster_%\";'"node2_galera_container-49a47d25 | FAILED | rc=1 >>ERROR 2002 (HY000): Can't connect to local MySQL serverthrough socket '/var/run/mysqld/mysqld.sock' (111)




60

node4_galera_container-76275635 | success | rc=0 >>Variable_name Valuewsrep_cluster_conf_id 18446744073709551615wsrep_cluster_size 1wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1wsrep_cluster_status non-Primary

In this example, nodes 2 and 3 have failed. The remaining operational server indicatesnon-Primary because it cannot achieve quorum.

2. Run the following command to rebootstrap the operational node into the cluster.

$ mysql -e "SET GLOBAL wsrep_provider_options='pc.bootstrap=yes';"node4_galera_container-76275635 | success | rc=0 >>Variable_name Valuewsrep_cluster_conf_id 15wsrep_cluster_size 1wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1wsrep_cluster_status Primary



The remaining operational node becomes the primary node and begins processing SQLrequests.

3. Restart MariaDB on the failed nodes and verify that they rejoin the cluster.

$ ansible galera_container -m shell -a "mysql \-h localhost -e 'show status like \"%wsrep_cluster_%\";'"node3_galera_container-3ea2cbd3 | success | rc=0 >>Variable_name Valuewsrep_cluster_conf_id 17wsrep_cluster_size 3wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1wsrep_cluster_status Primary



http://galeracluster.com/documentation-webpages/quorumreset.html#id1



61

4. If MariaDB fails to start on any of the failed nodes, run the mysqld command andperform further analysis on the output. As a last resort, rebuild the container for thenode.

11.3.3. Complete failure

If all of the nodes in a Galera cluster fail (do not shutdown gracefully), then the integrity ofthe database can no longer be guaranteed and should be restored from backup. Run thefollowing command to determine if all nodes in the cluster have failed:

$ ansible galera_container -m shell -a "cat /var/lib/mysql/grastate.dat"node3_galera_container-3ea2cbd3 | success | rc=0 >># GALERA saved stateversion: 2.1uuid: 338b06b0-2948-11e4-9d06-bef42f6c52f1seqno: -1cert_index:

node2_galera_container-49a47d25 | success | rc=0 >># GALERA saved stateversion: 2.1uuid: 338b06b0-2948-11e4-9d06-bef42f6c52f1seqno: -1cert_index:

node4_galera_container-76275635 | success | rc=0 >># GALERA saved stateversion: 2.1uuid: 338b06b0-2948-11e4-9d06-bef42f6c52f1seqno: -1cert_index:

All the nodes have failed if mysqld is not running on any of the nodes and all of the nodescontain a seqno value of -1.

Note

If any single node has a positive seqno value, then that node can be used torestart the cluster. However, because there is no guarantee that each node hasan identical copy of the data, it is not recommended to restart the cluster usingthe --wsrep-new-cluster command on one node.

11.3.4. Rebuilding a container

Sometimes recovering from a failure requires rebuilding one or more containers.

1. Disable the failed node on the load balancer.

Note

Do not rely on the load balancer health checks to disable the node. If thenode is not disabled, the load balancer will send SQL requests to it before itrejoins the cluster and cause data inconsistencies.



62

2. Use the following commands to destroy the container and remove MariaDB datastored outside of the container. In this example, node 3 failed.

$ lxc-stop -n node3_galera_container-3ea2cbd3$ lxc-destroy -n node3_galera_container-3ea2cbd3$ rm -rf /openstack/node3_galera_container-3ea2cbd3/*

3. Run the host setup playbook to rebuild the container specifically on node 3:

$ ansible-playbook -e @/root/rpc_deploy/user_variables.yml \playbooks/setup/host-setup.yml -l node3 -l node3_galera_container-3ea2cbd3

Note

The playbook will also restart all other containers on the node.

4. Run the infrastructure playbook to configure the container specifically on node 3:

$ ansible-playbook -e @/root/rpc_deploy/user_variables.yml \ playbooks/infrastructure/infrastructure-setup.yml \ -l node3_galera_container-3ea2cbd3

Note

The new container runs a single-node Galera cluster, a dangerous state be-cause the environment contains more than one active database with po-tentially different data.

$ ansible galera_container -m shell -a "mysql \-h localhost -e 'show status like \"%wsrep_cluster_%\";'"node3_galera_container-3ea2cbd3 | success | rc=0 >>Variable_name Valuewsrep_cluster_conf_id 1wsrep_cluster_size 1wsrep_cluster_state_uuid da078d01-29e5-11e4-a051-03d896dbdb2dwsrep_cluster_status Primary





63

5. Restart MariaDB in the new container and verify that it rejoins the cluster.

$ ansible galera_container -m shell -a "mysql \-h localhost -e 'show status like \"%wsrep_cluster_%\";'"node2_galera_container-49a47d25 | success | rc=0 >>Variable_name Valuewsrep_cluster_conf_id 5wsrep_cluster_size 3wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1wsrep_cluster_status Primary

node3_galera_container-3ea2cbd3 | success | rc=0 >>Variable_name Valuewsrep_cluster_conf_id 5wsrep_cluster_size 3wsrep_cluster_state_uuid 338b06b0-2948-11e4-9d06-bef42f6c52f1wsrep_cluster_status Primary


6. Enable the failed node on the load balancer.



64

12. Additional resourcesThese additional resources are designed to help you learn more about the Rackspace Pri-vate Cloud Software and OpenStack.

• If you are an advanced user and are comfortable with APIs, the OpenStack API documen-tation is available in the OpenStack API Documentation library.

• OpenStack API Quick Start• Programming OpenStack Compute API• OpenStack Compute Developer Guide• Rackspace Private Cloud Knowledge Center• OpenStack Manuals• OpenStack - Nova Developer Documentation• OpenStack - Glance Developer Documentation• OpenStack - Keystone Developer Documentation• OpenStack - Horizon Developer Documentation• OpenStack - Cinder Developer Documentation• OpenStack - Swift Developer Documentation

12.1. Document Change HistoryThis version replaces and obsoletes all previous versions. The most recent versions are listedin the following table:

Revision Date Summary of Changes

September 12, 2015 • Rackspace Private Cloud v10.1.14 Software update release

September 3, 2015 • Rackspace Private Cloud v10.1.13 Software update release

August 15, 2015 • Rackspace Private Cloud v10.1.12 Software update release

July 31, 2015 • Rackspace Private Cloud v10.1.11 Software update release

July 7, 2015 • Rackspace Private Cloud v10.1.10 Software update release

June 19, 2015 • Rackspace Private Cloud v10.1.9 Software update release



May 15, 2015 • Rackspace Private Cloud v10.1.6 Software update release

May 8, 2015 • Rackspace Private Cloud v10.1.5 Software update release

April 22, 2015 • Rackspace Private Cloud v10.1.4 Software update release

April 10, 2015 • Rackspace Private Cloud v10.1.3 Software update release

February 26, 2015 • Rackspace Private Cloud v10.1.2 Software Unlimited Availability release

December 19, 2014 • Rackspace Private Cloud v10.1.1 Software Limited Availability release

December 5, 2014 • Rackspace Private Cloud v10.1.0 Software Limited Availability release

November 14, 2014 • Rackspace Private Cloud v10.0.0 Software Limited Availability release

http://api.openstack.org/

http://docs.openstack.org/api/quick-start/content/

http://docs.openstack.org/api/openstack-compute/programmer/content/

http://docs.openstack.org/api/openstack-compute/2/content/

http://www.rackspace.com/knowledge_center/product-page/rackspace-private-cloud

http://docs.openstack.org

http://docs.openstack.org/developer/nova/

http://docs.openstack.org/developer/glance/

http://docs.openstack.org/developer/keystone

http://docs.openstack.org/developer/horizon

http://docs.openstack.org/developer/cinder/

http://docs.openstack.org/developer/swift/

Documents

Rackspace Private Cloud Installation Guideebook.konfigurasi.net/Openstack/Openstack Rackspace Private Cloud... · Rackspace Private Cloud Installa-tion Guide January 20, 2016 RPC