Hypervisor Selection in CloudStack and OpenStack

Tim Mackey – XenServer Community Manager and Evangelist

Hypervisor Selection in CloudStack 4.3 and OpenStack HavanaUnderstanding the choices available

virtg Deep Dive Day 2014

Building a successful cloudWhat are we trying to accomplish?

© Citrix 2014. More information at xenserver.org and follow me on twitter @XenServerArmy

Service Offerings

• Clearly define what you want to offerᵒ What types of applicationsᵒ Who has access, and who owns themᵒ What type of access

• Define how templates need to be managedᵒ Operating system supportᵒ Patching requirements

• Define expectations around compliance and availabilityᵒ Who owns backup and monitoring


Define Tenancy Requirements

• Department data local to departmentᵒ Where is the application data stored

• Data and service isolationᵒ VM migration and host HAᵒ Network services

• Encryption of PII/PCIᵒ Where do keys live when data location unknownᵒ Need encryption designed for the cloud

• Showback to stakeholdersᵒ More than just usage, compliance and audits


Virtualization Infrastructure

• Hypervisor defined by service offeringsᵒ Don’t select hypervisor based on “standards”ᵒ Understand true costs of virtualizationᵒ Multiple hypervisors are “OK”ᵒ Bare metal can be a hypervisor

• To “Pool” resources or notᵒ Is there a real requirement for pooled resourcesᵒ Can the cloud management solution do better?ᵒ Real cost of shared storage

• Primary storage defined by hypervisor

• Template storage defined by solutionᵒ Typically low cost options like NFS

Choice is a good thing….


Apache CloudStack

• Current release: 4.2.1 (4.3 imminent)

• Highly scalable

• Monolithic architecture

• Mostly written in Java

• Multi-hypervisor supportᵒ XenServer, KVM, OracleVM, vSphere,

Linux Containers, Bare metalᵒ 4.3 adds: Hyper-V

• Strong backing from Citrix, CloudOps, Shapeblue and others

Back UpDB

Management Server

MySQLDB

Replication

Management Server

Management Server

Load Balancer

InfrastructureResources


OpenStack

• Current release: Havana

• Scalable in 500 node blocks

• Distributed architecture

• Mostly written in Python

• Multi-hypervisor supportᵒ Group A: KVMᵒ Group B: Hyper-V, vSphere, XenServerᵒ Group C: All others deprecated in Icehouse

• Strong backing from HP, IBM, RAX, RHT, Canonical, Mirantis, Piston Cloud, SUSE, Cloudscaling

Leading hypervisor options


XenServer 6.2

FeatureSource code model Open Source (GPLv2)Maximum VM Density 650 (Linux)CloudStack VM Density 500CloudStack integration Direct XAPI callsOpenStack driver OpenStack nova-compute domUMaximum native cluster size 16Maximum pRAM 1 TBLargest VM 16 vCPU/128GBWindows Operating System All Windows supported by MicrosoftLinux Operating Systems RHEL, CentOS, Debian, Ubuntu, SLES, OELAdvanced features supported (CloudStack) ovs, Storage XenMotion, DMCAdvanced features supported (OpenStack) ovs, Storage XenMotion


vSphere 5.5

FeatureSource code model ProprietaryMaximum VM Density 512CloudStack VM Density 128CloudStack integration vCenterOpenStack driver vCenter – nova-compute node per clusterMaximum native cluster size 32Maximum pRAM 4 TBLargest VM 64 vCPU/1TBWindows Operating Systems DOS, All Windows Server/ClientLinux Operating Systems MostAdvanced features supported (CloudStack) HA, DRS, vDS, Storage vMotionAdvanced features supported (OpenStack) HA, DRS, vMotion


KVM

FeatureSource code model Open Source (GPLv2)Maximum VM Density 10 times the number of pCoresCloudStack VM Density 50CloudStack integration CloudStack Agent (libvirt)OpenStack driver libvirt driverMaximum native cluster size No native cluster supportMaximum pRAM 2 TBLargest VM 160 vCPU/2TBWindows Operating Systems Windows XP and higherLinux Operating Systems VariesAdvanced features supported (CloudStack) NoneAdvanced features supported (OpenStack) None


Microsoft Hyper-V

FeatureSource code model ProprietaryMaximum VM Density 1024CloudStack VM Density 1024CloudStack integration CloudStack Agent (C# calling WMI)OpenStack driver Use Cloudbase driverMaximum native cluster Size 64Maximum pRAM 4 TBLargest VM 64 vCPU/1TBWindows Operating Systems All Windows supported by MicrosoftLinux Operating Systems RHEL, CentOS, Debian, Ubuntu, SLES, OELAdvanced features supported (CloudStack) NoneAdvanced features supported (OpenStack) None

The CloudStack 4.3 world …

Defining the network


Flat Network – Basic Layer 3 Network

Option XenServer vSphere KVM Hyper-V

Security Groups Yes- bridge No Yes Yes

IPv6 No No Yes No

Multiple IPs per NIC Yes Yes Yes Yes

Nicira NVP Yes No Yes No

BigSwitch VNS Yes No Yes No

65.11.1.2

65.11.1.3

65.11.1.4

65.11.1.5

Public Network65.11.0.0/16

Guest VM 1

Guest VM 2

Guest VM 3

Guest VM 4

DHCP, DNS

CloudStackVirtual Router

Security Group 1

Security Group 2


VLANs for Private Cloud

Option XenServer vSphere KVM Hyper-VMax VLANs 800 254 1024 4094

IPv6 No No Yes No

Multiple IPs per NIC

Yes Yes Yes Yes

Nicira NVP Yes No Yes No

BigSwitch VNS Yes No Yes No

MidoKura No No Yes No

VPC Yes Yes Yes Yes

NetScaler Yes Yes Yes Yes

F5 BigIP Yes Yes Yes Yes

Juniper SRX No Yes Yes Yes

Cisco VNMC No Yes No No

10.1.1.1

10.1.1.3

10.1.1.4

10.1.1.5

Public Network/Internet

Guest Virtual Network 10.0.0.0/8VLAN 100

DHCP, DNSNATLoad BalancingVPN

Public IP 65.37.14.1

Gateway10.1.1.1

Guest VM 1

Guest VM 2

Guest VM 3

Guest VM 4



Beyond the VLAN – Network Virtualization


OVS GRE tunnels Yes No No No

Nicira STT tunnel Yes Yes Yes No

MidoNet No No Yes No

VXLAN No Yes Yes No

NVGRE No No No No

Nexus 1000v No Yes No No

Juniper Contrail Yes No No No

Palo Alto Yes Yes Yes No


Virtual Private Cloud and nTier Applications

Feature XenServer vSphere KVM Hyper-V

PVLAN Yes - ovs Yes ovs NoWeb

App

DB

Router

DC1

DC2DC3

DC4

DC5

DC6

VLAN 1

VLAN 2

VLAN 3

S2S VPN

PrivateGW


Delivering specific network services

• KVM ᵒ IPv6ᵒ Security groupsᵒ Large quantity of VLANs

• vSphereᵒ VXLAN required vSphere Enterprise Plusᵒ Cisco Nexus 1000v and ASA 1000v require vSphere Enterprise Plus

• XenServerᵒ Security groupsᵒ Large quantity of VLANsᵒ Juniper Contrail

Instances need a home …Storage, Storage and more Storage


Primary Storage OptionsFeature XenServer vSphere KVM Hyper-V

Local storage Yes Yes Yes YesNFS Yes Yes Yes NoSMB No No No SMB3Single path iSCSI Yes Yes Yes NoMultipath iSCSI PreSetup No No NoDirect array No VAAI No NoShared Mount No No Yes NoTemplate format VHD OVA QCOW2 VHDSolidFire Plugin Yes Yes Yes NoNetApp Plugin Yes Yes Yes NoZone wide No Yes Yes NoCeph RBD No No Yes NoClustered LVM No No Yes No

Cluster

Host

HostPrimary Storage


Secondary Storage Options


NFS Yes Yes Yes No

Swift(1) Yes Yes Yes No

S3 compatible (2) Yes Yes Yes No

SMB No No No Yes

(1) Requires NFS staging area(2) Can be region wide, but must not have NFS secondary storage in zone

ZoneSecondary Storage

Pod

Cluster

Host

HostPrimary Storage

Core virtualization capabilitiesThe limits and features which matter


CloudStack FeaturesFeature XenServer vSphere KVM Hyper-V

Disk IO Statistics Yes No Yes Yes

Memory Overcommit Yes (4x) Yes No No

Dedicated resources Yes Not with HA/DRS Yes Yes

Disk IO throttling No No Yes No

Disk snapshot (running) Yes Yes No No

Disk snapshot (pluggable) Partial Partial No No

Disk snapshot (Stopped) Yes Yes Yes Yes

Memory snapshot Yes Yes Yes No

Zone wide primary storage No Yes Yes No

Resize disk Offline Online Grow Online No

High availability CloudStack Native CloudStack CloudStack

CPU sockets 6.2 and higher Yes Yes Yes

Affinity groups Yes Yes Yes Yes


Multiple Hypervisor Support in CloudStack

• Networkingᵒ Ensure network labels matchᵒ Topology is intersect of chosen hypervisors

• Storageᵒ For system VMs to specific hypervisor typeᵒ Zone with primary storage limited

• Operationsᵒ vSphere Datacenter can not span zonesᵒ Hyper-V may not be mixed with other hypervisorsᵒ HA won’t migrate between hypervisorsᵒ Capacity planning at the cluster/pod level more difficult

The OpenStack Havana world …

Defining the networkThe goodness of Neutron (Quantum)


Flat Network – Basic Layer 3 Network


Security Groups Yes Yes-NVP Yes No

IPv6 No No No No

Multiple IPs per NIC Yes Yes Yes Yes

Nicira NVP Yes Yes Yes No

Firewall rules Yes No Yes No

Routing Yes No Yes No

65.11.1.2

65.11.1.3

65.11.1.4

65.11.1.5

Public Network65.11.0.0/16

Guest VM 1

Guest VM 2

Guest VM 3

Guest VM 4

DHCP, DNS


Security Group 1

Security Group 2


VLANs for Private Cloud

Option XenServer vSphere KVM Hyper-VMax VLANs 800 254 1024 4094

IPv6 No No No No

Multiple IPs Yes Yes Yes Yes 10.1.1.1

10.1.1.3

10.1.1.4

10.1.1.5

Public Network/Internet

Guest Virtual Network 10.0.0.0/8VLAN 100

DHCP, DNSNATLoad BalancingVPN

Public IP 65.37.14.1

Gateway10.1.1.1

Guest VM 1

Guest VM 2

Guest VM 3

Guest VM 4



Beyond the VLAN – Network Virtualization

Option XenServer vSphere KVM Hyper-VOVS GRE tunnels Yes No No NoNicira STT tunnel Yes Yes Yes NoVXLAN No Yes Yes NoNVGRE No No No NoNexus 1000v No Yes Yes NoNicira NVP Yes Yes Yes NoBigSwitch VNS No No Yes NoMidoKura No No Yes NoBrocade No No Yes NoPlumgrid No No Yes NoRyu No No Yes NoNEC No No Yes NoCisco VNMC No No Yes No

Instances need a home …Storage, Storage and more Storage


Instance Storage Options – Nova and Cinder

Feature XenServer vSphere KVM Hyper-VTemplate format VHD VMDK QCOW2 VHDLocal storage Yes Yes Yes YesNFS Yes Yes Yes YesFiber HBA No No Yes NoiSCSI Yes Yes Yes YesiSCSI CHAP Yes No Yes NoCeph No No Yes NoGluster No No Yes NoZFS No No Yes No


OpenStack FeaturesFeature XenServer vSphere KVM Hyper-V

Pause VM Yes No Yes Yes

Inject networking Guest agent Yes Flat (Debian) Yes

Inject file Yes No Yes cloudbase-init

Serial console Yes Yes Yes No

VNC consoles Yes Yes Yes Yes

SPICE console No No Yes No

Snapshot Yes Yes Yes Yes

Set administrator password Yes No No cloudbase-init

Auto configure disk Yes No No No

Evacuate host Yes No Yes No

Volume swap No No Yes No

Volume rate limiting No No Yes No


Multiple Hypervisor Support in OpenStack

• Capabilitiesᵒ Multiple hypervisor support varies by distroᵒ Most deployments are single hypervisorᵒ Difficult to schedule instances to compute nodesᵒ Use host aggregates

• Networkingᵒ Topology is intersect of chosen hypervisors – ML2 helps

• Operationsᵒ HA won’t migrate between hypervisorsᵒ Capacity planning at the cluster/pod level more difficultᵒ Hyper-V does not work with all Neutron plugins

Picking the “best one”When to use which hypervisor…


KVM

• Primary value proposition:ᵒ Low cost with available vendor supportᵒ Familiar administration modelᵒ Broad feature set with active development in both CloudStack and OpenStack

• Cloud use cases:ᵒ Linux centric workloadsᵒ Dev/test clouds ᵒ Web hostingᵒ Tenant density which dictates SDN options

• Weaknesses:ᵒ CloudStack: Requires use of an installed libvirt agentᵒ Limited native storage optionsᵒ No use of advanced native features


Microsoft Hyper-V

• Primary value proposition:ᵒ Unlimited Windows Server VM licensesᵒ Familiar Windows management paradigm

• Cloud use cases:ᵒ Windows and Linux workloadsᵒ Dev/test clouds ᵒ .Net application web hostingᵒ Desktop as a Service clouds

• Weaknesses:ᵒ Minimal use of advanced native featuresᵒ CloudStack: First introduced with CloudStack 4.3 (not yet released)


vSphere

• Primary value proposition:ᵒ Broad application and operating system supportᵒ Readily available pool of vSphere administration talentᵒ Large eco-system of vendor partnersᵒ CloudStack: Many features are native implementationsᵒ Direct feature integration via vCenter

• Cloud use cases:ᵒ Private enterprise cloudsᵒ Dev/test clouds

• Weaknesses:ᵒ vSphere up-front license and ongoing support costsᵒ vCenter integration requires redundant designsᵒ CloudStack: Single data center per zone model


XenServer

• Primary value proposition:ᵒ Low cost with available vendor supportᵒ Broad feature set with active development in both CloudStack and OpenStackᵒ CloudStack: Large install baseᵒ Direct integration via XAPI toolstack

• Cloud use cases:ᵒ Linux centric workloadsᵒ Dev/test clouds ᵒ Web hostingᵒ Desktop as a Service cloudsᵒ Large VM density and secure tenant isolation

• Weaknesses:ᵒ Minimal use of advanced native features


Tying it all Together

1. Define success criteria

2. Select a topology which works

3. Decide on storage options

4. Define supported configurations

5. Select preferred hypervisor(s)

6. Validate matrix

7. Build your Cloud

Work better. Live better.

Technology

Hypervisor Selection in CloudStack and OpenStack