View
8
Download
0
Category
Preview:
Citation preview
What’s in a cloud?An Overview of Virtualization and Openstack
2014-03-24
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 1/551/55
Demistifying “The Cloud”There’s a lot of talk about Cloud Computing - but what do weeven mean when we say “Cloud?”
• Software-as-a-Service?• Platform-as-a-Service?• Infrastructure-as-a-Service?• Magic?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/552/55
Demistifying “The Cloud”There’s a lot of talk about Cloud Computing - but what do weeven mean when we say “Cloud?”• Software-as-a-Service?
• Platform-as-a-Service?• Infrastructure-as-a-Service?• Magic?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/552/55
Demistifying “The Cloud”There’s a lot of talk about Cloud Computing - but what do weeven mean when we say “Cloud?”• Software-as-a-Service?• Platform-as-a-Service?
• Infrastructure-as-a-Service?• Magic?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/552/55
Demistifying “The Cloud”There’s a lot of talk about Cloud Computing - but what do weeven mean when we say “Cloud?”• Software-as-a-Service?• Platform-as-a-Service?• Infrastructure-as-a-Service?
• Magic?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/552/55
Demistifying “The Cloud”There’s a lot of talk about Cloud Computing - but what do weeven mean when we say “Cloud?”• Software-as-a-Service?• Platform-as-a-Service?• Infrastructure-as-a-Service?• Magic?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/552/55
Demistifying “The Cloud”There’s a lot of talk about Cloud Computing - but what do weeven mean when we say “Cloud?”• Software-as-a-Service?• Platform-as-a-Service?• Infrastructure-as-a-Service?• Magic?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/552/55
What exactly is a cloud?
NIST DefinitionCloud computing is a model for enabling convenient, on-demandnetwork access to a shared pool of configurable computingresources (e.g., networks, servers, storage, applications, andservices) that can be rapidly provisioned and released withminimal management effort or service provider interaction.
Cloud means:• On-demand (self service)• Elasticity (easily scale up/down)• Multi-tenancy
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 3/553/55
What exactly is a cloud?
NIST DefinitionCloud computing is a model for enabling convenient, on-demandnetwork access to a shared pool of configurable computingresources (e.g., networks, servers, storage, applications, andservices) that can be rapidly provisioned and released withminimal management effort or service provider interaction.
Cloud means:• On-demand (self service)• Elasticity (easily scale up/down)• Multi-tenancy
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 3/553/55
Infrastructure-as-a-Service (IaaS)
WikipediaIn the most basic cloud-service model, providers of IaaS offercomputers - physical or (more often) virtual machines - and otherresources.
Other resources can be:• Network (virtual networks, Load Balancing, etc…)• Storage (Object and Block)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 4/554/55
Infrastructure-as-a-Service (IaaS)
WikipediaIn the most basic cloud-service model, providers of IaaS offercomputers - physical or (more often) virtual machines - and otherresources.Other resources can be:• Network (virtual networks, Load Balancing, etc…)• Storage (Object and Block)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 4/554/55
Infrastructure-as-a-Service (IaaS)
WikipediaIn the most basic cloud-service model, providers of IaaS offercomputers - physical or (more often) virtual machines - and otherresources.Other resources can be:• Network (virtual networks, Load Balancing, etc…)• Storage (Object and Block)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 4/554/55
IaaS types
IaaS clouds come in two flavors: public and private
With a private IaaS cloud:
+ Can take full advantage of virtualization+ Own the hardware− Own the hardware− Have to manage cloud stack on top of infrastructure
With a public IaaS cloud:
+ Scaling & cost better (can go beyond your HW)+ Don’t own the hardware or the cloud stack− Don’t own the hardware or the cloud stack− Vendor lock-in
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/555/55
IaaS types
IaaS clouds come in two flavors: public and privateWith a private IaaS cloud:
+ Can take full advantage of virtualization
+ Own the hardware− Own the hardware− Have to manage cloud stack on top of infrastructure
With a public IaaS cloud:
+ Scaling & cost better (can go beyond your HW)+ Don’t own the hardware or the cloud stack− Don’t own the hardware or the cloud stack− Vendor lock-in
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/555/55
IaaS types
IaaS clouds come in two flavors: public and privateWith a private IaaS cloud:
+ Can take full advantage of virtualization+ Own the hardware
− Own the hardware− Have to manage cloud stack on top of infrastructure
With a public IaaS cloud:
+ Scaling & cost better (can go beyond your HW)+ Don’t own the hardware or the cloud stack− Don’t own the hardware or the cloud stack− Vendor lock-in
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/555/55
IaaS types
IaaS clouds come in two flavors: public and privateWith a private IaaS cloud:
+ Can take full advantage of virtualization+ Own the hardware− Own the hardware
− Have to manage cloud stack on top of infrastructure
With a public IaaS cloud:
+ Scaling & cost better (can go beyond your HW)+ Don’t own the hardware or the cloud stack− Don’t own the hardware or the cloud stack− Vendor lock-in
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/555/55
IaaS types
IaaS clouds come in two flavors: public and privateWith a private IaaS cloud:
+ Can take full advantage of virtualization+ Own the hardware− Own the hardware− Have to manage cloud stack on top of infrastructure
With a public IaaS cloud:
+ Scaling & cost better (can go beyond your HW)+ Don’t own the hardware or the cloud stack− Don’t own the hardware or the cloud stack− Vendor lock-in
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/555/55
IaaS types
IaaS clouds come in two flavors: public and privateWith a private IaaS cloud:
+ Can take full advantage of virtualization+ Own the hardware− Own the hardware− Have to manage cloud stack on top of infrastructure
With a public IaaS cloud:
+ Scaling & cost better (can go beyond your HW)+ Don’t own the hardware or the cloud stack− Don’t own the hardware or the cloud stack− Vendor lock-in
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/555/55
IaaS types
IaaS clouds come in two flavors: public and privateWith a private IaaS cloud:
+ Can take full advantage of virtualization+ Own the hardware− Own the hardware− Have to manage cloud stack on top of infrastructure
With a public IaaS cloud:+ Scaling & cost better (can go beyond your HW)
+ Don’t own the hardware or the cloud stack− Don’t own the hardware or the cloud stack− Vendor lock-in
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/555/55
IaaS types
IaaS clouds come in two flavors: public and privateWith a private IaaS cloud:
+ Can take full advantage of virtualization+ Own the hardware− Own the hardware− Have to manage cloud stack on top of infrastructure
With a public IaaS cloud:+ Scaling & cost better (can go beyond your HW)+ Don’t own the hardware or the cloud stack
− Don’t own the hardware or the cloud stack− Vendor lock-in
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/555/55
IaaS types
IaaS clouds come in two flavors: public and privateWith a private IaaS cloud:
+ Can take full advantage of virtualization+ Own the hardware− Own the hardware− Have to manage cloud stack on top of infrastructure
With a public IaaS cloud:+ Scaling & cost better (can go beyond your HW)+ Don’t own the hardware or the cloud stack− Don’t own the hardware or the cloud stack
− Vendor lock-in
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/555/55
IaaS types
IaaS clouds come in two flavors: public and privateWith a private IaaS cloud:
+ Can take full advantage of virtualization+ Own the hardware− Own the hardware− Have to manage cloud stack on top of infrastructure
With a public IaaS cloud:+ Scaling & cost better (can go beyond your HW)+ Don’t own the hardware or the cloud stack− Don’t own the hardware or the cloud stack− Vendor lock-in
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/555/55
Virtualization
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 6/556/55
Virtual Machines
“Virtual machines have finally arrived. Dismissed for anumber of years as merely academic curiosities, they arenow seen as cost-effective techniques for organizingcomputer systems resources to provide extraordinary systemflexibility and support for certain unique applications.”
– Robert Goldberg, 1974
• Main idea: transform a single machine into many• Don’t emulate every instruction, emulate only privileged ones• Virtual Machine Manager → VMs
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 7/557/55
Virtual Machines
“Virtual machines have finally arrived. Dismissed for anumber of years as merely academic curiosities, they arenow seen as cost-effective techniques for organizingcomputer systems resources to provide extraordinary systemflexibility and support for certain unique applications.”
– Robert Goldberg, 1974
• Main idea: transform a single machine into many• Don’t emulate every instruction, emulate only privileged ones• Virtual Machine Manager → VMs
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 7/557/55
Virtual Machines
“Virtual machines have finally arrived. Dismissed for anumber of years as merely academic curiosities, they arenow seen as cost-effective techniques for organizingcomputer systems resources to provide extraordinary systemflexibility and support for certain unique applications.”
– Robert Goldberg, 1974
• Main idea: transform a single machine into many• Don’t emulate every instruction, emulate only privileged ones• Virtual Machine Manager → VMs
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 7/557/55
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit:• Efficiency
◦ Most instructions should be executed natively• Equivalence
◦ Behavior in VM = Behavior in bare metal
• Resource control
◦ VMM has complete control
1Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/558/55
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit:• Efficiency
◦ Most instructions should be executed natively
• Equivalence
◦ Behavior in VM = Behavior in bare metal
• Resource control
◦ VMM has complete control
1Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/558/55
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit:• Efficiency
◦ Most instructions should be executed natively• Equivalence
◦ Behavior in VM = Behavior in bare metal• Resource control
◦ VMM has complete control
1Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/558/55
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit:• Efficiency
◦ Most instructions should be executed natively• Equivalence
◦ Behavior in VM = Behavior in bare metal
• Resource control
◦ VMM has complete control
1Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/558/55
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit:• Efficiency
◦ Most instructions should be executed natively• Equivalence
◦ Behavior in VM = Behavior in bare metal• Resource control
◦ VMM has complete control
1Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/558/55
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit:• Efficiency
◦ Most instructions should be executed natively• Equivalence
◦ Behavior in VM = Behavior in bare metal• Resource control
◦ VMM has complete control
1Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/558/55
Types of hypervisors
Examples:Xen, VMWare ESX
Examples:Virtualbox, QEMU
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 9/559/55
Types of hypervisors
Examples:Xen, VMWare ESX
Examples:Virtualbox, QEMU
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 9/559/55
x86 and Privilege RingsRing 3
Ring 2
Ring 1
Ring 0
Kernel
Device drivers
Applications
Device drivers
Least privileged
Most privileged
• Only ring 0 can execute privileged instructions• Linux/Windows: ring 0 (supervisor/kernel) and 3 (user)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 10/5510/55
Virtualization on x86x86: reputation for being unfriendly to virtualization• Not all privileged operations generate traps (e.g. popf)
• No MMU virtualization• 1998 VMware: binary translation
QuestionWhat are the advantages/disadvantages of this method?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5511/55
Virtualization on x86x86: reputation for being unfriendly to virtualization• Not all privileged operations generate traps (e.g. popf)• No MMU virtualization
• 1998 VMware: binary translation
QuestionWhat are the advantages/disadvantages of this method?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5511/55
Virtualization on x86x86: reputation for being unfriendly to virtualization• Not all privileged operations generate traps (e.g. popf)• No MMU virtualization• 1998 VMware: binary translation
QuestionWhat are the advantages/disadvantages of this method?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5511/55
Virtualization on x86x86: reputation for being unfriendly to virtualization• Not all privileged operations generate traps (e.g. popf)• No MMU virtualization• 1998 VMware: binary translation
QuestionWhat are the advantages/disadvantages of this method?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5511/55
Virtualization on x86x86: reputation for being unfriendly to virtualization• Not all privileged operations generate traps (e.g. popf)• No MMU virtualization• 1998 VMware: binary translation
QuestionWhat are the advantages/disadvantages of this method?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5511/55
Virtualization on x86x86: reputation for being unfriendly to virtualization• Not all privileged operations generate traps (e.g. popf)• No MMU virtualization• 1998 VMware: binary translation
QuestionWhat are the advantages/disadvantages of this method?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5511/55
Para-virtualization
QuestionWhat are the advantages/disadvantages of this method?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 12/5512/55
Para-virtualization
QuestionWhat are the advantages/disadvantages of this method?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 12/5512/55
Hardware Assisted Virtualization (HAV)
In 2006, Intel VT-x and AMD AMD-V ⇒ VMM much easier toimplement• Allowed for an unmodified guest OS• Introduced guest-mode execution• When a guest performs a privileged instruction, trap to VMM
via a VM Exit
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 13/5513/55
VM Entry/Exit Example
Guest mode
Host mode
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM
Allocate new memory
VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM
Allocate new memory VM Exit: PAGE_FAULT
VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM
Allocate new memory VM Exit: PAGE_FAULT
VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM
Allocate new memory VM Exit: PAGE_FAULT
VM Entry VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM
Allocate new memory VM Exit: PAGE_FAULT
VM Entry VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM
Allocate new memory VM Exit: PAGE_FAULT
VM Entry
Context Switch
VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM
Allocate new memory VM Exit: PAGE_FAULT
VM Entry
Context Switch VM Exit: CR_ACCESS
VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM
Allocate new memory VM Exit: PAGE_FAULT
VM Entry
Context Switch VM Exit: CR_ACCESS
VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Guest mode
Host mode
Launch VM
Allocate new memory VM Exit: PAGE_FAULT
VM Entry
Context Switch VM Exit: CR_ACCESS
…
…
…....
VM Entry
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
VM Entry/Exit Example
Turn off machine
Guest mode
Host mode
Launch VM
Allocate new memory VM Exit: PAGE_FAULT
VM Entry
Context Switch VM Exit: CR_ACCESS
…
…
…....
VM Exit: VM_OFF
VM Entry Deallocate VM
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5514/55
Intel VT-x
A new datastructure: VMCS• On VM Exit, contains the CPU & control state of a guest• Relevant guest state is passed to the VMM (I/O port, etc...)In 2008, VT-d features: more than CPU (EPT/IOMMU)
QuestionWhat are the advantages/disadvantages of HAV?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 15/5515/55
HyperTap2
• Can use HAV for robust security+reliability monitoring
VM_Exit_handler() { ... }
HyperTap tools
VM 1 monitors
VM n monitors
Gue
st V
M
Virtual Machines
Virtual Machines
VMIAI tools
VM_exit_handler(vcpu, vmcs) { Event_forwarder(vcpu, vmcs) // existing code Event_processing(…) VM_resume()
}
VM 1 (vcpu1, vcpu2)
Event Multiplexer
Virtual Machines
…
Guest execution
Host execution
VM exit events
Hypervisor
Exported helper functions: set_gpage_protect(…); read_gpage(…); …
Monitor 1.1 (e.g. HRKD)
Monitor 1.2 (e.g. GOSHD)
Hypervisor
Monitor h.1 (e.g. HHD)
VM n (vcpu3, vcpu4)
Monitor n.1 (e.g. HRKD)
… …
Virtual Machines
Virtual Machines
VMIAI tools
VM_exit_handler(vcpu, vmcs) { Event_forwarder(vcpu, vmcs) // existing code Event_processing(…) VM_resume()
}
VM 1 (vcpu1, vcpu2)
Event Buffers
Virtual Machines
…
Guest execution
Host execution
VM exit events
Hypervisor
Exported helper functions: set_gpage_protect(…); read_gpage(…); …
Monitor 1.1 (e.g. HRKD)
Monitor 1.2 (e.g. GOSHD)
Hypervisor
Monitor h.1 (e.g. HHD)
VM 2 (vcpu3, vcpu4)
Monitor 2.1 (e.g. HRKD)
… …
vcpu1 vcpu2 vcpu3 vcpu4
Architecture View
In-host View (ps/top, Tasks Manager)
Existing VMI View
VMIAI View
Kernel data structure
Hardware
Process 1
Process 2 (DKOM rootkit)
Process 3
1 3 2 Process 1
Process 3
Process 1
Process 3
Process 1
Process 3
Hyp
ervisor
Har
dw
are MMU
CR3
HVA
Privilege
VM Events
TSS
Process 2
Event Forwarder Event Mul8plexer
Virtual Machine 1
(vcpu 1, vcpu 2)
…
Guest execution
Host execution
VM exit events
Hypervisor
Helper APIs set_gpage_protect(…); read_gpage(…); …
Monitor 1.1 (e.g. HRKD)
Monitor 1.2
(e.g. PED)
Monitor n.1 (e.g. HRKD)
… Virtual
Machine n (vcpu 3, vcpu 4)
VM Monitors
vcpu1, vcpu2 events …
Non-blocking
Blocking
API call
HyperTap Core
vcpu3, vcpu4 events
KVM Hypervisor
Virtual Machine 1
Event Forwarder
Linux kernel (host OS)
Event Mul8plexer
HRKD
HRKD
PED
Monitors run in host machine
Virtual Machines
Kernel module Helper APIs (ioctl interfaces)
Virtual Machine n
XEN hypervisor
Virtual Machine 1
Event Forwarder
Event Mul8plexer
HRKD
HRKD
PED
DomU Virtual Machines
Helper APIs (hypercalls)
Dom
0
Virtual Machine n
Monitors
2C. Pham, Z. Estrada, P. Cao, Z. Kalbarczyk, R. Iyer “Reliability and Security Monitoring of Virtual MachinesUsing Hardware Architectural Invariants” 2014
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 16/5516/55
Open-Source Virtual MachineMonitors (VMMs)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 17/5517/55
Xen
• Xen para-virt upstream in Linux since 3.0• also supports HVM (HAV)• Amazon EC2
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 18/5518/55
KVM
• vCPUs scheduled as processes in Linux• Paravirtualized drivers: virtio• Default for Openstack
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 19/5519/55
Thoughts on Virtualization
• Virtualization has overhead• Many VMs run only one application• Many people running Linux on Linux
(e.g. KVM w/Linux guest)
QuestionCan we do better?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 20/5520/55
Linux Containers (LXC)
• All “guests” share the same kernel - no “trap and emulate”• Isolation via cgroups/namespaces
QuestionWhat are the advantages/disadvantages of containers?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 21/5521/55
Virtualization Summary
Paravirtualization• Xen
HAV• KVM
Overhead/double work• Linux Containers
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 22/5522/55
Openstack
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 23/5523/55
How to manage VMs?• Xen provides mgmt, KVM doesn’t• need for common interface
KVM
virsh virt-manager virt-factory
VIRTUALIZATION API
LXC OpenVZ UML ESXXen other...
oVirt
• Standardized method for managing multiple hypervisors
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 24/5524/55
How to manage VMs?• Xen provides mgmt, KVM doesn’t• need for common interface
KVM
virsh virt-manager virt-factory
VIRTUALIZATION API
LXC OpenVZ UML ESXXen other...
oVirt
• Standardized method for managing multiple hypervisorsZak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 24/55
24/55
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux)
• Can setup network/storage• Utilizes local user/group services
QuestionWhat are the limitations of something like libvirt?
• Network across entire environment?• No rapid VM creation• Weak multi-tenancy• Not a cloud (no utility abstraction)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5525/55
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux)• Can setup network/storage
• Utilizes local user/group services
QuestionWhat are the limitations of something like libvirt?
• Network across entire environment?• No rapid VM creation• Weak multi-tenancy• Not a cloud (no utility abstraction)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5525/55
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux)• Can setup network/storage• Utilizes local user/group services
QuestionWhat are the limitations of something like libvirt?
• Network across entire environment?• No rapid VM creation• Weak multi-tenancy• Not a cloud (no utility abstraction)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5525/55
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux)• Can setup network/storage• Utilizes local user/group services
QuestionWhat are the limitations of something like libvirt?
• Network across entire environment?• No rapid VM creation• Weak multi-tenancy• Not a cloud (no utility abstraction)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5525/55
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux)• Can setup network/storage• Utilizes local user/group services
QuestionWhat are the limitations of something like libvirt?
• Network across entire environment?• No rapid VM creation• Weak multi-tenancy• Not a cloud (no utility abstraction)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5525/55
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux)• Can setup network/storage• Utilizes local user/group services
QuestionWhat are the limitations of something like libvirt?
• Network across entire environment?• No rapid VM creation• Weak multi-tenancy• Not a cloud (no utility abstraction)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5525/55
Openstack Overview• Originated in 2010, by Rackspace and NASA• Rapid development - new (stable?) release every 6 months• Modular architecture made up of multiple projects that provide
a separate piece of an IaaS service, each with their ownREST API
• Command-line and GUI interfaces• Seeks to be to IaaS what Linux is to OSes
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 26/5526/55
Interest in Openstack
Jun 2010
Dec 2010
Jun 2011
Dec 2011
Jun 2012
Dec 2012
Jun 2013
Dec 2013
Date
0
20
40
60
80
100
Inte
rest
Google Trends for Openstack
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 27/5527/55
Openstack Architecture
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 28/5528/55
Openstack Architecture
Okay, that wasn’t so bad, why do you complainabout the complexity of dealing with this?
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 29/5529/55
Openstack Architecture - expanded
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 30/5530/55
Openstack Architecture
QuestionThat looks more complex than libvirt, why do I want to do this?
• Life is easier AFTER setting it up (mostly)• Lots of automation available - not always if you want to do
something special
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 31/5531/55
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications
• Contains a list of all the components in the system• Multiple backends (LDAP, MySQL, etc..)• Also acts as the index for API endpoints
◦ e.g. You can connect to the compute service viahttp://cloud1:8774
Example command to list known services:root@logging:~# keystone service-list+----------------------------------+----------+----------+----------------------------+| id | name | type | description |+----------------------------------+----------+----------+----------------------------+| 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service || 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service || 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service || b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service || f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service |+----------------------------------+----------+----------+----------------------------+
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5532/55
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications• Contains a list of all the components in the system
• Multiple backends (LDAP, MySQL, etc..)• Also acts as the index for API endpoints
◦ e.g. You can connect to the compute service viahttp://cloud1:8774
Example command to list known services:root@logging:~# keystone service-list+----------------------------------+----------+----------+----------------------------+| id | name | type | description |+----------------------------------+----------+----------+----------------------------+| 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service || 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service || 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service || b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service || f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service |+----------------------------------+----------+----------+----------------------------+
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5532/55
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications• Contains a list of all the components in the system• Multiple backends (LDAP, MySQL, etc..)
• Also acts as the index for API endpoints
◦ e.g. You can connect to the compute service viahttp://cloud1:8774
Example command to list known services:root@logging:~# keystone service-list+----------------------------------+----------+----------+----------------------------+| id | name | type | description |+----------------------------------+----------+----------+----------------------------+| 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service || 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service || 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service || b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service || f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service |+----------------------------------+----------+----------+----------------------------+
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5532/55
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications• Contains a list of all the components in the system• Multiple backends (LDAP, MySQL, etc..)• Also acts as the index for API endpoints
◦ e.g. You can connect to the compute service viahttp://cloud1:8774
Example command to list known services:root@logging:~# keystone service-list+----------------------------------+----------+----------+----------------------------+| id | name | type | description |+----------------------------------+----------+----------+----------------------------+| 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service || 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service || 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service || b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service || f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service |+----------------------------------+----------+----------+----------------------------+
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5532/55
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications• Contains a list of all the components in the system• Multiple backends (LDAP, MySQL, etc..)• Also acts as the index for API endpoints
◦ e.g. You can connect to the compute service viahttp://cloud1:8774
Example command to list known services:root@logging:~# keystone service-list+----------------------------------+----------+----------+----------------------------+| id | name | type | description |+----------------------------------+----------+----------+----------------------------+| 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service || 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service || 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service || b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service || f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service |+----------------------------------+----------+----------+----------------------------+
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5532/55
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications• Contains a list of all the components in the system• Multiple backends (LDAP, MySQL, etc..)• Also acts as the index for API endpoints
◦ e.g. You can connect to the compute service viahttp://cloud1:8774
Example command to list known services:root@logging:~# keystone service-list+----------------------------------+----------+----------+----------------------------+| id | name | type | description |+----------------------------------+----------+----------+----------------------------+| 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service || 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service || 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service || b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service || f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service |+----------------------------------+----------+----------+----------------------------+
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5532/55
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications• Contains a list of all the components in the system• Multiple backends (LDAP, MySQL, etc..)• Also acts as the index for API endpoints
◦ e.g. You can connect to the compute service viahttp://cloud1:8774
Example command to list known services:root@logging:~# keystone service-list+----------------------------------+----------+----------+----------------------------+| id | name | type | description |+----------------------------------+----------+----------+----------------------------+| 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service || 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service || 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service || b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service || f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service |+----------------------------------+----------+----------+----------------------------+
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5532/55
Piece-by-piece: Compute (nova)
• Originally contributed by NASA
• Communicates with the hypervisor to create/destroy virtualmachines
• Similar to job scheduler in HPC
◦ Support for libvirt, ESX, Hyper-V, etc...
• Each compute node connects to the others via a message bus(AMQP)
COMPUTEZak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 33/55
33/55
Piece-by-piece: Compute (nova)
• Originally contributed by NASA• Communicates with the hypervisor to create/destroy virtual
machines
• Similar to job scheduler in HPC
◦ Support for libvirt, ESX, Hyper-V, etc...
• Each compute node connects to the others via a message bus(AMQP)
COMPUTEZak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 33/55
33/55
Piece-by-piece: Compute (nova)
• Originally contributed by NASA• Communicates with the hypervisor to create/destroy virtual
machines• Similar to job scheduler in HPC
◦ Support for libvirt, ESX, Hyper-V, etc...• Each compute node connects to the others via a message bus
(AMQP)
COMPUTEZak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 33/55
33/55
Piece-by-piece: Compute (nova)
• Originally contributed by NASA• Communicates with the hypervisor to create/destroy virtual
machines• Similar to job scheduler in HPC
◦ Support for libvirt, ESX, Hyper-V, etc...
• Each compute node connects to the others via a message bus(AMQP)
COMPUTEZak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 33/55
33/55
Piece-by-piece: Compute (nova)
• Originally contributed by NASA• Communicates with the hypervisor to create/destroy virtual
machines• Similar to job scheduler in HPC
◦ Support for libvirt, ESX, Hyper-V, etc...• Each compute node connects to the others via a message bus
(AMQP)
COMPUTEZak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 33/55
33/55
Piece-by-piece: Image (glance)
• Provides VM images to the compute service for booting
• Copies image from storage to the compute node, with acopy-on-write file for each instance
• Can use local storage or an object store
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 34/5534/55
Piece-by-piece: Image (glance)
• Provides VM images to the compute service for booting• Copies image from storage to the compute node, with a
copy-on-write file for each instance
• Can use local storage or an object store
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 34/5534/55
Piece-by-piece: Image (glance)
• Provides VM images to the compute service for booting• Copies image from storage to the compute node, with a
copy-on-write file for each instance• Can use local storage or an object store
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 34/5534/55
glance and nova with libvirt
openstack-glance openstack-novaVM
VM
libvirtbase
libvirtinstancedisks
• Compute server pulls the image from glance at instance start• Instance disks are copy-on-write images (e.g. QCOW) off of
this base• QCOW images are deleted upon instance termination
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 35/5535/55
Piece-by-piece: Block Storage (cinder)
• Block storage: persistent disk for instances
• Can be used as boot disks or additional storage - dynamically• Built on top of iSCSI, etc• Integration for enterprise block storage (NetApp, IBM, etc...)• NFS/Ceph/gluster as well
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 36/5536/55
Piece-by-piece: Block Storage (cinder)
• Block storage: persistent disk for instances• Can be used as boot disks or additional storage - dynamically
• Built on top of iSCSI, etc• Integration for enterprise block storage (NetApp, IBM, etc...)• NFS/Ceph/gluster as well
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 36/5536/55
Piece-by-piece: Block Storage (cinder)
• Block storage: persistent disk for instances• Can be used as boot disks or additional storage - dynamically• Built on top of iSCSI, etc
• Integration for enterprise block storage (NetApp, IBM, etc...)• NFS/Ceph/gluster as well
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 36/5536/55
Piece-by-piece: Block Storage (cinder)
• Block storage: persistent disk for instances• Can be used as boot disks or additional storage - dynamically• Built on top of iSCSI, etc• Integration for enterprise block storage (NetApp, IBM, etc...)
• NFS/Ceph/gluster as well
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 36/5536/55
Piece-by-piece: Block Storage (cinder)
• Block storage: persistent disk for instances• Can be used as boot disks or additional storage - dynamically• Built on top of iSCSI, etc• Integration for enterprise block storage (NetApp, IBM, etc...)• NFS/Ceph/gluster as well
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 36/5536/55
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace
• Object storage: distributed object/file store• Similar to Amazon S3• Eventually consistent• Online capacity adjustment• Versioned writes• Quota management• rsync based replication
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5537/55
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace• Object storage: distributed object/file store
• Similar to Amazon S3• Eventually consistent• Online capacity adjustment• Versioned writes• Quota management• rsync based replication
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5537/55
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace• Object storage: distributed object/file store• Similar to Amazon S3
• Eventually consistent• Online capacity adjustment• Versioned writes• Quota management• rsync based replication
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5537/55
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace• Object storage: distributed object/file store• Similar to Amazon S3• Eventually consistent
• Online capacity adjustment• Versioned writes• Quota management• rsync based replication
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5537/55
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace• Object storage: distributed object/file store• Similar to Amazon S3• Eventually consistent• Online capacity adjustment
• Versioned writes• Quota management• rsync based replication
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5537/55
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace• Object storage: distributed object/file store• Similar to Amazon S3• Eventually consistent• Online capacity adjustment• Versioned writes
• Quota management• rsync based replication
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5537/55
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace• Object storage: distributed object/file store• Similar to Amazon S3• Eventually consistent• Online capacity adjustment• Versioned writes• Quota management
• rsync based replication
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5537/55
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace• Object storage: distributed object/file store• Similar to Amazon S3• Eventually consistent• Online capacity adjustment• Versioned writes• Quota management• rsync based replication
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5537/55
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute
• Now its own project (formerly quantum)• Provides abstraction layer for creating virtual networks
◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc...
• Also provides services like load balancing• Part of the official release since 2012.2
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5538/55
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute• Now its own project (formerly quantum)
• Provides abstraction layer for creating virtual networks
◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc...
• Also provides services like load balancing• Part of the official release since 2012.2
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5538/55
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute• Now its own project (formerly quantum)• Provides abstraction layer for creating virtual networks
◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc...• Also provides services like load balancing• Part of the official release since 2012.2
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5538/55
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute• Now its own project (formerly quantum)• Provides abstraction layer for creating virtual networks
◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc...
• Also provides services like load balancing• Part of the official release since 2012.2
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5538/55
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute• Now its own project (formerly quantum)• Provides abstraction layer for creating virtual networks
◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc...• Also provides services like load balancing
• Part of the official release since 2012.2
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5538/55
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute• Now its own project (formerly quantum)• Provides abstraction layer for creating virtual networks
◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc...• Also provides services like load balancing• Part of the official release since 2012.2
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5538/55
Openstack Architecture (Reprise)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 39/5539/55
Tying it all together (simplified)
Hardware
OS KVM
libvirt
computenetwork image
API
...
identity
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 40/5540/55
But wait, there’s more!
I only focused on the “core components” for a working cloud, butthere are plenty more• Ceilometer - Metering/Monitoring• Heat - Automation (Similar to AWS CloudFormation)• Savannah - Tighter Hadoop Integration
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 41/5541/55
An actual production use case
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 42/5542/55
ANL Magellan experiences
• CPU Performance was good• I/O bandwidth was decent• I/O latency was terrible
• Not worth runnning tightly-coupled HPC workloads• Need OS bypasss for infiniband and GPUs• “Developer productivity went through the roof”
◦ Easy, low-cost experimentation• IaaS good for data-centric workloads (bioinformatics)
◦ Many folks that run hadoop do so on top of AWS
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 43/5543/55
ANL Magellan experiences
• CPU Performance was good• I/O bandwidth was decent• I/O latency was terrible• Not worth runnning tightly-coupled HPC workloads• Need OS bypasss for infiniband and GPUs
• “Developer productivity went through the roof”◦ Easy, low-cost experimentation
• IaaS good for data-centric workloads (bioinformatics)◦ Many folks that run hadoop do so on top of AWS
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 43/5543/55
ANL Magellan experiences
• CPU Performance was good• I/O bandwidth was decent• I/O latency was terrible• Not worth runnning tightly-coupled HPC workloads• Need OS bypasss for infiniband and GPUs• “Developer productivity went through the roof”
◦ Easy, low-cost experimentation• IaaS good for data-centric workloads (bioinformatics)
◦ Many folks that run hadoop do so on top of AWS
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 43/5543/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)
◦ Always run into some problem◦ Never the same problem twice
• Automated tools are your friend (packstack, puppet)• Really convenient for kernel hacking• Developers like it a lot
◦ Can give them root
• Rapid Release
− Things are always changing+ Noticeable improvement with each release
• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)◦ Always run into some problem
◦ Never the same problem twice• Automated tools are your friend (packstack, puppet)• Really convenient for kernel hacking• Developers like it a lot
◦ Can give them root
• Rapid Release
− Things are always changing+ Noticeable improvement with each release
• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)◦ Always run into some problem◦ Never the same problem twice
• Automated tools are your friend (packstack, puppet)• Really convenient for kernel hacking• Developers like it a lot
◦ Can give them root
• Rapid Release
− Things are always changing+ Noticeable improvement with each release
• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)◦ Always run into some problem◦ Never the same problem twice
• Automated tools are your friend (packstack, puppet)
• Really convenient for kernel hacking• Developers like it a lot
◦ Can give them root
• Rapid Release
− Things are always changing+ Noticeable improvement with each release
• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)◦ Always run into some problem◦ Never the same problem twice
• Automated tools are your friend (packstack, puppet)• Really convenient for kernel hacking
• Developers like it a lot
◦ Can give them root
• Rapid Release
− Things are always changing+ Noticeable improvement with each release
• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)◦ Always run into some problem◦ Never the same problem twice
• Automated tools are your friend (packstack, puppet)• Really convenient for kernel hacking• Developers like it a lot
◦ Can give them root• Rapid Release
− Things are always changing+ Noticeable improvement with each release
• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)◦ Always run into some problem◦ Never the same problem twice
• Automated tools are your friend (packstack, puppet)• Really convenient for kernel hacking• Developers like it a lot
◦ Can give them root
• Rapid Release
− Things are always changing+ Noticeable improvement with each release
• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)◦ Always run into some problem◦ Never the same problem twice
• Automated tools are your friend (packstack, puppet)• Really convenient for kernel hacking• Developers like it a lot
◦ Can give them root• Rapid Release
− Things are always changing+ Noticeable improvement with each release
• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)◦ Always run into some problem◦ Never the same problem twice
• Automated tools are your friend (packstack, puppet)• Really convenient for kernel hacking• Developers like it a lot
◦ Can give them root• Rapid Release
− Things are always changing
+ Noticeable improvement with each release• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)◦ Always run into some problem◦ Never the same problem twice
• Automated tools are your friend (packstack, puppet)• Really convenient for kernel hacking• Developers like it a lot
◦ Can give them root• Rapid Release
− Things are always changing+ Noticeable improvement with each release
• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)◦ Always run into some problem◦ Never the same problem twice
• Automated tools are your friend (packstack, puppet)• Really convenient for kernel hacking• Developers like it a lot
◦ Can give them root• Rapid Release
− Things are always changing+ Noticeable improvement with each release
• VMs are just as reliable as with libvirt, mgmt not so much
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5544/55
Hypervisor PerformanceComparison
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 45/5545/55
Experimental SetupSystems:• Dell R720 (2013) (Machine 1)• Homebuilt (2011) (Machine 2)• Ubuntu 12.04 LTS
Bioinformatics Applications:• Paired-end short read
alignment• Burrows-Wheeler Aligner• Novoalign
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 46/5546/55
Memory Performance
0
1000
2000
3000
4000
5000
6000
7000
8000M
iB/s
Copy Scale Add Triad
STREAM ResultsPHYKVMXEN-pvLXC
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 47/5547/55
Disk Performance
0
100
200
300
400
500
600M
iB/s
0
200
400
600
800
1000
1200
1400
See
ks/s
Seq. Write Seq. Read Seek
Bonnie++ ResultsPHYKVMXEN-pvLXC
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 48/5548/55
PHY KVM XEN-pv LXC0.95
1.00
1.05
1.10
1.15
1.20
1.25M
ean
Run
time
(nor
mal
ized
toph
ysic
al)
1.00
1.19
1.09
1.00
BWA Runtime
PHY KVM XEN-pv LXC
0.96
0.98
1.00
1.02
1.04
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00 1.00
1.03
0.99
Novoalign Runtime
PHY KVM XEN-pv LXC0.95
1.00
1.05
1.10
1.15
1.20
1.25
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00
1.11
1.00 1.00
BWA Runtime (Machine 2)
PHY KVM XEN-pv LXC
0.96
0.98
1.00
1.02
1.04
1.06
1.08
1.10
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00
1.06
0.99
0.00
Novoalign Runtime (Machine 2)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 49/5549/55
PHY KVM XEN-pv LXC0.95
1.00
1.05
1.10
1.15
1.20
1.25M
ean
Run
time
(nor
mal
ized
toph
ysic
al)
1.00
1.19
1.09
1.00
BWA Runtime
PHY KVM XEN-pv LXC
0.96
0.98
1.00
1.02
1.04
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00 1.00
1.03
0.99
Novoalign Runtime
PHY KVM XEN-pv LXC0.95
1.00
1.05
1.10
1.15
1.20
1.25
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00
1.11
1.00 1.00
BWA Runtime (Machine 2)
PHY KVM XEN-pv LXC
0.96
0.98
1.00
1.02
1.04
1.06
1.08
1.10
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00
1.06
0.99
0.00
Novoalign Runtime (Machine 2)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 49/5549/55
PHY KVM XEN-pv LXC0.95
1.00
1.05
1.10
1.15
1.20
1.25M
ean
Run
time
(nor
mal
ized
toph
ysic
al)
1.00
1.19
1.09
1.00
BWA Runtime
PHY KVM XEN-pv LXC
0.96
0.98
1.00
1.02
1.04
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00 1.00
1.03
0.99
Novoalign Runtime
PHY KVM XEN-pv LXC0.95
1.00
1.05
1.10
1.15
1.20
1.25
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00
1.11
1.00 1.00
BWA Runtime (Machine 2)
PHY KVM XEN-pv LXC
0.96
0.98
1.00
1.02
1.04
1.06
1.08
1.10
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00
1.06
0.99
0.00
Novoalign Runtime (Machine 2)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 49/5549/55
PHY KVM XEN-pv LXC0.95
1.00
1.05
1.10
1.15
1.20
1.25M
ean
Run
time
(nor
mal
ized
toph
ysic
al)
1.00
1.19
1.09
1.00
BWA Runtime
PHY KVM XEN-pv LXC
0.96
0.98
1.00
1.02
1.04
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00 1.00
1.03
0.99
Novoalign Runtime
PHY KVM XEN-pv LXC0.95
1.00
1.05
1.10
1.15
1.20
1.25
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00
1.11
1.00 1.00
BWA Runtime (Machine 2)
PHY KVM XEN-pv LXC
0.96
0.98
1.00
1.02
1.04
1.06
1.08
1.10
Mea
nR
untim
e(n
orm
aliz
edto
phys
ical
)
1.00
1.06
0.99
0.00
Novoalign Runtime (Machine 2)
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 49/5549/55
Multiple VMs
PHY
PHY-pin
KVM
KVM-pin
XEN-pv
XEN-pv-pin
LXC
LXC-pin0.95
1.00
1.05
1.10
1.15
1.20
1.25
1.30
1.35
Mea
nR
untim
e(n
orm
aliz
edto
sing
le)
1.17
1.12
1.28 1.29
1.19 1.181.15
1.22
BWA Runtime (8 Concurrent Executions)
PHY
PHY-pin
KVM
KVM-pin
XEN-pv
XEN-pv-pin
LXC
LXC-pin0.95
1.00
1.05
1.10
1.15
Mea
nR
untim
e(n
orm
aliz
edto
sing
le)
1.16
1.061.07 1.08 1.08 1.08
1.12
1.07
Novoalign Runtime (8 Concurrent Executions)
• Only on the Dell R720• Still not utilizing this machine fully
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 50/5550/55
Application Run Time Statistics
0 20 40 60 80 1000.0
0.5
1.0
1.5
Act
ive
CP
Us BWA System Statistics
UserSys
0 20 40 60 80 1000
1
2
3
%M
emor
y
UsedMappedCache
0 20 40 60 80 100Run completion (%)
0
200
400
600
MiB
/s ReadWrite
0 20 40 60 80 1000.0
0.5
1.0
1.5
Act
ive
CP
Us Novoalign System Statistics
UserSys
0 20 40 60 80 1000
1
2
3
%M
emor
y
UsedMappedCache
0 20 40 60 80 100Run completion (%)
0100200300400
MiB
/s ReadWrite
• Gathered using collectl• Only for a single run
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 51/5551/55
Some Conclusions
• CPU pinning can be effective (but be careful)
• Performance for a given VMM is dependent on the applicationbeing executed
• CPU-bound applications run at near-baremetal speeds invirtualized environments
• VMM scheduling overheard is low when VMs are notoverprovisioned
• Linux Containers have low performance overhead
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 52/5552/55
Some Conclusions
• CPU pinning can be effective (but be careful)• Performance for a given VMM is dependent on the application
being executed
• CPU-bound applications run at near-baremetal speeds invirtualized environments
• VMM scheduling overheard is low when VMs are notoverprovisioned
• Linux Containers have low performance overhead
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 52/5552/55
Some Conclusions
• CPU pinning can be effective (but be careful)• Performance for a given VMM is dependent on the application
being executed• CPU-bound applications run at near-baremetal speeds in
virtualized environments
• VMM scheduling overheard is low when VMs are notoverprovisioned
• Linux Containers have low performance overhead
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 52/5552/55
Some Conclusions
• CPU pinning can be effective (but be careful)• Performance for a given VMM is dependent on the application
being executed• CPU-bound applications run at near-baremetal speeds in
virtualized environments• VMM scheduling overheard is low when VMs are not
overprovisioned
• Linux Containers have low performance overhead
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 52/5552/55
Some Conclusions
• CPU pinning can be effective (but be careful)• Performance for a given VMM is dependent on the application
being executed• CPU-bound applications run at near-baremetal speeds in
virtualized environments• VMM scheduling overheard is low when VMs are not
overprovisioned• Linux Containers have low performance overhead
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 52/5552/55
For more info...https://www.usenix.org/legacy/events/sec2000/robin.html
http://www.vmware.com/pdf/asplos235_adams.pdf
http://www.vmware.com/files/pdf/VMware_paravirtualization.pdf
http://software.intel.com/en-us/blogs/2009/06/25/virtualization-and-performance-understanding-vm-exits
http://wiki.xen.org/wiki/Event_Channel_Internals
http://www.docstoc.com/docs/45758154/Understanding-Intel-%C2%AE-Virtualization-Technology-(VT)
http://wiki.openstack.org
http://devstack.org
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 53/5553/55
Slide theme courtesy Flip Tanedo, CornellImages are rights of their respective owners
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 54/5554/55
Machine Statistics
Machine 1:• dual-socket 8 core Intel Xeon E5-2660 2.2GHz CPUs (3.0GHz
Turbo boost), 20MiB cache• 128GiB of DDR3-1333MHz• 8 SAS 10K RAID 10, 1024MiB cacheMachine 2:• dual-socket 6 core Intel Xeon E5645 2.40GHz (2.67 GHz
Turbo boost), 12MiB cache• 32GiB of DDR3-1333Mhz• 1 TiB SATA Disk
Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 55/5555/55
Recommended