Virtualize with KVMTools and Enterprise usage

by Vincent Van der Kussen

#whoami

● Using Linux since 1996

● Sysadmin / Engineer / Opensource Enthousiast

● @vincentvdk

● http://blog.vanderkussen.org

So what is this KVM thing anyway?

It's NOT!

KVM : overview

Kernel-based Virtual Machine, a Kernel module that turns Linux into a hypervisor

Created by Quramnet (bought by Red Hat 2008)

Supports x86 (32 and 64 bit), s390, Powerpc

Included in Linux kernel since 2.6.20

Needs a CPU with virtualization extention !

KVM : overview

● Each vm is a process

● Each virtual cpu is a thread

● Open source

Features

CPU and memory overcommit

High performance paravirtual I/O

Hotplug (cpu, block, nic)

SMP guests

Live Migration

Power management

Features

PCI Device Assigenment and SR-IOV

KSM (Kernel Samepage Merging)

SPICE

NUMA

Not your typical VMWare workstation or VirtualBox.

VirtManager comes close but still a little rough on the edges

KVM is meant for virtualization on servers. Desktop usage is not always ideal (think vSphere, HyperV,..).

KVM : differences

Overview Tools

libvirt (api)

Virsh

Qemu

VirtManager

Libguestfs

virt-v2v

Overview tools

sVirt

virt-*

Boxgrinder

VirtIO (not really a tool)

Spice

Libvirt : The Virtualization API

Libvirt : overview

● “The Virtualization API”

● Toolkit to interact with the virtualization layer

● Written in C

● LGPL● Has bindings for common languages (Python,

Ruby, Perl, PHP, Java,..)● Supports AMQP via libvirt-qpid

Libvirt : overview

● Provides management of

– Virtual machines

– Virtual networks

– Storage

● Does NOT provide multi node management features like load balancing

● Suspend / resume support

Libvirt : terminology

Node

a physical machine

Libvirt : termonology

Hypervisor

A software layer that allows a node to run virtual

machines

Libvirt : terminologie

Domain

An instance (“virtual machine”) running on the

hypervisor

Libvirt : overview

Supports also :

XEN KVM

VMWare

LXC

OpenVZ

Microsoft HyperV

Virtualbox

virsh

Virsh overview

Command line for libvirt

Enables scripting

Libvirtd needs to be started

Most commands require root privileges

Virsh usage

virsh <command> <domain-id> [OPTIONS]virsh <command> <domain-id> [OPTIONS]

Virsh usage

Connect to hypervisor on the localhost

virsh connect qemu:///systemvirsh connect qemu:///system

Connect to hypervisor on remote system over SSH

virsh -c qemu+ssh://user@host.domain.foo/systemvirsh -c qemu+ssh://user@host.domain.foo/system

virshvirsh

or

Virsh usage

Virsh : storage

● Define storage pool● Local filesystem (ext3, ext4,...) ● Directory based● ISCSI● NFS● LVM (uses volume group)● Physical disk● multipath

Virsh : create storage pool

[root@localhost ~]# virsh pool-define dir_pool.xml Pool LOCAL-STORAGE defined from dir_pool.xml[root@localhost ~]# virsh pool-define dir_pool.xml Pool LOCAL-STORAGE defined from dir_pool.xml

[root@localhost ~]# virsh pool-start LOCAL-STORAGEPool LOCAL-STORAGE started[root@localhost ~]# virsh pool-start LOCAL-STORAGEPool LOCAL-STORAGE started

<pool type="dir"><name>LOCAL-STORAGE</name> <target> <path>/VMS</path> </target></pool>

Xml file ->

Virsh : create storage pool

[root@localhost ~]# virsh pool-listName State Autostart -----------------------------------------default active yes LOCAL-STORAGE active no

[root@localhost ~]# virsh pool-listName State Autostart -----------------------------------------default active yes LOCAL-STORAGE active no

[root@localhost ~]# virsh pool-autostart LOCAL-STORAGEPool LOCAL-STORAGE marked as autostarted

[root@localhost ~]# virsh pool-autostart LOCAL-STORAGEPool LOCAL-STORAGE marked as autostarted

Virsh : create VM

● VM is defined in xml file

● Manual creation of VM's disks

[root@localhost ~]# qemu-img create /VMS/vm02-fromxml.img 5GFormatting '/VMS/vm02-fromxml.img', fmt=raw size=5368709120 [root@localhost ~]# qemu-img create /VMS/vm02-fromxml.img 5GFormatting '/VMS/vm02-fromxml.img', fmt=raw size=5368709120

● Define VM in libvirt and start it

[root@localhost ~]# virsh define vm02_from_xml.xmlDomain vm02-fromxml defined from vm02_from_xml.xml

[root@localhost ~]# virsh start vm02-fromxmlDomain vm02-fromxml started

[root@localhost ~]# virsh define vm02_from_xml.xmlDomain vm02-fromxml defined from vm02_from_xml.xml

[root@localhost ~]# virsh start vm02-fromxmlDomain vm02-fromxml started

1 <domain type='kvm'> 2 <name>vm02-fromxml</name> 3 <uuid></uuid> 4 <memory>128288</memory> 5 <currentMemory></currentMemory> 6 <vcpu>2</vcpu> 7 <os> 8 <type arch='x86_64' machine='pc-0.14'>hvm</type> 9 <boot dev='hd'/>10 </os>11 <features>12 # features such as ACPI are defined here13 </features>14 <devices>15 <emulator>/usr/bin/qemu-kvm</emulator>16 <disk type='file' device='disk'>17 <driver name='qemu' type='raw' cache='none'/>18 <source file='/VMS/vm02-fromxml.img'/>19 <target dev='vda' bus='virtio'/>20 <alias name='virtio-disk0'/>21 <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>22 </disk>23 </devices>24 </domain>

Virsh : list vm's

[root@localhost ~]# virsh list --all Id Name State---------------------------------- 4 vm01 running - vm02-fromxml shut off

[root@localhost ~]#

[root@localhost ~]# virsh list --all Id Name State---------------------------------- 4 vm01 running - vm02-fromxml shut off

[root@localhost ~]#

Virsh : live migration

virsh migrate --live <guestname> qemu+ssh://<destination host>/systemvirsh migrate --live <guestname> qemu+ssh://<destination host>/system

Virsh : snapshots

● Only on qcow2 images

# virsh snapshot-create <domain># virsh snapshot-create <domain>

# virsh snapshot-list <domain># virsh snapshot-list <domain>

Name Creation Time State---------------------------------------------------

1295973577 2011-01-25 17:39:37 +0100 running 1295978837 2011-01-25 19:07:17 +0100 running

Name Creation Time State---------------------------------------------------

1295973577 2011-01-25 17:39:37 +0100 running 1295978837 2011-01-25 19:07:17 +0100 running

# virsh snapshot-restore <domain> <snapshotname># virsh snapshot-restore <domain> <snapshotname>

Qemu

qemu

● KVM istself cannot create VM's

● KVM is “just” a hypervisor

● Qemu as user space process

● Qemu communicates via /dev/kvm

qemu

● Qemu-img

– Check

– Convert

– Create

– Snapshot

– Info

Qemu

[root@HV01 ~]# qemu-img info /var/lib/libvirt/images/WIN7.img image: /var/lib/libvirt/images/WIN7.imgfile format: rawvirtual size: 12G (12884901888 bytes)disk size: 12G[root@HV01 ~]#

[root@HV01 ~]# qemu-img info /var/lib/libvirt/images/WIN7.img image: /var/lib/libvirt/images/WIN7.imgfile format: rawvirtual size: 12G (12884901888 bytes)disk size: 12G[root@HV01 ~]#

Inspect image

qemu

Example :

Convert raw to vmdk image

qemu-img convert -O vmdk <raw image> outimage.vmdkqemu-img convert -O vmdk <raw image> outimage.vmdk

Qemu

● file

● blkverify

● sheepdog

● blkdebug

● qcow2

● vvfat

● cow

● Raw

● vmdk

● vdi

● qcow

Supported disk formats

virtio

● "full virtualization" is a nice feature because It allows you to run any operating system virtualized.

● However, it's slow because the hypervisor has

to emulate actual physical devices.

virtio

● Virtio drivers solve this

● Virtio drivers know they run in a virtualized environment. (no emulation)

● Virtio drivers operate with hypervisor

virt-*

virt-*

● Virt-install

● Virt-viewer

● Virt-clone

● virt-top

Virt-install

[root@localhost ~]# virt-install --connect qemu:///system --graphics vnc,listen=0.0.0.0 --name=vm01 --ram 512 --disk pool=LOCAL-STORAGE,size=10,bus=virtio,cache=none /--network network=default,model=virtio /--location=http://ftp.debian.org/debian/dists/squeeze/main/installer-amd64 Cannot open display: Run 'virt-viewer --help' to see a full list of available command line optionsDomain installation still in progress. You can reconnect to the console to complete the installation process.

[root@localhost ~]# virt-install --connect qemu:///system --graphics vnc,listen=0.0.0.0 --name=vm01 --ram 512 --disk pool=LOCAL-STORAGE,size=10,bus=virtio,cache=none /--network network=default,model=virtio /--location=http://ftp.debian.org/debian/dists/squeeze/main/installer-amd64 Cannot open display: Run 'virt-viewer --help' to see a full list of available command line optionsDomain installation still in progress. You can reconnect to the console to complete the installation process.

● Can create VM and disks in one go

Virt-viewer

● Connect to the VM's console with – RDP – VNC – SPICE

● Default vnc port -> 5900

● virt-viewer

virt-viewer -c qemu:///system xpsp2virt-viewer -c qemu:///system xpsp2

virt-clone

● Clones an existing VM (Duh!!)

● Copies disk images

● Defines new machine based on source

– MAC address is auto generated

● Deploy many similar machines from one image

(template).

[root@HV01 ~]# virt-clone -o vm01 -n vm01-copy -f / /VMS/vm01-copy.imgAllocating 'vm01-copy.img' | 10 GB 03:37

Clone 'vm01-copy' created successfully.[root@HV01 ~]#

[root@HV01 ~]# virt-clone -o vm01 -n vm01-copy -f / /VMS/vm01-copy.imgAllocating 'vm01-copy.img' | 10 GB 03:37

Clone 'vm01-copy' created successfully.[root@HV01 ~]#

virt-clone

virt-top

virt-top 20:40:19 - x86_64 2/2CPU 2666MHz 3962MB4 domains, 4 active, 4 running, 0 sleeping, 0 paused, 0 inactive D:0 O:0 X:0CPU: 2.8% Mem: 2560 MB (2560 MB by guests)

ID S RDRQ WRRQ RXBY TXBY %CPU %MEM TIME NAME 14 R 0 0 1220 0 2.5 25.0 315:52.95 WIN7 15 R 0 4 146 42 0.2 12.0 0:54.64 vm01 11 R 0 0 0.2 12.0 15:38.34 vm02-fromxml 16 R 0 0 104 0 0.0 12.0 0:54.82 vm01-copy

virt-top 20:40:19 - x86_64 2/2CPU 2666MHz 3962MB4 domains, 4 active, 4 running, 0 sleeping, 0 paused, 0 inactive D:0 O:0 X:0CPU: 2.8% Mem: 2560 MB (2560 MB by guests)

ID S RDRQ WRRQ RXBY TXBY %CPU %MEM TIME NAME 14 R 0 0 1220 0 2.5 25.0 315:52.95 WIN7 15 R 0 4 146 42 0.2 12.0 0:54.64 vm01 11 R 0 0 0.2 12.0 15:38.34 vm02-fromxml 16 R 0 0 104 0 0.0 12.0 0:54.82 vm01-copy

Virtual machine manager

Virtual Machine Manager

● A GUI for virsh and virt-*

● Runs only on Linux/Unix

● Can discover machines running libvirtd (nodes)

● Local development / testing

Virtual Machine Manager

libguestfs

libguestfs :Converting, Inspecting and modifying VM's

libguestfs

● Manipulation of VM images

● Inspection of VM images

● Exists out of several separate tools

– List is available on http://virt-tools.org/docs/index.html

● No need to login on the VM itself

● Can be run on an active VM

What is this image?

Developers like to keep stuff

Is it still needed?

Libguestfs : virt-inspector

● What if you just have an image

● You cannot login to machine

● You would like to know what version of a

package is installed

● Generate list of all VM's

Libguestfs : virt-inspector

virt-inspector -d WIN7virt-inspector -d WIN7

[root@HV01 VMS]# virt-inspector -a /var/lib/libvirt/images/WIN7.img [root@HV01 VMS]# virt-inspector -a /var/lib/libvirt/images/WIN7.img

Inspect the domain

Inspect an image

1 <name>windows</name>

2 <arch>i386</arch>

3 <distro>windows</distro>

4 <product_name>Windows 7 Professional</product_name>

5 <product_variant>Client</product_variant>

6 <major_version>6</major_version>

7 <minor_version>1</minor_version>

8 <windows_systemroot>/Windows</windows_systemroot>

9 <windows_current_control_set>ControlSet001</windows_current_control_set>

10 <hostname>btr-PC</hostname>

11 <format>installed</format>

1 <applications>

2 <application>

3 <name>Mozilla Firefox 6.0.2 (x86 en-US)</name>

4 <display_name>Mozilla Firefox 6.0.2 (x86 en-US)</display_name>

5 <version>6.0.2</version>

6 <install_path>C:\Program Files\Mozilla Firefox</install_path>

7 <publisher>Mozilla</publisher>

8 <url>http://www.mozilla.com/en-US/</url>

9 <description>Mozilla Firefox 6.0.2 (x86 en-US)</description>

10 </application>

11 </applications>

Libguestfs : virt-df

[root@HV01 ~]# virt-df vm02-fromxmlFilesystem 1K-blocks Used Available Use%vm02-fromxml:/dev/sdb 3516418 3516418 0 100%vm02-fromxml:/dev/sda1 495844 30573 439671 7%vm02-fromxml:/dev/VolGroup/lv_root 3652680 623180 2843952 18%[root@HV01 ~]#

[root@HV01 ~]# virt-df vm02-fromxmlFilesystem 1K-blocks Used Available Use%vm02-fromxml:/dev/sdb 3516418 3516418 0 100%vm02-fromxml:/dev/sda1 495844 30573 439671 7%vm02-fromxml:/dev/VolGroup/lv_root 3652680 623180 2843952 18%[root@HV01 ~]#

● Allows you to view filesystem layout and space free/used

Virt-df <name of domain>

Libguestfs : virt-win-reg

● Modify registry of Windows machines

virt-win-reg --merge <vmname>.img viostor-regvirt-win-reg --merge <vmname>.img viostor-reg

Libguestfs : virt-win-reg

guestfish -i <vmname>.img

<fs>upload viostor.sys /WINDOWS/system32/drivers/viostor.sys

guestfish -i <vmname>.img

<fs>upload viostor.sys /WINDOWS/system32/drivers/viostor.sys

virt-win-reg --merge <vmname>.img viostor-regvirt-win-reg --merge <vmname>.img viostor-reg

Manually Prepare a Windows image with virtio drivers

Libguestfs : virt-tar-out

● Create backups from files on the VM

# virt-tar-out -d vm01 /home home-vm01.tar# virt-tar-out -d vm01 /home home-vm01.tar

● Create backups from files on the disk image

# virt-tar-out -a /VMS/vm01.img /home home-vm01-disk.tar# virt-tar-out -a /VMS/vm01.img /home home-vm01-disk.tar

virt-v2v : migration of virtual machines

Virt-v2v : short overview

● Tool to automate migrations to KVM/RHEV

● Migrate to KVM or RHEV from

– XEN

– VMWare

– KVM

● Installs virtio drivers and reconfigures machine

● Can be scripted for bulk operations

Virt-v2v : migrate from Vmware to RHEV

Virt-v2v : migrate from VMWare to RHEV

virt-v2v -ic esx://esx.example.com/?no_verify=1 -o rhev -os storage.example.com:/exportdomain --network rhevm vm-name

virt-v2v -ic esx://esx.example.com/?no_verify=1 -o rhev -os storage.example.com:/exportdomain --network rhevm vm-name

Virt-v2v : migrate VMWare to KVM

sVirt : secure your host and VM's

sVirt

Virtualization introduces new risks

sVirt

“Old” days

Machines where physically separated and “attackable” via the network

sVirt

With Virtualization :

A flawed hypervisor can give access to ALL VM's running on it

Bypassing network security

sVirt

sVirt :

Apply MAC for guest and resources

MAC policy is set by hypervisor host

Based on SELinux (don't reinvent the wheel)

Auto apply or manual

[root@HV01 VMS]# ls -Z |grep vm01qemu qemu system_u:object_r:svirt_image_t:s0:c656,c817 vm01-copy.imgqemu qemu system_u:object_r:svirt_image_t:s0:c690,c972 vm01.img

[root@HV01 VMS]# ls -Z |grep vm01qemu qemu system_u:object_r:svirt_image_t:s0:c656,c817 vm01-copy.imgqemu qemu system_u:object_r:svirt_image_t:s0:c690,c972 vm01.img

Disk images

Processes

[root@HV01 VMS]# ps auxZ | grep vm01 |awk '{print $1" " $22}' system_u:system_r:svirt_t:s0:c690,c972 vm01system_u:system_r:svirt_t:s0:c656,c817 vm01-copy

[root@HV01 VMS]# ps auxZ | grep vm01 |awk '{print $1" " $22}' system_u:system_r:svirt_t:s0:c690,c972 vm01system_u:system_r:svirt_t:s0:c656,c817 vm01-copy

A lot of tools but what to do with them?

RHEV is first example

Build your own stack?

Development environment?

Enterprise usage

RHEV 2.x

● RHEV (Red Hat enterprise Virtualization)

– Red Hat started adopting KVM (actually they bought it)

– Released 2.1

– Running on Windows platform ( the horror.. )

– RHEV 2.2 will be latest release in 2.x series

RHEV 3.0

● Port from .Net to Java

● Manager runs on Jboss

– no more windows server side

● Hypervisor based on RHEL 6

– Gives performance increase

● Local storage

● Reporing engine included

● WAN optimized SPICE

RHEV 3.0

● Power User portal

– Users can manage their own environment

● RESTful API

● Will be open sourced

RHEV adoption

OVA

● Open Virtualization Alliance● Foster the adoption of KVM as an enterprise-

ready open virtualization solution ● Accelerate the emergence of an ecosystem of

third-party solutions around KVM.● Encourage interoperability, promote best

practices, and highlight examples of customer successes.

Ova : members

specvirt

● Virtualization benchmark (score@ 'n' VM's)

● KVM

– 3894@240

● VmWare ESX 4.1

– 3723@228

conclusions

● KVM is till young

● Adoption is growing

● Builds on Linux features (feature velocity)

– Selinux, Cgroups, Scheduler, ...

● A lot of community activity

● scalability

● Growing ECO system

● No vendor lockin

● OVA (open virtualization alliance)

Questions?

http://www.flickr.com/photos/kalavinka/4617897952/

http://www.flickr.com/photos/zakh/337938459/

http://www.cyberciti.biz/faq/linux-kvm-vnc-for-guest-machine/

http://www.centos.org/docs/5/html/5.2/Virtualization/chap-Virtualization-Managing_guests_with_virsh.html

http://heprc.phys.uvic.ca/sites/heprc.phys.uvic.ca/files/reports/vliet-wtr.pdf

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Virtualization/index.html

Grind out appliances

BoxGrinder : overview

● Create a VM “offline”

● Runs on JBOSS

● Use puppet/chef/.. to configure it further

● Move from Fedora to RHEL by adjusting the file

●