1

Open Source Virtualization HacksOpen Source Virtualization Hacks

Niel M. Bornsteinniel@bornstein.atlanta.ga.us

O'Reilly Open Source Convention 2008

2

AgendaAgenda

The Abstract The Technology The Hacks

3

The AbstractThe Abstract

Open source virtualization systems use the usual suite of tools and languages—can anyone out there say Python and XML?

In this session, you’ll see some ways to build a management system, using the tools you already know, to do some wicked things with virtual machines.

Along the way you’ll learn about the Open Virtual Machine Format (OVF) and some of the other emerging standards that are helping modern data centers run.

4

The TechnologyThe Technology

This presentation will make use of the Xen hypervisor on openSUSE 11.0, using libvirt's python bindings.

The hacks and methods presented should be equally applicable to other operating systems and hypervisors supported by libvirt.

5

The Technology – openSUSE 11.0The Technology – openSUSE 11.0

“The openSUSE project is a community program sponsored by Novell.

“Promoting the use of Linux everywhere, openSUSE.org provides free, easy access to the world's most usable Linux distribution, openSUSE.

“The openSUSE project gives Linux developers and enthusiasts everything they need to get started with Linux.”

http://software.opensuse.org/

6

The Technology – Xen 3.2.1The Technology – Xen 3.2.1

“The Xen® hypervisor, the powerful open source industry standard for virtualization, offers a powerful, efficient, and secure feature set for virtualization of x86, x86_64, IA64, PowerPC, and other CPU architectures.

“It supports a wide range of guest operating systems including Windows®, Linux®, Solaris®, and various versions of the BSD operating systems.”

http://xen.org/

7

The Technology – libvirt 0.4.0The Technology – libvirt 0.4.0

“A toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes)

“Free software available under the GNU Lesser General Public License

“A long term stable C API “A set of bindings for common languages “A CIM provider for the DMTF virtualization schema”

http://libvirt.org/python.html

8

The Technology – Python 2.5.2The Technology – Python 2.5.2

“Python is a dynamic object-oriented programming language that can be used for many kinds of software development.”

http://python.org

9

The HacksThe Hacks

What’s a Hack? “Hacks are tools, tips, and tricks that help users

solve problems. They are aimed at intermediate-level power users and scripters.”

✔ http://oreilly.com/hacks/ Eleven Hacks in Three Categories

Exploration Basic Tasks Advanced Tasks

10

Hack #1 – The xm CommandHack #1 – The xm Command

Most Linux geeks like to use the command line. The Xen hypervisor provides a command line management user interface. The xm command allows you to do most common single-host management tasks.NAME       xm ­ Xen management user interface

SYNOPSIS       xm <subcommand> [args]

DESCRIPTION       The xm program is the main interface for managing Xen guest domains.       The program can be used to create, pause, and shutdown domains. It can       also be used to list current domains, enable or pin VCPUs, and attach       or detach virtual block devices.

       The basic structure of every xm command is almost always:

         xm <subcommand> <domain­id> [OPTIONS]

...

11

Hack #1 – The xm Command (cont'd)Hack #1 – The xm Command (cont'd)

The command line is very useful, but can be tricky to do some tasks.

For example, to install a new VM from scratch, you first need to define it using configuration files. Only then can you use the xm create command to start the install process.

✔ More about the configuration file format, and ways to edit it, later.

Because it's a complex process, virt-manager makes it much easier to build a new VM.

12

Hack #2 – The virsh CommandHack #2 – The virsh Command

virsh is another command line tool, using libvirt to provide access to a variety of hypervisors in a generic manner.

More about libvirt later!NAME       virsh ­ management user interface

SYNOPSIS       virsh <subcommand> [args]

DESCRIPTION       The virsh program is the main interface for managing virsh guest       domains. The program can be used to create, pause, and shutdown       domains. It can also be used to list current domains. Libvirt is a C       toolkit to interact with the virtualization capabilities of recent ver‐       sions of Linux (and other OSes). It is free software available under       the GNU Lesser General Public License. Virtualization of the Linux       Operating System means the ability to run multiple instances of Operat‐       ing Systems concurrently on a single hardware system where the basic       resources are driven by a Linux instance. The library aim at providing       long term stable C API initially for the Xen paravirtualization but       should be able to integrate other virtualization mechanisms, it cur‐       rently also support QEmu and KVM....

13

Hack #2 – The virsh Command (cont'd)Hack #2 – The virsh Command (cont'd)

The entire libvirt API is available from the virsh command.

chonju:~ # virshWelcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands       'quit' to quit

virsh # dominfo 1Id:             1Name:           opensuse11UUID:           60c451d0­9009­2bd6­1cfb­f71e9ec6926dOS Type:        linuxState:          blockedCPU(s):         1CPU time:       3.3sMax memory:     393216 kBUsed memory:    393216 kB

virsh # shutdown 1Domain 1 is being shutdown

virsh # 

14

Hack #3 – virt-manager Hack #3 – virt-manager

virt-manager is a project hosted by Red Hat designed to provide a graphical UI for managing virtual machines.

It is written in Python with Glade and GTK+ and uses libvirt to access a variety of virtualization systems.

It is included in most modern Linux distributions.

15

Hack #3 – virt-manager (cont'd)Hack #3 – virt-manager (cont'd)

You can even view multiple virtualization technologies through a single interface.

Select the File -> Open Connection menu command and you will see that you can also connect to a QEMU hypervisor in the same interface, as well as making remote connections to other machines.

16

Hack #3 – virt-manager (cont'd)Hack #3 – virt-manager (cont'd)

Building a new VM is also simple. Click on the New button to bring up the vm-install

process, which can also be launched directly from the command line.

The vm-install wizard will present you with all the options you need to configure a new VM, and it's much easier than manually editing configuration files.

Tip: To build a VM with an ISO install source, loop mount the iso first:

✔ losetup ­f /path/to/iso

17

Hack #4 – libvirt + PythonHack #4 – libvirt + Python

After the command line and the GUI tool, the next step is to build your own tools programmatically.

Using libvirt, you can manage a variety of virtualization systems from a variety of programming languages.

For these examples, we'll use Xen and Python, because, well, it's really easy.

After the first example, I will omit error checking for brevity!

18

Hack #4 – libvirt + Python (cont'd)Hack #4 – libvirt + Python (cont'd)

The following sample code comes from the libvirt website:#!/usr/bin/python

import libvirtimport sys

conn = libvirt.openReadOnly(None)if conn == None:    print 'Failed to open connection to the hypervisor'    sys.exit(1)

try:    dom0 = conn.lookupByName("Domain­0")except:    print 'Failed to find the main domain'    sys.exit(1)

print "Domain 0: id %d running %s" % (dom0.ID(), dom0.OSType())print dom0.info()

19

Hack #5 – Query the Hypervisor's CapabilitiesHack #5 – Query the Hypervisor's Capabilities

Before taking any action using libvirt, it's a good idea to get an idea of what the hypervisor is capable of doing.

libvirt provides a couple of convenient methods for doing this:

virConnect.getCapabilities() returns information in XML format about the hypervisor.

virConnect.getInfo() returns a python dict of information about the physical node.

20

Hack #5 – Query the Hypervisor's Capabilities Hack #5 – Query the Hypervisor's Capabilities (cont'd)(cont'd)

#!/usr/bin/python

import libvirt

conn = libvirt.open(None)

print conn.getCapabilities()

info = conn.getInfo()print "CPU model: %s" % info[0]print "memory: %d kB" % info[1]print "# of CPUs: %d" % info[2]print "CPU freq: %d MHz" % info[3]print "# of NUMA cell: %d" % info[4]print "# of CPU sockets: %d" % info[5]print "# of cores per socket: %d" % info[6]print "# of threads per core: %d" % info[7]

21

Hack #6 – Start/Stop a VMHack #6 – Start/Stop a VM

Now that we know what the hypervisor is capable of, we can start a VM instance.

We've already seen how to do this with the xm command and virt-manager; now we'll do it from a Python script.

#!/usr/bin/python

import sys, libvirt

vmname = sys.argv[1]

conn = libvirt.open(None)dom = conn.lookupByName(vmname)dom.create()

dom = conn.lookupByName(vmname)print "%d started" % dom.ID()

22

Hack #6 – Start/Stop a VM (cont'd)Hack #6 – Start/Stop a VM (cont'd)

Similarly, we can stop a VM, no matter whether it was started through libvirt, the xm command, or virt-manager.

#!/usr/bin/python

import sys, libvirt

vmid = int(sys.argv[1])

conn = libvirt.open(None)dom = conn.lookupByID(vmid)dom.shutdown()

23

Hack #7 – Migrate a VMHack #7 – Migrate a VM

In real life, you may occasionally need to move a running VM from one physical host to another. This is referred to as “migrating” the VM.

I can't demonstrate doing this on a single laptop, but I can show you some code that should work!

24

Hack #7 – Migrate a VM (cont'd)Hack #7 – Migrate a VM (cont'd)#!/usr/bin/python

import sys, getpass, libvirt

vmname = sys.argv[1]uri = sys.argv[2]mydata = "" 

def getCredentials(credentials, data):    for credential in credentials:        print credential[1] + ":",        if credential[0] == libvirt.VIR_CRED_AUTHNAME:            data = sys.stdin.readline()            data = data[0:len(data)­1]            credential[4] = data        elif credential[0] == libvirt.VIR_CRED_PASSPHRASE:            credential[4] = getpass("")        else:            return ­1    return 0

flags = [libvirt.VIR_CRED_AUTHNAME,libvirt.VIR_CRED_PASSPHRASE]auth = [flags,getCredentials,mydata]

localconn = libvirt.open(None)dom = localconn.lookupByName(vmname)remoteconn = libvirt.openAuth(None,auth,0)dom.migrate(remoteconn,libvirt.VIR_MIGRATE_LIVE,None,uri,0)

25

Hack #7 – Migrate a VM (cont'd)Hack #7 – Migrate a VM (cont'd)

In reality, it ends up being much easier to use the xm migrate command to do this. If I had to make a recommendation, I'd say that unless you're building a complete management system, you should just use the xm or virsh tools to migrate your VMs:xm migrate ­­live domain newhostvirsh migrate ­­live domain newhost

In these commands, the domain parameter is either the name or id of the domain, and the newhost parameter is the URI of the new host.

26

Hack #7 – Migrate a VM (cont'd)Hack #7 – Migrate a VM (cont'd)

The /etc/xen/xend­config.sxp file that ships with Xen does not allow migrations. Make the following changes to turn migration on:

Uncomment this line and change no to yes:✔ #(xend­relocation­server no)

Uncomment this line:✔ #(xend­relocation­port 8002)

Uncomment this line:✔ #(xend­relocation­address '')

Customize this line for your security needs:✔ (xend­relocation­hosts­allow '^localhost$ ^localhost\\.localdomain$')

Then restart xend with the command rcxend restart

27

Hack #8 – Edit a VM ConfigurationHack #8 – Edit a VM Configuration

A VM can be thought of as a combination of configuration information and a disk image.

You can change the VM configuration when a VM instance is not running by editing the configuration file (in Xen, that's the file in /etc/xen/vm/vmname).

28

Hack #8 – Edit a VM Configuration (cont'd)Hack #8 – Edit a VM Configuration (cont'd)

It's easy enough to modify a config by hand, once you understand the format.

name="opensuse11"uuid="60c451d0­9009­2bd6­1cfb­f71e9ec6926d"memory=384vcpus=1on_poweroff="destroy"on_reboot="restart"on_crash="destroy"localtime=0keymap="en­us"builder="linux"bootloader="/usr/lib/xen/boot/domUloader.py"bootargs="­­entry=xvda2:/boot/vmlinuz­xen,/boot/initrd­xen"extra=" "disk=[ 'file:/var/lib/xen/images/opensuse11/disk0,xvda,w', ]vif=[ 'mac=00:16:3e:49:b8:b2', ]vfb=['type=vnc,vncunused=1']

29

Hack #8 – Edit a VM Configuration (cont'd)Hack #8 – Edit a VM Configuration (cont'd)

But why do it by hand when we've got Python?#!/usr/bin/python

import sys, os

original = sys.argv[1]new = sys.argv[2]

f = file(original, "rb")lines = f.readlines()dict = {}

for line in lines:    pieces = line.partition("=")    dict[pieces[0]] = eval(pieces[2])

for key in dict.keys():    if key == "name":        dict[key] = new    elif key == "disk":        disks = dict[key]        dict[key] = []        for disk in disks:            dict[key].append(disk.replace(os.path.basename(original),new))    elif key == "vif":        dict[key] = None    elif key == "uuid":        dict[key] = None    if dict[key] != None:        print "%s=%s" % ( key, repr(dict[key]) )

30

Hack #8 – Edit a VM Configuration (cont'd)Hack #8 – Edit a VM Configuration (cont'd)

You can change the configuration of a running VM instance using the xm or virsh commands, or using libvirt.#!/usr/bin/python

import sys, libvirt

vmname = sys.argv[1]maxMemory = int(sys.argv[2])

conn = libvirt.open(None)dom = conn.lookupByName(vmname) print dom.maxMemory() dom.create()        dom.setMaxMemory(maxMemory)dom = conn.lookupByName(vmname)print dom.maxMemory()

31

Hack #9 – Edit a VM ImageHack #9 – Edit a VM Image

Editing a VM image file is called “cracking it open”. The image file needs to be mounted as a loopback filesystem.

The easiest way to crack open a Xen image is to loop mount it using lomount.lomount ­diskimage disk0 ­partition 2 /mnt Bear in mind that the image file will almost

certainly have more than one partition. This method will not work with LVM disks!

32

Hack #10 – Clone a VMHack #10 – Clone a VM

Now we have edited a VM configuration and image. Cloning a VM is as simple as copying the

configuration and image to a new location, then editing the configuration and “personalizing” the image.

You just saw how our friend Python can easily read in a configuration file and make the necessary changes.

33

Hack #10 – Clone a VM (cont'd)Hack #10 – Clone a VM (cont'd)

Just one little problem: my disk image is 8 gigs. That takes a while to copy

The solution: Copy-on-Write. Use the QEMU CoW (qcow) format Convert your raw image to qcow using img2qcow

img2qcow destination source There is also a corresponding qcow2raw

34

Hack #11 – Create an ApplianceHack #11 – Create an Appliance

Appliances can be defined using Open Virtual Machine Format (OVF), a standard proposed by VMware under the auspices of the Distributed Management Task Force (DMTF).

OVF is: A packaging format for software applications

✔ Bundle together your VMs for a self-contained application

A transport mechanism for virtual machine templates

✔ Distribute your OVFs as images which must be installed before they can be run

35

Hack #11 – Create an Appliance (cont'd)Hack #11 – Create an Appliance (cont'd)

OVF is not: An efficient execution environment

✔ VMs must be installed before they can be run A format requiring a hypervisor

✔ By itself, OVF does not have any dependence on a hypervisor

36

Hack #11 – Create an Appliance (cont'd)Hack #11 – Create an Appliance (cont'd)

The OVF file is a zip file containing An XML file (.ovf) with schema

http://schemas.dtmf.org/ovf/envelope✔ May refer to external URIs for actual VM images✔ May contain virtual hardware resource

descriptions May contain a manifest file (.mf) May contain a certificate file (.cert)

37

Hack #11 – Create an Appliance (cont'd)Hack #11 – Create an Appliance (cont'd)

Today, there is no OVF tool for Xen, only VMware! IBM is sponsoring an Open OVF project

http://www.xen.org/files/xensummitboston08/open-ovf-proposal.pdf

38

Hack #11 – Create an Appliance (cont'd)Hack #11 – Create an Appliance (cont'd)

For another take on appliances, check out Nat Friedman's talk:

The Future of Linux is Software Appliances✔ Friday 10:45 in Portland 255

39

Q&AQ&A