Installing and Configuring Xen

8/4/2019 Installing and Configuring Xen

1/54

Blueprints

Installing and configuring Xen onRed Hat Enterprise Linux 5.2 on System x


2/54


3/54

Blueprints

Installing and configuring Xen onRed Hat Enterprise Linux 5.2 on System x


4/54

NoteBefore using this information and the product it supports, read the information in Appendix F,Notices, on page 43.

First Edition (June 2008)

Copyright International Business Machines Corporation 2008.US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.


5/54

Contents

Preface . . . . . . . . . . . . . . . vIntroduction . . . . . . . . . . . . . . v

Installing and configuring Xen . . . . . 1Installing required Xen-related software . . . . . 1

Installing Xen during RHEL 5.2 install . . . . . 1Installing Xen after RHEL 5.2 install . . . . . 3

Making Xen the default boot kernel . . . . . . 4For NUMA machines . . . . . . . . . . 4Verify that your machine is running correctly inNUMA mode . . . . . . . . . . . . . 4For Non-NUMA machines . . . . . . . . . 5

Creating Paravirtualized guests . . . . . . . . 6Creating Paravirtualized guests on NUMAmachines . . . . . . . . . . . . . . 6Configuring CPU assignments (CPU pinning in

NUMA machines) . . . . . . . . . . . 9Creating paravirtualized guests on non-NUMAmachines . . . . . . . . . . . . . . 12

Configuring Xen for performance . . . . . . . 22Domain0 . . . . . . . . . . . . . . 23Other performance factors . . . . . . . . 24

Appendix A. Troubleshooting . . . . . 31

Appendix B. NUMA and the wrapperscript . . . . . . . . . . . . . . . 33

Appendix C. Common Xen commands 35xm create [-c] configfile [name=value] . . . . . . 35xm info . . . . . . . . . . . . . . . . 35xm list [domain-ids] . . . . . . . . . . . . 36xm shutdown [options] domain-id . . . . . . . 36virsh list . . . . . . . . . . . . . . . 36virsh create FILE. . . . . . . . . . . . . 37virsh destroy domain-id . . . . . . . . . . 37virsh dumpxml domain-id . . . . . . . . . . 37virsh shutdown domain-id. . . . . . . . . . 37

Appendix D. Source code. . . . . . . 39

Appendix E. Related information forXen . . . . . . . . . . . . . . . . 41

Appendix F. Notices . . . . . . . . . 43Trademarks . . . . . . . . . . . . . . 44

Copyright IBM Corp. 2008 iii


6/54

iv Blueprints: Installing and configuring Xen on Red Hat Enterprise Linux 5.2 on System x


7/54

Preface

Introduction

This blueprint provides step-by-step instructions for installing and configuring a Xen host and paravirtualguests on RHEL5.2 based NUMA and non_NUMA running on System x hardware.

Intended audience

This information is intended for Linux system administrators with a moderate level of knowledge ofsystem administration, but little to no knowledge of server virtualization using the Xen hypervisor.

Scope and purpose

This blueprint provides step-by-step instructions for installing and configuring a Xen host and guests onSystem x servers running Red Hat Enterprise Linux 5.2 operating system. All the steps that aredemonstrated here were tested on various System x platforms running the Red Hat Enterprise Linux 5.2

operating system.

Hardware and software requirements

For hardware and software requirements, see the Red Hat Enterprise Linux Virtualization guide(http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/index.html).

IBM Services

Linux offers flexibility, options, and competitive total cost of ownership with a world class enterpriseoperating system. Community innovation integrates leading edge technologies and best practices intoLinux.

IBM is a leader in the Linux community with over 600 developers in the IBM Linux Technology Centerworking on over 100 open source projects in the community. IBM supports Linux on all IBM servers,storage, and middleware, offering the broadest flexibility to match your business needs.

For more information about IBM and Linux, visit us at ibm.com/linux (http://www.ibm.com/linux)

IBM Support

Questions and comments regarding this documentation and the related IBM scripts may be posted on theDeveloperWorks Xen Virtualization Blueprint Community Forum:

http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1272

The IBM developerWorks discussion forums let you ask questions, share knowledge, ideas, and opinionsabout technologies and programming techniques with other developerWorks users. Use the forumcontent at your own risk. While IBM will attempt to provide a timely response to all postings, the use ofthis DeveloperWorks forum does not guarantee a response to every question that is posted, nor do wevalidate the answers or the code that are offered.

Copyright IBM Corp. 2008 v
https://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/index.htmlhttp://www.ibm.com/linuxhttp://www.ibm.com/developerworks/forums/forum.jspa?forumID=1272http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1272http://www.ibm.com/linuxhttps://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/index.html


8/54

Typographic conventions

The following typographic conventions are used in this blueprint:

Bold Identifies commands, subroutines, keywords, files, structures, directories, and otheritems whose names are predefined by the system. Also identifies graphical objectssuch as buttons, labels, and icons that the user selects.

Italics Identifies parameters whose actual names or values are to be supplied by the user.

Monospace Identifies examples of specific data values, examples of text similar to what youmight see displayed, examples of portions of program code similar to what youmight write as a programmer, messages from the system, or information youshould actually type.

vi Blueprints: Installing and configuring Xen on Red Hat Enterprise Linux 5.2 on System x


9/54

Installing and configuring Xen

Xen is a virtual machine monitor (also known as a hypervisor) which allows one to deploy multiple virtualservers on a single physical server. Using software to make a single server appear to be many servers is

known as virtualization. Virtualization offers many benefits such as server consolidation, increasedutilization of server resources, and the ability to simultaneously run a mix of operating systems and theirapplications on a single hardware platform.

Installing required Xen-related software

You can install Xen and related software during your Red Hat Enterprise Linux 5.2 install or after RedHat Enterprise Linux 5.2 has already been installed.

Installing Xen during RHEL 5.2 installDuring a Red Hat Enterprise Linux 5.2 install, you have the option to install the Virtualization packagegroup. This package group includes all the needed software for setting up the Xen environment.

To install the Virtualization package group, follow these steps:

1. Provide the Service + Virtualization Installation Number when prompted at the beginning of theinstall.

Copyright IBM Corp. 2008 1


10/54

2. Continue with partitioning, network setup, time zone, and root password steps.

3. When prompted to choose additional tasks, select Virtualization.

Figure 1. Installation number

2 Blueprints: Installing and configuring Xen on Red Hat Enterprise Linux 5.2 on System x


11/54

4. Continue with the install process.

5. If your machine is non-NUMA, skip the Making Xen the default boot kernel on page 4 topic as it is

done automatically for you.

Note: The Virtualization package group is not available in the additional task menu if the InstallationNumber was not provided in the beginning of the install.

Installing Xen after RHEL 5.2 installIf your system already has RHEL5.2 installed, you can use yum to install the Virtualization packagegroup that includes all the Xen-related software required.

Follow these instructions:

1. Run yum groupinstall Virtualization to install Virtualization (Xen) packages.

2. The Red Hat installer installs the Xen-enabled kernel and associated packages. The following packagesare installed:

Virtualization-en-US noarchgnome-applet-vm x86_64kernel-xen x86_64libvirt x86_64libvirt i386virt-manager x86_64virt-viewer x86_64xen x86_64

Note: The xen-libs and libvirt packages may be listed twice. You can ignore this anomaly.

Figure 2. Virtualization task

Installing and configuring Xen 3


12/54

3. If one or more of the packages are missing, run yum install to install the missingpackages.

Making Xen the default boot kernel

You may decide that it is more convenient if in every reboot, the machine is booted to the Xen kernel sothat you can bring up the Xen environment. If you have installed the virtualization package group during

the Red Hat Enterprise Linux 5.2 install, and if your machine is non-NUMA, you can skip this section asthe Xen kernel should now be the default kernel.

You can go to IBM BladeCenter and System x Reference Sheets (http://www.redbooks.ibm.com/Redbooks.nsf/pages/xref) and find the model number of your machine to check if it is a Non UniformMemory Architecture (NUMA) machine.

For NUMA machinesThis topic describes the process for specifying the Xen kernel as the default boot kernel for NUMAmachines.

1. Edit /boot/grub/menu.lst and specify the Xen kernel as the default boot kernel.

2. Make sure that default=0 is set and the first entry in the file is the Xen kernel you want to boot. Addnuma=on to the kernel line. The edited file should look like this:

# grub.conf generated by anaconda## Note that you do not have to rerun grub after making changes to this file# NOTICE: You have a /boot partition. This means that# all kernel and initrd paths are relative to /boot/, eg.# root (hd0,0)# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00# initrd /initrd-version.img#boot=/dev/sdadefault=0

timeout=5splashimage=(hd0,0)/grub/splash.xpm.gzhiddenmenutitle Red Hat Enterprise Linux Server (2.6.18-86.el5xen)

root (hd0,0)kernel /xen.gz-2.6.18-86.el5 numa=onmodule /vmlinuz-2.6.18-86.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quietmodule /initrd-2.6.18-86.el5xen.img

These changes ensure that Xen is the default kernel and NUMA is enabled.

Note: In Red Hat Enterprise Linux 5.2, there is a known issue when booting NUMA on machines

with more than 256GB memory. Please read Red Hat Enterprise Linux 5.2 Release Notes(http://www.redhat.com/docs/manuals/enterprise/) for more information. The workaround isto add xenheap_megabytes=64 dom0_mem=512m along with numa=on as the kernel parameters.

3. Once the correct packages are installed, verify that the Xen kernel is booted:

# uname -r2.6.xx-yy.el5xen

4. Xen-enabled kernels from Red Hat contain the string xen within the version. If some other kernel is booted, check the /boot/grub/menu.lst file. Reboot the system, then verify that uname -r output iscorrect.

Verify that your machine is running correctly in NUMA modeBefore you go any further, you should verify that your machine is running correctly in NUMA mode.

Enter the following command:

http://www.redbooks.ibm.com/Redbooks.nsf/pages/xrefhttp://www.redbooks.ibm.com/Redbooks.nsf/pages/xrefhttp://www.redbooks.ibm.com/Redbooks.nsf/pages/xrefhttp://www.redbooks.ibm.com/Redbooks.nsf/pages/xrefhttp://www.redbooks.ibm.com/Redbooks.nsf/pages/xrefhttp://www.redhat.com/docs/manuals/enterprise/http://www.redhat.com/docs/manuals/enterprise/http://www.redbooks.ibm.com/Redbooks.nsf/pages/xref


13/54

# xm info | grep nr_nodes

If nr_nodes is reported to be 1, NUMA mode is off or you do not have NUMA aware machine. Check ifnuma=on is passed to the Xen kernel when booting. If numa=on is not passed as kernel parameter, thensee For NUMA machines on page 4 for more information.

You can find more information about xm info command in the xm info on page 35.

For Non-NUMA machinesThis topic describes the process for specifying the Xen kernel as the default boot kernel for non-NUMAmachines. If you installed virtualization package during installation, you can skip this step.

1. Edit /boot/grub/menu.lst and specify the Xen kernel as the default boot kernel.

2. Make sure that default=0 is set and the first entry in the file is the Xen kernel that you want to boot.

Figure 3. xm info output



14/54

# grub.conf generated by anaconda## Note that you do not have to rerun grub after making changes to this file# NOTICE: You have a /boot partition. This means that# all kernel and initrd paths are relative to /boot/, eg.# root (hd0,0)# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00# initrd /initrd-version.img

#boot=/dev/sdadefault=0timeout=5splashimage=(hd0,0)/grub/splash.xpm.gzhiddenmenutitle Red Hat Enterprise Linux Server (2.6.18-86.el5xen)

root (hd0,0)kernel /xen.gz-2.6.18-86.el5module /vmlinuz-2.6.18-86.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quietmodule /initrd-2.6.18-86.el5xen.img

3. Once the correct packages are installed, verify that the Xen kernel is booted:

# uname -r2.6.xx-yy.el5xen

4. Xen-enabled kernels from Red Hat contain the string xen within the version. If some other kernel is

booted, check the /boot/grub/menu.lst file. Reboot the system, then verify that uname -r output isnow correct.

Creating Paravirtualized guests

You can create a paravirtualized guest on a NUMA or non-NUMA machine.

Creating Paravirtualized guests on NUMA machinesThis topic describes creating paravirtualized guests on NUMA-capable machines.

If your machine is NUMA-capable, you need to download a special script file, wrapper.py, from XENNUMA wrapper code (http://www.ibm.com/developerworks/apps/download/

index.jsp?contentid=310979&filename=bsdwrapper.zip&method=http). After you have copied wrapper.py,follow these instructions to install one or more guest operating systems.:

1. Run chmod +x wrapper.py to make it an executable.

2. Run the following command to start installing guests.

./wrapper.py -name=NAME -file=DISKFILE -file-size=DISKSIZE -ram=MEMORY-location=LOCATION -p -nographics

Table 1. Description of options for ./wrapper.py

Option Description

NAME The name of the new guest virtual machine instance. This must be unique amongst all guestsknown to the hypervisor on this machine, including those not currently active. To redefine anexisting guest, use the virsh(1) tool to shut it down and delete it prior to running wrapper.py.This parameter will be prompted for if omitted on the command line.

DISKFILE The path to the file, disk partition, or logical volume to use as the backing store for the guestsvirtual disk. If the path does not exist, then --file-size option should also be specified,allowing the disk to be created. This parameter can be repeated multiple times to add manydisk. This parameter will be prompted for if omitted on the command line.

DISKSIZE The size of the file to create for the guest virtual disk, if the path given to --file does notalready exist. The size is to be specified in gigabytes, with fractional numbers allowed. Thestorage for the file will not be pre-allocated unless the --nonsparse option is also specified.This parameter will be prompted for if omitted on the command line & a new file is to becreated.

http://www.ibm.com/developerworks/apps/download/index.jsp?contentid=310979&filename=bsdwrapper.zip&method=httphttp://www.ibm.com/developerworks/apps/download/index.jsp?contentid=310979&filename=bsdwrapper.zip&method=httphttp://www.ibm.com/developerworks/apps/download/index.jsp?contentid=310979&filename=bsdwrapper.zip&method=httphttp://www.ibm.com/developerworks/apps/download/index.jsp?contentid=310979&filename=bsdwrapper.zip&method=http


15/54

Table 1. Description of options for ./wrapper.py (continued)

Option Description

MEMORY The amount of RAM to allocate for guest instance in megabytes. If the hypervisor does nothave enough free memory, it is usual for it to automatically take memory away from the hostoperating system to satisfy this allocation. This parameter will be prompted for if omitted onthe command line.

LOCATION DIRECTORYPath to a local directory containing an installable distribution image

nfs:host:/pathAn NFS server location containing an installable distribution image

http://host/pathAn HTTP server location containing an installable distribution image

ftp://host/pathAn FTP server location containing an installable distribution image

-p Forces this guest to be a paravirtualized guest. If the host supports both para and fullvirtualization, and neither this parameter nor the --hvm are specified, this will be promptedfor interactively.

--nographics Disables all interactive prompts for the guest virtual console. No graphical console will beallocated for the guest. A text based console will always be available on the first serial port (orequivalent paravirtualised console device).

For example, the following command creates a paravirtualized guest named firstVM. A disk file of15.5 GB is created at /mnt/xen/firstVM/xvda. It is assigned 512 MB of memory.

./wrapper.py -name=firstVM -file=/mnt/xen/firstVM/xvda -file-size=15.5 -ram=512-location=http://installserver/rhel5.2/ -p --nographics

3. At this point, the wrapper starts and brings up the screens to configure the guest operating systeminstallation. It displays the screen similar to the screen Figure 4 on page 8 to select network for guestOS. You can choose to go with the default which is DHCP configuration or select static IP for theguest.



16/54

4. If manual configuration is chosen, then the following screen is displayed.

5. Enter the static IP and the network mask in the first line followed by the Gateway IP as shown inFigure 5.

Figure 4. Configure TCP/IP

Figure 5. Manual TCP/IP Configuration



17/54

6. Anaconda starts. You may see a warning screen. Select Yes because xvda is the DISKFILE that youentered at the wrapper command line.

7. Continue with the install. Once the install is complete, you should be able to run xm list to see theinstalled guests.

Figure 7 shows a host with two guests, firstVM and secondVM. These guests are assigned the domainIDs

of 18 and 19 respectively. Both the domains have 512 MB memory and are in blocked state. You can getmore info on xm by looking at the xm man pages.

Configuring CPU assignments (CPU pinning in NUMA machines)The wrapper script attempts to allocate CPUs to the guest OS from one single node. This results inmemory and CPU being assigned from the same node for a guest VM. This is much more efficient thenspreading it across nodes.

If you want to create a different assignment of nodes (CPU affinity) to your guest OSs, you can try thefollowing steps.

Figure 6. Warning screen

Figure 7. Displays xm list



18/54

In this example, change the CPU affinity of secondVM to node 1 instead of node 0:

1. Find out the current CPU affinity of each guests/domains using the guests domain ID.

virsh vcpuinfo

Figure 8 shows that out of the 64 CPUs on this machine, domain IDs 18 and 19 have affinity for first16 CPUs (designated by symbol y in the first 16 places). Domain ID 18 is running on CPU 13 anddomain ID 19 is using CPU 5. Look at node_to_cpu mapping in Figure 3 on page 5. It shows thatfirst 16 CPUs are assigned to node 0. So, the wrapper script set the affinity such that these two VMsrun on node 0 of this machine.

2. Shut down the guest domain.

xm shutdown secondVM

Figure 9 shows the command to shutdown the VM. Once the domain is shut down, xm list no longer

lists that guest.

3. Open /etc/xen/ file, it will look similar to /etc/xen/secondVM displayed below inFigure 10 on page 11.

Figure 8. Affinity for first 16 CPUs

Figure 9. Shutdown domain



19/54

4. Notice the line cpus = 0-15. This sets the CPU affinity. Currently it is set to the first 16 CPUs. Changeit to point to second node by editing this line as shown in Figure 11.

5. The CPU affinity has changed to point to node 1. Start secondVM now with new setting and check itsCPU affinity.

Figure 10. CPU affinity

Figure 11. Change to point to second node



20/54

As shown in Figure 12, the affinity has changed to CPUs in the second node.

6. Figure 13 shows that the secondVM domain is using the memory from node 1.

Notice that when the secondVM is shutdown, it releases 512MB of memory and available memory innode 1 goes up by that much amount.

Creating paravirtualized guests on non-NUMA machinesThis topic discusses creating paravirutalized guests on non-NUMA machines.

1. Login to an X-windows session, start a terminal session, and run virt-manager.

Figure 12. secondVM domain uses memory from node 1

Figure 13. Shutting down secondVM



21/54

A new virt-manager window opens.

Note: If your host system is not running X windows, you may alternatively perform these steps on aremote system which is running X windows by logging into the host machine via ssh -X.

2. Right-click the localhost entry, then select Connect, and a Domain-0 entry will be createdautomatically. Domain-0 (dom0 for short) is the operating system instance which has specialmanagement privileges and direct access to the physical hardware. dom0 is responsible for creatingthe additional domain U (or domU) operating systems instances which will act as the virtualmachines that run the users applications.

3. Select the localhost entry in virt-manager, then click New as shown in Figure 15 on page 14 to installone or more domU guest operating systems.

Figure 14. Virtual Machine Manager - new virt-manager window



22/54

4. The Creating a new virtual system wizard screen opens as shown in Figure 16 on page 15. ClickForward.

Figure 15. Install one or more domU guest operating systems



23/54

The Naming your virtual system window opens.

5. Enter a new domain name of your choice. In this example, domU-example is used. Click Forward.

Figure 16. Creating a new virtual system



24/54

6. In the Choosing a virtualization method window, select Paravirtualized or Fully Virtualizedmethods. For this example, choose Paravirtualized. Click Forward.

Figure 17. Naming your virtual system



25/54

Note: With proper hardware and drivers, it is possible to run unmodified guest operating systemsby running Xen in full virtualization mode. Full virtualization installation and configurationis, however, beyond the scope of this blueprint.

7. Specify installation media and then click Forward. The most common options would be to select thelocal CD/DVD drive if you have physical media, or to point to a network-accessible installationsource. For example:

Figure 18. Choosing a virtualization method



26/54

Note: It is recommended that you use the nfs::/ format instead ofnfs:/// format.

The Assigning storage space screen opens.

8. Specify which storage device(s) you want to allocate to your virtual machine, then click Forward. Forexample:

Figure 19. Locating installation media



27/54

9. On the Connect to host network dialog, leave the default of Virtual network selected, then clickForward.

Figure 20. Assigning storage space



28/54

10. On the Allocate memory and CPU dialog, specify the amount of memory to use at startup time, andat steady state. Also, select the number of desired virtual CPUs. Then click Forward:

Figure 21. Connect to host network dialog



29/54

11. Verify the installation setting that you specified on the Ready to begin installation dialog. ClickBack to change any settings. When you are satisfied, click Finish.

Figure 22. Allocate memory and CPU dialog



30/54

12. The virtual machine console with the installer running opens. Perform the installation as you wouldon a non-virtual machine.

Important: If you are managing your system remotely via VNC or ssh forwarding, there is a bugthat will prevent the virtual machine console from opening. To workaround thisproblem, you should either run this procedure at the machines local console, or use thealternative text-mode virt-install method.

Configuring Xen for performancePerformance tuning is not simply a series of basic steps from 1 to N resulting in a best tuned system.This topic discusses general tuning tips to determine performance bottlenecks while running in a Xenenvironment.

Remember that these are just a few basic areas that should be explored and thought out as you designand configure your system(s) for your workloads. Each workload and each combination of workloadswill have unique characteristics that will require specific tunings to achieve desired performance levels.Hopefully the areas described in this section will be a reasonable foundation to understand yourperformance and the areas you may need to change or tweak for best results.

Figure 23. Ready to begin installation



31/54

Domain0The most significant factor for tuning Xen is to ensure that Domain0 has exclusive use of sufficient CPUresource to manage the system efficiently.

Failing to dedicate exclusive CPU resources to Domain0 can result in extremely poor performance, anddepending the number of user domains, could cause the user domains to function incorrectly. Dedicating

CPU resource to Domain0 requires three basic steps.

Appending dom0_max_vpus=X to the Xen boot lineFirst, you need to append dom0_max_vcpus=X to the Xen boot line in /boot/grub/menu.lst, where Xis the number of vcpus dedicated to Domain0

In the following examples and descriptions, well assume 2 VCPUs will be dedicated to Domain0. Anexample kernel boot entry could look like this:

title Red Hat Enterprise Linux Server (2.6.18-86.el5xen)root (hd0,0)kernel /xen.gz-2.6.18-86.el5module /vmlinuz-2.6.18-86.el5xen ro root=/dev/VolGroup01/LogVol00 rhgb quietmodule /initrd-2.6.18-86.el5xen.img

After appending dom0_max_vcpus=2, it looks like this:

title Red Hat Enterprise Linux Server (2.6.18-86.el5xen)root (hd0,0)kernel /xen.gz-2.6.18-86.el5 dom0_max_vcpus=2module /vmlinuz-2.6.18-86.el5xen ro root=/dev/VolGroup01/LogVol00 rhgb quietmodule /initrd-2.6.18-86.el5xen.img

The Xen kernel needs to be restarted for this change to take affect.

Bind or pin dom0s VCPUSecond, for good performance it is critical that you bind or pin each of dom0s VCPUs to a physicalprocessor.

To do so, issue the follow commands:xm vcpu-pin Domain-0 0 0xm vcpu-pin Domain-0 1 1

The first command binds dom0s VCPU # 0 to physical processor # 0. The second command bindsdom0s VCPU # 1 to physical processor # 1. Depending on the number of VCPUs being dedicated toDomain0, more or less individual commands would need to be issued.

The amount of physical processors to dedicate to Domain0 depends on your workload(s) and the numberof user domains running. Several external studies suggest an average total CPU utilization target of ~60%is best. This target provide good utilization level of the system resources, while leaving sufficientheadroom to accommodate peaks in demand helping to maintain good Quality of Service.

For more information, see CPU utilization on page 24.

Configuring user domainsThird, each of the user domains must be configured to avoid using the physical processors dedicated toDomain0s VCPUs.

To restrict which physical processors each user domain runs on, the following line in the user domainsconfiguration file must be changed. Assuming a 16 cpu systems, the configuration file line:

cpus = "0-15"

needs to be changed to



32/54

cpus = "2-15"

In the case of a multi-chassis system, with NUMA characteristics, the user domain configuration filesshould be adjusted to bind specific sets of user domains to the physical CPUs in each system node.Assume now we have a 32 cpu system, with two nodes, each containing 16 cpus. Changes are required toadjust the cpu field in each user domain configuration file that will be bound to run on the secondsystem node, would look like this:

cpus = "16-31"

Binding user domains to specific nodes is very important for efficiency which ultimately affectsperformance. Internal testing has shown that running user domains in an unbounded fashion vs boundedcan impact performance by a factor of 2-3x or more, depending on your workload.

Fortunately, Red Hat Enterprise Linux 5.2 uses the libvirt toolkit, which automatically creates theconfiguration files that spreads and binds user domains across the nodes of your system. However, at thistime, libvirt does not reserve CPU resources for Domain0, so the three previously described steps need to

be performed manually.

Other performance factors

In addition to having to tune Xen itself, there are several other factors that can limit performance of agiven system configuration. These factors are: CPU utilization and capacity, disk and network I/Outilization, and memory.

When any of these factors nears the limit of the capabilities, performance will be limited or capped.System administrators or IT support need to monitor these factors to ensure their systems are not limitingperformance.

This section will touch on some methods to measure and monitor these factors, and help the end user todetermine if they are impacting their performance. These factors can often be related to the systemtopology or configuration. This section will not delve into the numerous ways a system configurationcould be changed to alleviate problems, since this relies heavily on the way this system fits into a usersoverall IT structure.

CPU utilizationThe first factor to evaluate is CPU utilization.

To view the CPU utilization in Xen for the entire system or for each guest, use the tool calledVirt-manager. This can be invoked via Applications->System Tools->Virtual Machine Manager. Thistool is GUI based and will visually show details about controlling or master Xen domain called dom0 aswell as any guest domain domU. Provided below are figures illustrating how to use the tool to viewCPU characteristics of each domain.

Right-click a Domain and select Details from the Virtual Machine Manager window. This example selectsDomain0.



33/54

This Details window displays the information about the total system CPU and memory utilization. forDomain0.

Figure 24. Selecting Domain0



34/54

Right-click dom1 (Domain1) and select Details.

Figure 25. Displaying details about the total system CPU and memory utilization



35/54

This Details window displays information about the total system CPU and memory utilization forDomain1.

Figure 26. Selecting Domain1



36/54

From Virt-Manager, you can determine the CPU utilization of any part of your Xen configuration. Thisprovides the utilization details to aid in balancing the resources in the Xen configuration to provideoptimal utilization and performance of the overall system as well as each user domain.

Disk I/OThe second factor is disk I/O. This can affect all workloads not just Xen configurations.

The disk subsystem must be able to handle the amount of I/O activity produced by the system.Improperly configured system can have substantial impact on the overall performance of the system.With Xen and numerous guest domains that can be supported, serious thought needs to be given to theproper disk types and configuration. For most critical environments, RAID10 array is recommended. Thisprovides mirroring for redundancy and striping to take full advantage of the concurrency across all of thedisk (spindles). RAID5 is another alternative that provides redundancy that stripes parity across alldrives. RAID5 requires fewer disks, but increases the total number of writing operations to the array due

to having to rewrite the parity stripe to each disk. RAID5 also utilizes more CPU to compute the parityfor each write I/O. Your final choice will likely depend on costs or maximum limitations of the diskinterface. Disks are available in three major interfaces, fibre, SAS, and SATA.

Determining the utilization of your disk subsystem can be obtained through a variety of tools. Once suchtool is iostat. An example usage might be: iostat x 5, which produces the following output:

Linux 2.6.11.4-20a-bigsmp (Asteroid) 05/13/2008

avg-cpu: %user %nice %sys %iowait %idle0.40 43.53 0.15 0.31 55.62

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util

Figure 27. Displaying details about the total system CPU and memory utilization



37/54

fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 42.80 42.80 0.00sda 1.15 1.16 0.07 0.72 1.79 15.01 0.90 7.51 21.41 0.01 16.30 3.97 0.31sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 28.34 0.00 88.42 30.15 0.00

There is a lot of information iostat reports for each device seen by Linux. Refer to the Linux man pagefor a detailed explanation of all the fields. From the iostat output, the read (rKB/s) and write (wKBs)transfer rates can be determined, as well as the average amount of time each I/O request is waiting to be

serviced (await) by the disk subsystem. A strong indicator of an inadequate storage subsystem is largeawait times. The lower the result the better. Bear in mind that a number of factors can affect theawait results, including high CPU utilization. This tool, however, can be a good barometer of a disksubsystems health.

Network I/OThe third factor is the network I/O. If you workload makes heavy use of networking, it could be a

bottleneck.

Again, many tools are available to collect and report network performance metrics. A good program forcollecting all types of data is sar. It reports some very useful information about network traffic, whichcan be used to determine if your network is limiting your performance. A sample out of sar is:

Linux 2.6.5-7.139-pseries64 (sqh) 04/24/05

08:34:52 IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s08:35:02 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.0008:35:02 eth0 3.40 0.40 265.13 34.37 0.00 0.00 0.0008:35:02 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.0008:35:02 eth2 0.00 0.00 0.00 0.00 0.00 0.00 0.0008:35:02 eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.0008:35:02 eth4 0.00 0.00 0.00 0.00 0.00 0.00 0.0008:35:02 eth5 0.00 0.00 0.00 0.00 0.00 0.00 0.0008:35:02 eth6 0.00 0.00 0.00 0.00 0.00 0.00 0.0008:35:02 eth7 0.00 0.00 0.00 0.00 0.00 0.00 0.00

sar will show you all your network devices and variety of metrics about each. You will need to knowthe capability of your network hardware and compare it to the results such as these to determine if yourreaching network saturation and may need to additional network adapters or higher capacity adapters.

MemoryThe fourth factor is memory. Having sufficient amounts of memory for your specific workload is crucialto achieve reasonable to optimal performance.

With insufficient memory, less of workloads working-set will fit into memory which will require theworkload to go to disk more often. This will negatively impact performance and compounds therequirements of disk subsystem which could provide a double penalty. For more information aboutmemory utilization, see CPU utilization on page 24.



38/54



39/54

Appendix A. Troubleshooting

This topic discusses troubleshooting tips.

v Virtualization option does not show up as indicated in Figure 2 on page 3.

Be sure to enter the Server + Virtualization license key during install.

v Some of the Xen packages are missing.

Check Installing Xen after RHEL 5.2 install on page 3 for a list of packages. If there are missingpackages, they can be installed by running yum install .

v My machine does not boot Xen.

Follow directions in Making Xen the default boot kernel on page 4 to verify that xen is the defaultboot kernel.

v How do I know if my machine is NUMA or booted as NUMA?

Follow directions in Creating Paravirtualized guests on NUMA machines on page 6.

v What is the preferred install location when using the NUMA wrapper or virt-manager GUI?

During testing, http install worked the best. However, nfs installs and local media were also tried.v My machine fails to boot to NFS install source

If you were using the nfs:/// format to specify your install source. Trythe nfs::/ format.

v How do I remote log into my guest machine?

If you have followed our instruction to stay with the default Virtual network option in step 9, and ifyou choose to take the default of DHCP in the Configure TCP/IP step during the guest OS install,the new guest virtual machine will be assigned a local IP address (192.168.122.x). That is, you do notneed to have a real DHCP server running to get a local IP address assigned. And if you elect to allowssh access during the firewall setup step, you can access the new Xen guest by ssh into the local IPaddress from the host machine (ssh [email protected]). To see what local IP address is assigned toyour new guest, in the virtual console of the new guest machine (from virt-manager) use the command

ifconfig to find out.

You can find more information in the Red Hat Enterprise Linux 5 Virtualization Guide .

(https://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/index.html)

https://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/index.htmlhttps://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/index.html


40/54



41/54

Appendix B. NUMA and the wrapper script

Non-Uniform Memory Access (NUMA) is a architecture where memory access time depends on thelocation of the memory relative to the processor.

A processor can access its local memory faster than memory located on another node. NUMA providesseparate memory for each processor thereby avoiding the bottleneck caused when several processors tryto access the same memory. However, it is possible for memory access to span between nodes. NUMAprovides mechanism to move data between nodes. But this type of remote memory access takes longerthan one which is confined to single node.

The Wrapper script discussed in this topic and used to allocate memory and CPUs to guest domains,takes into account this fact. It tries to allocate memory and CPUs from one single node so that the guestdomain does not have to make calls to another node which can get expensive. Confining the memoryand CPUs to one single node and keeping memory and CPUs both on the same node improvesperformance of guest domains.

You can download the wrapper code from the ibm.com site at:

Xen NUMA wrapper code .

www.ibm.com/developerworks/apps/download/index.jsp?contentid=310979&filename=bsdwrapper.zip&method=http

http://www.ibm.com/developerworks/apps/download/index.jsp?contentid=310979&filename=bsdwrapper.zip&method=httphttp://www.ibm.com/developerworks/apps/download/index.jsp?contentid=310979&filename=bsdwrapper.zip&method=http


42/54



43/54

Appendix C. Common Xen commands

Virsh and xm and some of the common Xen commands. These commands can be used to create guestdomains, shutdown guest domains. This topic discusses some of the commonly used Xen commands.

xm create [-c] configfile [name=value]

The create sub command requires a configfile and can optionally take a series of name value pairs thatadd to or override variables defined in the config file. See xmdomain.cfg for full details of that fileformat, and possible options used in either the configfile or Name=Value combinations.

Configfile can either be an absolute path to a file, or a relative path to a file located in /etc/xen.

Create will return as soon as the domain is started. This does not mean the guest OS in the domain hasactually booted, or is available for input.

OPTIONS-c Attach console to the domain as soon as it has started. This is useful for determining issues with

crashing domains.

EXAMPLES

with config file

xm create Fedora4

This creates a domain with the file /etc/xen/Fedora4, and returns as soon as it is run.

xm info

Print information about the Xen host in name: value format.

When reporting Xen bugs, this information should be submitted as part of the bug report.

Sample xen domain info looks as follows:

host : talonrelease : 2.6.12.6-xen0version : #1 Mon Nov 14 14:26:26 EST 2005machine : i686nr_cpus : 2nr_nodes : 1sockets_per_node : 2cores_per_socket : 1

threads_per_core : 1cpu_mhz : 696hw_caps : 0383fbff:00000000:00000000:00000040total_memory : 767free_memory : 37xen_major : 3xen_minor : 0xen_extra : -develxen_caps : xen-3.0-x86_32xen_pagesize : 4096platform_params : virt_start=0xfc000000xen_changeset : Mon Nov 14 18:13:38 2005 +0100

7793:090e44133d40



44/54

cc_compiler : gcc version 3.4.3 (Mandrakelinux10.2 3.4.3-7mdk)

cc_compile_by : sdaguecc_compile_domain : (none)cc_compile_date : Mon Nov 14 14:16:48 EST 2005xend_config_format : 2

xm list [domain-ids]Prints information about one or more domains. If no domains are specified it prints out informationabout all domains.

An example format for the list is as follows:

Name ID Mem(MiB) VCPUs State Time(s)Domain-0 0 98 1 r----- 5068.6Fedora3 164 128 1 r----- 7.6Fedora4 165 128 1 ------ 0.6Mandrake2006 166 128 1 -b---- 3.6Mandrake10.2 167 128 1 ------ 2.5Suse9.2 168 100 1 ------ 1.8

Name is the name of the domain. ID the domain numeric id. Mem is the size of the memory allocated tothe domain. VCPUS is the number of VCPUS allocated to domain. State is the run state. Time is the totalrun time of the domain as accounted for by Xen.

xm shutdown [options] domain-id

Gracefully shuts down a domain. This coordinates with the domain OS to perform graceful shutdown, sothere is no guarantee that it will succeed, and may take a variable length of time depending on whatservices must be shutdown in the domain. The command returns immediately after signalling the domainunless that -w flag is used.

The behavior of what happens to a domain when it reboots is set by the on_shutdown parameter of thexmdomain.cfg file when the domain was created.

OPTIONS

-a Shutdown all domains. Often used when doing a complete shutdown of a Xen system.

-w Wait for the domain to complete shutdown before returning.

virsh list

Prints information about one or more domains. If no domains are specified it prints out informationabout all domains.

Example

An example format for the list is as follows:

virsh listId Name State

----------------------------------

0 Domain-0 running2 fedora paused

Name is the name of the domain. ID the domain numeric id. State is the run state.



45/54

virsh create FILE

Create a domain from an XML . This is an easy way to create one if you have a pre-existing xenguest created via xm create .

Examplevirsh dumpxml > file

virsh destroy domain-id

Immediately terminate the domain domain-id.

This doesnt give the domain OS any chance to react, and it the equivalent of ripping the power cord outon a physical machine. In most cases you will want to use the shutdown command instead.

virsh dumpxml domain-id

Output the domain informations as an XML dump to stdout, this format can be used by the createcommand.

virsh shutdown domain-id

Gracefully shuts down a domain.

This coordinates with the domain OS to perform graceful shutdown, so there is no guarantee that it willsucceed, and may take a variable length of time depending on what services must be shutdown in thedomain.

For a xen guest vm the behavior of what happens to a domain when it reboots is set by the on_shutdownparameter of the xmdomain.cfg file when the domain was created.

Appendix C. Common Xen commands 37


46/54



47/54

Appendix D. Source code

This topic contains the source code.

#!/usr/bin/python -tt

import os, sys, stringimport libvirtimport libxml2

def get_xml_to_doc(conn):xml = conn.getCapabilities()try:

doc = libxml2.parseDoc(xml)except Exception,e:

sys.exit(e)return doc

#check if there are active cpus associated with the given celldef get_numcpus(id):

cpulist = []ctx = capabilities_doc.xpathNewContext()try:

cpu_count = int(ctx.xpathEval("number(/capabilities/topology/cells/cell[@id = %s]/cpus/@num)"% (id)))

except Exception,e:sys.exit(e)

return cpu_count

def cell_to_cpus(conn,id):cpu_list = []ctx = capabilities_doc.xpathNewContext()cpus = ctx.xpathEval("/capabilities/topology/cells/cell[@id = %s]/cpus/cpu" % (id))for cpu in cpus:

try:

cpu_id = int(cpu.xpathEval("number(@id)"))except Exception,e:sys.exit(e)

cpu_list.append(cpu_id)cpu_list.sort()return cpu_list

def parse_memory():mem = 0args = sys.argvfor i in args:

try:(arg, val) = i.split("=")if arg == "--ram":

try:mem = int(val)

except:break

except:if i == "-r":

try:mem = int(args[args.index("-r") + 1])

except:break

continuemem = mem * 1024 * 1024return mem



48/54

# return a list of (cell, mem_size) pairs, ordered from smallest to largest# mem_size that satisfy minimum memory requirement. Ignore cells that have no# cpus associated with them.def find_cells(mem):

cell_list = []info = conn.getInfo()nr_cells = info[4]

if nr_cells == 1:# not NUMAreturn cell_list

cell_mem = conn.getCellsFreeMemory(0, nr_cells)for i in range(nr_cells):

if cell_mem[i] > mem:nr_cpus=get_numcpus(i)if nr_cpus != 0:

cell_info = (i, cell_mem[i])cell_list.append(cell_info)

cell_list.sort(cmp=lambda x,y: cmp(x[1], y[1]))return cell_list

def apply_policy(cell_list, policy):if policy == "min":

cell_choice = cell_list[0][0]return cell_choice

if policy == "max":cell_choice = cell_list[-1][0]return cell_choice

def create_cpustring(list):cpu_string="--cpuset="for i in list:

cpu_string = cpu_string + str(i) + ","cpu_string = cpu_string[:-1]return cpu_string

# begin main code

global capabilities_docconn = libvirt.open("xen")guest_ram = parse_memory()capabilities_doc=get_xml_to_doc(conn)if guest_ram == 0:

sys.exit("Error: Guest memory size not specified")cells = find_cells(guest_ram)if len(cells) == 0:

#not NUMA so no cpuset to createargs = ["virt-install"] + sys.argv[1:]os.execv("/usr/sbin/virt-install", args)

cell_choice = apply_policy(cells, "min")cpu_list = cell_to_cpus(conn, cell_choice)cpu_set = create_cpustring(cpu_list)final_args = ["virt-install", cpu_set] + sys.argv[1:]

os.execv("/usr/sbin/virt-install", final_args)



49/54

Appendix E. Related information for Xen

Listed here are Web sites that relate to Xen, virtualization, and Red Hat Linux.

v Red Hat Enterprise Linux 5 Virtualization Guide(https://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/index.html)

v Red Hat Enterprise Linux 5.1 Virtualization Guide

(https://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/en-US/RHEL510/Virtualization_Guide/index.html)

v Virtual Machine Manager

(http://virt-manager.et.redhat.com/)

v libvirt: The virtualization API

(http://www.libvirt.org/)

v The Xen virtual machine monitor

(http://www.cl.cam.ac.uk/research/srg/netos/xen/index.html)

v redhat.com/

(https://www.redhat.com/)

v ibm.com/linux

(https://www.ibm.com/linux)

v IBM Linux Xen Virtualization Blueprint Support Forum

(http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1272)

v IBM BladeCenter and System x Reference Sheets

(http://www.redbooks.ibm.com/Redbooks.nsf/pages/xref)

v Red Hat Enterprise Linux 5.2 Release Notes

(http://www.redhat.com/docs/manuals/enterprise/)

v Xen NUMA wrapper code

www.ibm.com/developerworks/apps/download/index.jsp?contentid=310979&filename=bsdwrapper.zip&method=http

https://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/index.htmlhttps://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/en-US/RHEL510/Virtualization_Guide/index.htmlhttp://virt-manager.et.redhat.com/http://www.libvirt.org/http://www.cl.cam.ac.uk/research/srg/netos/xen/index.htmlhttps://www.redhat.com/http://www.ibm.com/linuxhttp://www.ibm.com/developerworks/forums/forum.jspa?forumID=1272http://www.redbooks.ibm.com/Redbooks.nsf/pages/xrefhttp://www.redhat.com/docs/manuals/enterprise/http://www.ibm.com/developerworks/apps/download/index.jsp?contentid=310979&filename=bsdwrapper.zip&method=httphttp://www.ibm.com/developerworks/apps/download/index.jsp?contentid=310979&filename=bsdwrapper.zip&method=httphttp://www.redhat.com/docs/manuals/enterprise/http://www.redbooks.ibm.com/Redbooks.nsf/pages/xrefhttp://www.ibm.com/developerworks/forums/forum.jspa?forumID=1272http://www.ibm.com/linuxhttps://www.redhat.com/http://www.cl.cam.ac.uk/research/srg/netos/xen/index.htmlhttp://www.libvirt.org/http://virt-manager.et.redhat.com/https://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/en-US/RHEL510/Virtualization_Guide/index.htmlhttps://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/index.html


50/54



51/54

Appendix F. Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries.Consult your local IBM representative for information on the products and services currently available inyour area. Any reference to an IBM product, program, or service is not intended to state or imply thatonly that IBM product, program, or service may be used. Any functionally equivalent product, program,or service that does not infringe any IBM intellectual property right may be used instead. However, it isthe users responsibility to evaluate and verify the operation of any non-IBM product, program, orservice.

IBM may have patents or pending patent applications covering subject matter described in thisdocument. The furnishing of this document does not give you any license to these patents. You can sendlicense inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where suchprovisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATIONPROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS ORIMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OFNON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Somestates do not allow disclaimer of express or implied warranties in certain transactions, therefore, thisstatement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodicallymade to the information herein; these changes will be incorporated in new editions of the publication.IBM may make improvements and/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Licensees of this program who wish to have information about it for the purpose of enabling: (i) theexchange of information between independently created programs and other programs (including thisone) and (ii) the mutual use of the information which has been exchanged, should contact:

IBM CorporationDept. LRAS/Bldg. 00311400 Burnet Road

Austin, TX 78758-3498U.S.A.

Such information may be available, subject to appropriate terms and conditions, including in some cases,payment of a fee.

The licensed program described in this document and all licensed material available for it are providedby IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement orany equivalent agreement between us.



52/54

For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual PropertyDepartment in your country or send inquiries, in writing, to:

IBM World Trade Asia CorporationLicensing2-31 Roppongi 3-chome, Minato-kuTokyo 106-0032, Japan

IBM may use or distribute any of the information you supply in any way it believes appropriate withoutincurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, theirpublished announcements or other publicly available sources. IBM has not tested those products andcannot confirm the accuracy of performance, compatibility or any other claims related to non-IBMproducts. Questions on the capabilities of non-IBM products should be addressed to the suppliers ofthose products.

Any references in this information to non-IBM Web sites are provided for convenience only and do not inany manner serve as an endorsement of those Web sites. The materials at those Web sites are not part ofthe materials for this IBM product and use of those Web sites is at your own risk.

This information contains examples of data and reports used in daily business operations. To illustratethem as completely as possible, the examples include the names of individuals, companies, brands, andproducts. All of these names are fictitious and any similarity to the names and addresses used by anactual business enterprise is entirely coincidental.

Trademarks

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International BusinessMachines Corporation in the United States, other countries, or both. If these and other IBM trademarkedterms are marked on their first occurrence in this information with a trademark symbol (&regtm; and&tm;), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this

information was published. Such trademarks may also be registered or common law trademarks in othercountries. A current list of IBM trademarks is available on the Web at Copyright and trademarkinformation at www.ibm.com/legal/copytrade.shtml

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarksof Adobe Systems Incorporated in the United States, and/or other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.



53/54


54/54

Printed in USA

Documents

Installing and Configuring Xen