Control hypervisor via libvirt

[email protected] P1

Control Hypervisor via Libvirt

Sean Chang

2011/4


大綱

• 為什麼要用 libvirt

• Lifecycle of VM

• 順帶一提 virsh

• 順帶一提 xm

• 進入重點 libvirt

• Libvirt 使用範例

• Libvirt 使用經驗

• Libvirt 議題

• DEMO


管理 hypervisor 的方法

• 下指令 o xm

o virsh

• 自己寫程式 o Libvirt

o A virtualization API

o Open source

o 可程式化


飲水思源

了解VM(Guest domain)的生命週期


Guest domain status

State Description

Undefine

d

This is the baseline state. An undefined guest domain has not been

defined or created in any way.

Defined A defined guest domain has been defined but is not running. This

state could also be described as Stopped.

Running A running guest domain is defined and being executed on a hypervisor.

Paused A paused guest domain is in a suspended state from the Running

state. Its memory image has been temporarily stored, and it can be

resumed to the Running state without the guest domain operating

system being aware it was ever suspended.

Saved A saved domain has had its memory image, as captured in the

Paused state, saved to persistent storage. It can be restored to the

Running state without the guest domain operating system being

aware it was ever suspended.


Guest domain lifecycle


VM(Guest domain) status


VM Lifecycle

What does

Libvirt actully

do？


工欲善其事

必先利其器


xm

• Xen management user

interface

• The xm program is the main

interface for managing Xen

guest domains. The program

can be used to create, pause,

and shutdown domains. It can

also be used to list current

domains, enable or pin

VCPUs, and attach or detach

virtual block devices.

• Example

o xm create Fedora12

o xm start Fedora12

o xm help

o xm list

o xm mem-set {domain-id,

mem}

o xm migrate {domain-id,

host-ip}

o etc…沒事多玩指令，多玩指令沒事


virsh(1/2)

• virsh 是個用來管理客座端和

hypervisor（管理程序）的指令

列介面工具。

• 不過說穿了，這個指令列也是用 libvirt 作為底層函式庫寫出來的。

指令 Description

help 印出基本協助資訊。

list 列出所有客座端。

dumpxml 輸出客座端的 XML 配置檔案。

create 由一個 XML 配置檔案來建立客座端並啟用新

的客座端。

start 啟用一個未啟用中的客座端。

destroy 強制客座端停下。

define 輸出客座端的 XML 配置檔案。

domid 顯示客座端的 ID。

domuuid 顯示客座端的 UUID。

dominfo 顯示客座端的相關資訊。

domname 顯示客座端的名稱。

domstate 顯示客座端的狀態。

quit 退出互動式終端機。

reboot 重新啟動客座端。

restore 恢復之前已儲存於一個檔案中的客座端。

resume 恢復一個暫停中的客座端。

save 將客座端目前的狀態儲存至一個檔案中。

shutdown 正常地停用客座端。

suspend 將客座端暫停。

undefine 刪除與某個客座端有關聯的所有檔案。

migrate 將客座端遷移至另一部主機上。


virsh(2/2)

• Example o # virsh connect {hostname

OR URL}

o # virsh create

configuration_file.xml

o # virsh suspend {domain-

id, domain-name or

domain-uuid}

o etc…沒事多玩指令，多玩指令沒事

指令 Description

setmem 為客座端設置分配記憶體。

setmaxmem 設定 hypervisor 所能使用的記憶體最大限制。

setvcpus 更改分配給客座端的虛擬 CPU 數量。

vcpuinfo 顯示有關於客座端的虛擬 CPU 資訊。

vcpupin 控制客座端的虛擬 CPU 相似性。

domblkstat 顯示正在執行中的客座端的區塊裝置數據。

domifstat 顯示執行中的客座端的網路介面卡數據。

attach-device 透過使用某個 XML 檔案中的裝置定義來將裝置附加至客座端上。

attach-disk 附加新磁碟裝置至客座端。

attach-interface 附加新網路介面卡至客座端。

detach-device 將裝置從客座端上分離，接受與 attach-device 指令相同類型的

XML 描述。

detach-disk 將磁碟裝置由客座端上移除。

detach-interface 將網路介面卡由客座端上移除。


不是不寫程式

不過時後也已經到了


Libvirt(http://libvirt.org/index.html)

• A toolkit to interact with the virtualization

capabilities of recent versions of Linux (and

other OSes), see our project goals for details.

• Free software available under the GNU Lesser

General Public License.(LGPL)

• A long term stable C API

• A set of bindings for common languages.(Java、Python…etc)

• Latest release: 0.8.8: Feb 17 2011

http://libvirt.org/goals.html

http://www.opensource.org/licenses/lgpl-license.html

http://www.opensource.org/licenses/lgpl-license.html


Terminology and goals

Xen

…

一般稱呼 Libvirt的術語解釋

VM Domain An instance of an operating system (or

subsystem in the case of container

virtualization) running on a virtualized machine

provided by the hypervisor

Xen Hypervisor A layer of software allowing to virtualize a

node in a set of virtual machines with possibly

different configurations than the node itself

Physical

machine

Node A single physical machine


Libvirt drivers

• Hypervisor drivers o LXC - Linux Containers

o OpenVZ

o QEMU

o Test - Used for testing

o UML - User Mode Linux

o VirtualBox

o VMware ESX

o VMware

Workstation/Player

o Xen

o 支援Multi-Vendor(表面上…)

• Storage drivers o Directory backend

o Local filesystem backend

o Network filesystem backend

o Logical Volume Manager

(LVM) backend

o Disk backend

o iSCSI backend

o SCSI backend

o Multipath backend

http://libvirt.org/drvlxc.html

http://libvirt.org/drvopenvz.html

http://libvirt.org/drvqemu.html

http://libvirt.org/drvtest.html

http://libvirt.org/drvuml.html

http://libvirt.org/drvvbox.html

http://libvirt.org/drvesx.html

http://libvirt.org/drvvmware.html

http://libvirt.org/drvvmware.html

http://libvirt.org/drvxen.html

http://libvirt.org/storage.html










API concepts(1/2)

Objects exposed


API concepts(2/2)

Libvirt drivers

就是OS

或是應用程式

API呼叫

deamon

remote local


Xen config V.S. XML

• VM的設定檔 o For Xen

o For libvirt


細述 XML unique

全虛擬

最大的實體資源分配

網路採bridge方式對外連通

這張虛擬網卡

掛在node的xenbr0

VNC port


Python example – 基本VM操作


Python example – 掛載/卸載磁碟已經準備好

掛載的image空間


我不入地獄誰…

往下挖


認證

• 呼叫遠端的libvirtd需要先經過認證，認證方式如下

• ssh o ssh username root: xen+ssh://[email protected]/

o 然後輸入帳號密碼，缺點，如果程式要全自動化會被卡在這裡。

• authorized key o 製作Public / private key

o #ssh-keygen -t rsa

o 放在相對應的目錄: /root/.ssh/


Debug / loggin

• Log message

o Generated at runtime by the libvirt code

o Timestamp, a category, a priority level, function name and line number indicating

where the message originated from, and a formatted message.

• Priority levels

o 1 (or debug) - log all messages

o 2 (or info) - log all non-debugging information

o 3 (or warn) - log only warnings and errors - this is the default

o 4 (or error) - log only errors

• Filter

o x:name

o 1:qemu

o 1:qemu 4:remote

• Log output

o 3:syslog:libvirtd 1:file:/tmp/libvirt.log


虛擬裝置也支援熱插拔嗎?

• 半虛擬技術支援在不開機的情況下，CPU、MEM、DISK

都可以做調整。

• 全虛擬CPU、MEM要調整一定得重開機。

• 全虛擬DISK熱插拔則視VM的kernel版本而定。

• The Xen device model is more or less unchanged in the pv-ops

kernel. Converting a driver from the xen-unstable or 2.6.18-xen tree

should mostly be a matter of getting it to compile. There have been

changes in the Linux device model between 2.6.18 and 2.6.26, so

converting a driver will mostly be a matter of forward-porting to the

new kernel, rather than any Xen specific issues.

• Kernel version > 2.6.26 可能比較穩定有機會實現全虛擬的硬碟熱插拔。


Xen disk kinds and supporting

• Disk type

o IDE

o Virtual disk

o iSCSI

o USB Disk

• 實測熱插拔

xen 4.0 + 2.6.32.11 / Dell

Storage

IDE Virtual SCSI USB Disk

centos5.4final Kernel 2.6.18

× ○ × ×

RH6beta Kernel 2.6.18

× × × ×


API Issues

• create 跟 define 有什麼不同? o 都是用來創建VM的API

o 差別在生命週期不完全相同

o create: 創建VM之外，順便把VM開起來，shutdown後該VM自動從hypervisor卸除。

o define: 僅創建VM，須靠start API把VM開起來，但永遠定義在hypervisor裡。甚至連migrate後也在。

• shutdown 跟 destroy 有什麼不同? o 都是要把VM的狀態從running改成stop

o Shutdown：就是一般作業系統的關機。

o Destroy：以實體主機來看，就是直接把插頭拔掉。


Development trick(1/3) - Problem

• Libvirt 在偵測 xen 上的VM實際狀態不穩定

• 連續下五十次偵測VM status的結果 [2, 665600L, 444416L, 1, 178818312669L][2, 665600L, 444416L, 1, 178818786889L][2, 665600L, 444416L, 1, 178819270686L][2,

665600L, 444416L, 1, 178819782827L][2, 665600L, 444416L, 1, 178820286716L][0, 665600L, 444416L, 1, 178825799216L][2,

665600L, 444416L, 1, 178826484955L][2, 665600L, 444416L, 1, 178827054507L][2, 665600L, 444416L, 1, 178827646602L][2,

665600L, 444416L, 1, 178828162295L][2, 665600L, 444416L, 1, 178828718993L][2, 665600L, 444416L, 1, 178830313690L][2,

665600L, 444416L, 1, 178831831052L][2, 665600L, 444416L, 1, 178832305724L][2, 665600L, 444416L, 1, 178832859023L][2,

665600L, 444416L, 1, 178833339605L][2, 665600L, 444416L, 1, 178833848167L][2, 665600L, 444416L, 1, 178834372305L][2,

665600L, 444416L, 1, 178834895557L][2, 665600L, 444416L, 1, 178835386865L][2, 665600L, 444416L, 1, 178835922103L][2,

665600L, 444416L, 1, 178836494779L][2, 665600L, 444416L, 1, 178836973848L][2, 665600L, 444416L, 1, 178837452908L][2,

665600L, 444416L, 1, 178838008290L][2, 665600L, 444416L, 1, 178838553784L][2, 665600L, 444416L, 1, 178839024984L][2,

665600L, 444416L, 1, 178839462699L][2, 665600L, 444416L, 1, 178839987321L][2, 665600L, 444416L, 1, 178840510368L][2,

665600L, 444416L, 1, 178841185524L][2, 665600L, 444416L, 1, 178841873368L][2, 665600L, 444416L, 1, 178842393659L][2,

665600L, 444416L, 1, 178842910878L][2, 665600L, 444416L, 1, 178843454749L][2, 665600L, 444416L, 1, 178843947320L][2,

665600L, 444416L, 1, 178844427452L][2, 665600L, 444416L, 1, 178844942837L][2, 665600L, 444416L, 1, 178845472009L][2,

665600L, 444416L, 1, 178846036556L][2, 665600L, 444416L, 1, 178846502921L][2, 665600L, 444416L, 1, 178847082404L][2,

665600L, 444416L, 1, 178847612475L][2, 665600L, 444416L, 1, 178848158639L][2, 665600L, 444416L, 1, 178848527989L][2,

665600L, 444416L, 1, 178849177317L][2, 665600L, 444416L, 1, 178849704749L][2, 665600L, 444416L, 1, 178850274864L][2,

665600L, 444416L, 1, 178850717630L][2, 665600L, 444416L, 1, 178851307315L][2, 665600L, 444416L, 1, 178852079044L][2,

665600L, 444416L, 1, 178852861964L][2, 665600L, 444416L, 1, 178854056260L][2, 665600L, 444416L, 1, 178854907434L][2,

665600L, 444416L, 1, 178856457345L][2, 665600L, 444416L, 1, 178856918293L][2, 665600L, 444416L, 1, 178857394787L][1,

665600L, 444416L, 1, 178857716163L][2, 665600L, 444416L, 1, 178858177594L][2, 665600L, 444416L, 1, 178858773954L][2,

665600L, 444416L, 1, 178859309076L][2, 665600L, 444416L, 1, 178859897892L][2, 665600L, 444416L, 1, 178860400666L][2,

665600L, 444416L, 1, 178860937146L][2, 665600L, 444416L, 1, 178861383848L][1, 665600L, 444416L, 1, 178861889219L][2,

665600L, 444416L, 1, 178862333614L][2, 665600L, 444416L, 1, 178862848250L][2, 665600L, 444416L, 1, 178863325489L][2,

665600L, 444416L, 1, 178863674106L][2, 665600L, 444416L, 1, 178864153743L][2, 665600L, 444416L, 1, 178864647934L][2,

665600L, 444416L, 1, 178864970657L][2, 665600L, 444416L, 1, 178865558487L][2, 665600L, 444416L, 1, 178865890459L][2,

665600L, 444416L, 1, 178866309383L][2, 665600L, 444416L, 1, 178866793058L][2, 665600L, 444416L, 1, 178867306348L][2,

665600L, 444416L, 1, 178867872870L][2, 665600L, 444416L, 1, 178868439241L][2, 665600L, 444416L, 1, 178868849504L][2,

665600L, 444416L, 1, 178869398668L][2, 665600L, 444416L, 1, 178869830930L][1, 665600L, 444416L, 1, 178870348265L][2,

665600L, 444416L, 1, 178870855305L][2, 665600L, 444416L, 1, 178894806221L][2, 665600L, 444416L, 1, 178895432020L][2,

665600L, 444416L, 1, 178896569901L][2, 665600L, 444416L, 1, 178898049374L][1, 665600L, 444416L, 1, 178899215693L][2,

665600L, 444416L, 1, 178900047112L]


Development trick(2/3) - Solution

• 如果狀態不穩，我們無法取得VM確實的狀態。

• 參考 VirtManager 的解法，利用以下API的特性

o 找出可以幫忙的 libvirt API。

o listDomainsID(): 列出所有xen上在跑的domain。

o 因為只有running的VM狀態不穩，所以利用listDomianID()這個API把這些可能不穩的VM先抓出來，在把0,1,2等狀態都歸納為runnung。


Development trick(3/3) - Code

31

1

2

3

4

5


libvirt 處理 Xen 與 KVM 的行為

• 不同的 hypervisor，libvirt的行為也會有所差異

• 以下以Xen跟KVM為例：


開發心得

• 多用python的特有的API查詢指令help()、dir()來幫助自己掌握libvirt的版況

• 不要太期待libvirt可以幫你做太多很強大的事情，官網很多只定義好介面，尚未實作。舉例:

• 要熟悉XML config

• 要熟悉hypervisor的行為


參考資料

• http://docs.fedoraproject.org/zh-

TW/Fedora/12/html/Virtualization_Guide/chap-Virtualization_Guide-

Managing_guests_with_virsh.html

• http://linux.die.net/man/1/xm

• http://libvirt.org/index.html

• http://libvirt.org/sources/virshcmdref/html/

• http://wiki.xensource.com/xenwiki/XenParavirtOps

http://docs.fedoraproject.org/zh-TW/Fedora/12/html/Virtualization_Guide/chap-Virtualization_Guide-Managing_guests_with_virsh.html







http://linux.die.net/man/1/xm

http://libvirt.org/index.html

http://libvirt.org/sources/virshcmdref/html/


DEMO

• 準備 VM o define

o create

• 操作 VM o start

o suspend

o resume

o reboot

o shutdown

• 添加/卸載設備 o attach

o detach

• 取得 VM o lookupByXXX

• 跨主機(下回分曉…) o virtManager 介紹

o migration

o live migration


Thank you

• 末記

o 這份資料的技術實作於2010~2011，如非最新資料，敬請見諒

• End

o The technology information was implemented in 2010 ~

2011, sorry for no update on time.

Technology

Control hypervisor via libvirt