Heterogeneous Live Migration of Virtual Machines Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou,...

Heterogeneous Live Migration of Virtual Machines

Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou, Haibo Chen, and Binyu Zang

Parallel Processing Institute, Fudan Universityhttp://ppi.fudan.edu.cn

The market of Virtualization is booming

• Virtual data-center, grid computing, and cloud computing

• “By 2010, at least 30%of non-desktop IT infrastructure will be virtualized (up from less than 5 percent in 2007).”– www.cio.com/article/170653

• What can virtualization do?– Sever consolidation– Mobility

• Virtual machine live migration– …

VM Live Migration

• Benefits– Load balancing– Online maintenance – Fault tolerance

23/4/18 PPI, Fudan University 3

VirtualMachine

Virtual Machine

VMMHardware Hardware

VMM

War between different VMM venders

• The battle in virtualization area is and will be continuing – Vmware, Microsoft, kvm, …

– They will likely co-exisit in a long time• Linux kernel 2.6.23 embeds different vmm support

– Xen, lguest, kvm

• Cloud Computing• More resources brings inevitably heterogeneity

Virtual Machine

VirtualMachine

Motivation

• Heterogeneity of different VMMs– Break uniformity and mobility– Downgrade the applicability of live migration

23/4/18 PPI, Fudan University 5

VMM 2VMM 1Hardware Hardware

Contribution: Vagrant

• A preliminary study aim at– Bridging the heterogeneity of different VMMs and

support live migration

• Goal– Provide common migration protocol– Use common format of VM abstraction– Provide a pool of memory migration algorithms

• Preliminary results– Support VM migration across Xen and KVM– Accept able performance

Outline

• Heterogeneity among VMMs

• Vagrant Design and Implementation

• Evaluation

• Conclusion

23/4/18 PPI, Fudan University 7

Heterogeneity in Abstracting Resources

• CPU– Internal virtual CPU format

• Memory– Memory management information

• I/O Devices– Virtual I/O Device format

23/4/18 PPI, Fudan University 8

Heterogeneity in Migration Algorithms

• Stop-and-copy

• Pre-copy

• Push and pull

23/4/18 PPI, Fudan University 9

Heterogeneity in Migration Protocols

• Different set of control commands

• Example – Xen VMM initiates migration by sending a string

”receive” to the target– KVM sends a special integer to start migration

23/4/18 PPI, Fudan University 10

Outline

• Heterogeneity among VMMs

• Vagrant Design and Implementation

• Evaluation

• Conclusion

23/4/18 PPI, Fudan University 11

Architecture of Vagrant

• Vagrant acts as a middle man between different VMMs

23/4/18 PPI, Fudan University 12

VMM A

VM

Management VM

VMM B

Vagrant Agency

StateTransformer

Algorithm pool

Management VM

Vagrant Agency

StateTransformer

Algorithm pool

Bridging the Heterogeneity

• CPU State– Intercept the state from local virtual CPU module

– Essential CPU state is transformed into Vagrant format

– Requirements :• VMMs can expose same CPU features

23/4/18 PPI, Fudan University 13

Bridging the Heterogeneity (2/2)

• I/O Devices– Intercept the state from local device module• H/W based virtualization relies on device emulation

– Eg. QEMU used by Xen, KVM and VirtualBox

– Requirements:• VMMs can support all I/O devices existing in the

migrated VM

23/4/18 PPI, Fudan University 14

Migration Protocol

• Initiate and Control the migration process– Common migration protocol and commands– Negotiation before the migration• Ensure the same capability of pre- and post- migration

environment– CPU features, supported I/O devices

23/4/18 PPI, Fudan University 15

VMM A VMM B

Hey,I want a RTL8169 NIC,

Can you support?

Migration Algorithms

• Each VMM has its own migration algorithm

23/4/18 PPI, Fudan University 16

Stop and Copy

23/4/18 PPI, Fudan University 17

VirtualMachine

Virtual Machine

VMMHardware Hardware

VMM

Simply stop the VM and send

VM state

May cause very long

downtime, but total

time is short

Pre-copy Based Algorithm

• Sending memory pages while the VM is still running

23/4/18 PPI, Fudan University 18

VirtualMachine

Virtual Machine

VMMHardware Hardware

VMM

Sending memory pages

VM is still running

Stop the VM

Send whole VM state to destination

Pre-copy Reduces

downtime

Algorithm Pool

• Algorithm pool– At least one common algorithm is needed– Choose the algorithm by negotiation

• Ideally, support a set of different algorithms– Choose appropriate algorithm in different

scenarios– eg, the user wants to finish the migration as soon

as possible, stop and copy is better

23/4/18 PPI, Fudan University 19

Implementation

• A prototype based on– X86 with VT-x extensions– Xen 3.1.0 and KVM-48

– CPU• Both expose most CPU features

– I/O Device• Both rely on QEMU• VMM support for some devices, such as PIC/APIC/IOAPIC

23/4/18 PPI, Fudan University 20

Outline• Heterogeneity among VMMs

• Vagrant Design and Implementation

• Evaluation– Migration time– Application performance

• Conclusion23/4/18 PPI, Fudan University 21

Experimental Setup(1/2)

• Machine setup:– Migration between pair of PCs with

• 2.33 GHz Intel Core Duo CPU, 2GB RAM• RTL8169 NIC• 250 GB 7200 RPM SATA disk

– NFS server• 3.0 GHz Pentium IV with 1GB RAM• Intel Pro 100/1000 Ethernet NIC• 250 GB 7200 RPM SATA disk

– Machines were connected via switched Gigabit Ethernet

23/4/18 PPI, Fudan University 22

Experimental Setup (2/2)

• VM configuration– 128 MB of RAM– 5GB disk image with installation of Fedora Core 6

• Migration time under different workload– Idle, kernel-compile, memtest86, apache benchmark (ab)

• Application performance– Migrating a running Web Server

• Continuously serves a 256 KB file to a remote client

23/4/18 PPI, Fudan University 23

Questions for Evaluation

• Whether Vagrant has an acceptable total migration and minimal downtime?

• Whether applications during the migration can still provide acceptable performance?

• Is Vagrant’s performance comparable to other migration framework, eg. Xen and KVM?

23/4/18 PPI, Fudan University 24

Total Migration Time

23/4/18 PPI, Fudan University 25

Vagrant adopts batched memory transfer while KVM uses one-by-one manner

Migration Downtime

23/4/18 PPI, Fudan University 26

Downtime in Vagrant in comparable to homogeneous live migration

Migrating a Running Web Server

23/4/18 PPI, Fudan University 27

Memory Pre-copy, about 10% downgrade

Less than 1 second

Related Work

• Homogenous live migration over LAN– Vmotion, Xen (Clark et al. ’05) and KVM

• Live wide-area migration– Bradford et al. (vee’07)– “VM Turntable” (FGCS’06)

• Tons of virtualization related work

23/4/18 PPI, Fudan University 28

Conclusion

• Heterogeneous live VM migration framework– General abstraction of VMs and migration protocol

– No changes to hosted OS and application

– Minor changes to VMM abstraction

– Comparable downtime and end-to-end time

23/4/18 PPI, Fudan University 29

Questions?

• Thank you!

3023/4/18 PPI, Fudan University

Heterogeneous Live Migration of Virtual Machines

Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou, Haibo Chen, and Binyu Zang

Parallel Processing Institute, Fudan Universityhttp://ppi.fudan.edu.cn

Backup Slides

• Control commands in Vagrant

• CPU state in Vagrant

• I/O devices configured for the VM

• Memory migrating algorithm

23/4/18 PPI, Fudan University 32

Control Commands in Vagrant

• Common set of control commands– Each has a 3 seconds timeout

23/4/18 PPI, Fudan University 33

CPU State in Vagrant

• The necessary information to be transferred includes– GPR, control regs, segment regs, debug regs, msr,

fpu regs, sysenter registers– Time stamp counter– Pending events

23/4/18 PPI, Fudan University 34

I/O devices configured for the VM

• The migrated VM is configured with– Keyboard and mouse– Rtc– PIC/APIC/IOAPIC– PIT– IDE disk– NIC– Serial port– VGA

23/4/18 PPI, Fudan University 35

Memory migrating algorithm

• Algorithm implemented by Vagrant– Pre-copy in batched manner• Batch size is 1024

23/4/18 PPI, Fudan University 36