View
2
Download
0
Category
Preview:
Citation preview
SECURITY IN LIVE VIRTUAL MACHINE MIGRATION
A Thesis by
Shah Payal Hemchand
B.E., Government College of Engineering, North Maharashtra University, Jalgaon, 2008
Submitted to Department of Electrical and Computer Engineering
and faculty of the Graduate School of
Wichita State University
in partial fulfillment of
the requirements for the degree of
Master of Science
December 2011
ii
© Copyright 2011 by Shah Payal Hemchand
All Rights Reserved
iii
SECURITY IN LIVE VIRTUAL MACHINE MIGRATION
The following faculty members have examined the final copy of this thesis for form and content,
and recommend that it be accepted in partial fulfillment of the requirement for the degree of
Master of Science with a major in Computer Networking.
_________________________________
Ravi Pendse, Committee Chair
_________________________________
Abu Asaduzzaman, Committee Member
_________________________________
Linda Kliment, Committee Member
iv
DEDICATION
To the Almighty, my family, for their continuing support and patience; to my WSU friends
for their significant advice and time throughout the completion of my thesis.
v
ACKNOWLEDGEMENTS
I sincerely thank my thesis advisor, Dr. Ravindra Pendse for his devoted motivation and
supervision throughout my career at Wichita State University. His guidance helped me complete
my thesis successfully. By working as a Graduate Research Assistant for him I gained
knowledge, and professional work ethics.
I take this opportunity to thank Amarnath Jasti for his constant support and guidance
throughout my thesis. His suggestion and advice helped me understand the technology and gain
more knowledge. His opinion towards my academic and career were valuable. I would like to
thank members of the committee for their effort and time.
I would like to extend my gratitude towards to Yonatan Assefa and all those who directly
or indirectly helped motivate me with my research.
vi
ABSTRACT
Virtualization has become an essential technology for organizations. With on-demand
services from vendors, a substantial rise in the use of virtualization has been noticed. Today there
are various kinds of virtualization techniques offering different advantages. One of the important
features of virtualization is live virtual machine (VM) migration. In live VM migration, the
controls of a VM are migrated from one physical host to another. Workload balancing, and
server maintenance becomes easy by migrating the VM. The ability to reboot or shut down the
physical server without affecting running applications is greatly beneficial to an organization.
With this indispensable feature of live VM migration, the security factor is still
unanswered. Very little research has been done in exploring the security concerns inherent while
data moves between the two physical machines. This thesis looks at this poorly explored area
and attempts to propose a proper solution, and thereby maintain security. Man–in–the–middle
attack could be created by sniffing data between the hypervisors and confidentiality is lost. Data
in transit could be read and then tampered with or misused, and can create havoc in the network
and bring it down completely.
The research shows how a malicious attacker can sniff the data while performing live
VM migration over Xen hypervisor and exploit information. Using this experiment the author
proposes strategies that can be used to have a secure live migration process.
.
vii
TABLE OF CONTENTS
Chapter Page
1. INTRODUCTION 1
1.1 Overview 1
1.2 Storage Area Network 2
1.3 Security 4
1.4 Problem Identification 6
1.5 Organization of Thesis 6
2. VIRTUALIZATION 7
2.1 Definition 7
2.2 Origin of Virtualization 7
2.3 Need of Virtualization 8
2.4 Types of Virtualization 9
3. LITERATURE SURVEY 13
3.1 Security in Virtualization 13
3.2 Related Work 15
4. XEN OVERVIEW 19
4.1 Xen Hypervisor 19
4.1.1 Why Xen? 20
4.1.2 Other Features of Xen 20
4.1.3 Working of Xen 21
4.2 Requirements for Live VM Migration 22
viii
TABLE OF CONTENTS (cont…)
Chapter Page
5. TEST BED AND RESULTS 24
5.1 Hardware 24
5.2 Software
5.2.1openSUSE 24
5.3 Test Bed 24
5.4 Results 26
5.5 Proposed Solution 29
5.5.1 IPSec Tunnel 29
5.5.2 Hypervisor Encryption 31
6. FUTURE WORK AND CONCLUSION 32
6.1 Future Work 32
6.2 Conclusion 32
REFERENCES 34
ix
LIST OF FIGURES
Figures Page
1 Basic SAN Environment 3
2 Key elements of Data Center 4
3 OS Virtualization 10
4 Paravirtualization 11
5 Full Virtualization 12
6 Test Bed 25
7 Three-way handshake between host A and host B 26
8 Three-way handshake capture 27
9 Wireshark capture showing ascii values 28
10 Live VM migration through IPSec tunnel 30
11 Encrypted data after VM migration through IPSec tunnel 30
x
LIST OF ABBREVIATIONS
AoE ATA over Ethernet
DoS Denial – of – Service
IDS Intrusion Detection System
IPS Intrusion Prevention System
iSCSI Internet Small Computer System Interface
NFS Network File System
OS Operating System
SAN Storage Area Network
VM Virtual Machine
VMM Virtual Machine Monitor
1
Chapter 1
Introduction
1.1 Overview
In an IT environment, aggregation of computing technology and storage resources
enables an enterprise to reduce the operational cost, and improve efficiency and flexibility in the
hardware utilization. Organizations have huge servers, costly data centers, consume a lot of
power, require and space and generates large amount of heat. In earlier times, it was cumbersome
to maintain the larger mainframe computers. However all the work was done by the mainframe
and workstations were the dumb terminals [1]. Organizations had to invest a considerable
amount of time and money into the mainframe technology to keep up with tasks. Mainframe
Computers were quite expensive, occupied more space for hardware, and required more human
attention and more resource consumption. Organizations required more hardware for different
projects, and to manage more data and software; huge data centers were required. To meet
business requirements, exponentially growing data, organizations sought effective and maximal
storage capabilities. All this led to the rise of virtualization and Storage Area Network (SAN) [2]
to gain higher network efficiency. Virtualization came into play to consolidate the individual
servers and today more and more servers are opting for SAN storage, thereby making the local
hard drives insignificant.
With the advent of virtualization [3], hardware is consolidated in data centers to a single
modern server that runs several virtual servers, thereby maintaining high performance and
efficiency. Less hardware utilization leads to less consumption of electricity, less space used and
reduced number of electrical components; leading to Green Computing [4]. Virtualization has
gained extensive attention in the growing world of technology. Hardware miniaturization and the
capability of installing more software require an organization to use multi-tenant services. Due to
2
rapid growth and popularity, virtualization is said to be, “The Fundamental block for today‟s
technological world”. Virtualization is further explained in detail in Chapter 2.
1.2 Storage Area Network (SAN)
Every small organization has a strategy to build a central storage so servers, new software
or larger applications which need more memory are constantly developed. Storing data on a local
hard disk is becoming archaic. So the idea of having a central storage that constitutes a SAN
could be as simple as just plugging in the terminal in the mainframe world.
SAN is a collection of hubs, fabric, and software. It is a high-speed network of storage
elements, such as shared storage arrays, and clusters servers, where one or more interfaces create
complete connectivity. A basic SAN environment is as shown in Figure 1. In a typical SAN
environment, uninterrupted availability of data is very important for the survival of an
organization. It is essential to have a reliable infrastructure to ensure that data is available at all
times. The key characteristics of a typical data center are capacity, security; availability,
performance, data integrity and scalability [5]. If all are met appropriately then an organization
can have unremitting services, as shown in Figure 2. Following is the brief description of key
characteristics of a data center:
3
Figure 1 Basic SAN Environment
a) Capacity: As per the growing needs and user demands, capacity requirements should
be increased to provide uninterrupted services. Data center operations must have
adequate resources to store a large amount of data. If more resources are required,
then the data should be reallocated rather than adding new resources.
b) Security: Proper integration and authentication should be maintained in the network
to ensure enough security. Clients should have complete access to allocated resources
on the specific storage arrays.
c) Availability: In an organization, data should be available all the time without any
inconvenience.
4
Figure 2 Key elements of Data Center
d) Performance: The system should have the ability to provide optimal performance by
offering services at high speed.
e) Data Integrity: Data integrity should be maintained by mechanisms such as error
detection and correction.
f) Scalability: The storage should be able to grow with user demands. The system
should be able to allocate additional processing capabilities for storage on demand
without interrupting the other services.
1.3 Security
With the upcoming needs of organizations, more attention is required to provide security
in a network [6]. Storage and virtualization are the evolving technologies, and so are the threats
5
and vulnerabilities. Features in such technologies can lead to unwanted and unexpected security
threats. Often technology is implemented without being aware of the security consequences.
Eavesdropping, data theft, and hacking are the basic security concerns within an organization
and over the Internet. Security could be broken accidentally or intentionally. If the critical
infrastructure is compromised, the consequences would be severe and lead to data loss. SAN is
considered wealthy since it has valuable information and sensitive data. If SAN fabric is
configured using conventional methods without any security controls, then compromising the
network not be difficult.
Organizations should think carefully about their data storage needs for a secure storage
and an integrated solution for threats and vulnerabilities. Security should be implemented layer-
by-layer, so that if one layer is compromised, the assets of the other layers are under protection.
Proper security measures, tools, and encryption techniques should be implemented to have a
secure organization. The technology should be understood to the core and eventually security
should be maintained by taking preventive measures across the network. A virtual environment
is more susceptible to security threats. Possible ways to break into virtualization are to
compromise a Virtual Machine (VM), hack the hypervisor, insert a rootkit, and sniffing the
traffic. All these possible ways need attention in an organization to have fail-proof security.
With virtualization, live VM migration [7] has gained more popularity. Live VM
migration attracted the users because of its transparent nature, and clean separation between
hardware and software. Administrators can consolidate the system load, improve power
efficiency, and improve infrastructure maintenance and flexible relocation of the sources [8].
Security is a bigger concern in Live VM migration as it poses threats to VM as well as network.
Data is in flight as well as at rest, which could be sniffed and security can collapse.
6
1.4 Problem Identification
An indispensable feature in the world of virtualization is live migration of a Virtual
Machine (VM). Live VM migration means migration of an entire OS and its associated
applications from one physical machine to another. An administrator can ensure complete
utilization of the available resources on any one machine or another. But with some added
advantage comes a threat. Security is the biggest concern while doing migration and has not
received its due attention. The VMs are migrated “live”, without disrupting the applications and
the network with the least minimal downtime. During the migration, an attacker in the middle
can gain control over the VM or capture the traffic and try to recover the data. The traffic
between the hypervisors can be sniffed and is more vulnerable to man-in-the-middle attacks or
Denial-of-Service (DoS). Confidentiality, data integrity, and essential credentials are lost easily.
Similarly, this could cause killing of the available resources for that particular VM. After losing
vital data, the host machines might have to shut the VM‟s to protect themselvess from an attack.
1.5 Organization of Thesis
This thesis brings a new vision and direction to less focused areas of securing the live
VM migration. Chapter 1 provides a brief overview of security in SAN and virtualization.
Chapter 2 explains virtualization, needs and types of virtualization in depth. Chapter 3 provides a
literature survey of how VM is migrated with minimal downtime and securing live VM
migration. Chapter 4 gives a detailed explanation how live VM migration could cause a big
security threat and about Xen hypervisor. Chapter 5 consists of the experiment and the results.
Chapter 6 concludes the thesis and recommendations are made for future work for securing live
VM migration.
7
Chapter 2
Virtualization
2.1 Definition
Virtualization is the emulation of hardware in the software platform. Virtualization in
simple words is defined as creating a Virtual Machine (VM) or a virtual copy of a device or a
resource, such as a storage device, network or an Operating System (OS). The VM fetches
resources from the physical resources (such as servers, and OS) and makes them look like a
multiple logical resource and vice-versa. Each application needs one server, which increases
cost. Virtualization offers the ability to run multiple applications and OS on the same machines.
Thus a small number of servers in the data center can perform better and accomplish more work
than before
2.2 Origin of Virtualization
Christopher Strachey, the first Professor of Computation at Oxford University and leader
of the programming research group, brought the term “Virtualization” to life, in his paper „Time
Sharing in Large Fast Computers‟. The term Virtual Machine Monitor (VMM) came into being
in 1960 [9] as a software abstraction layer partitioning a network platform into one or more
VMs. Bob Muglia, Senior Vice-President for server and tools business at Microsoft Cooperation
says “Virtualization is an approach to deploying computer resource that isolates different layers
– hardware, software, data, network and storage from each other” [10]. IBM wanted to logically
partition the mainframe computers into VM [18, 19 and 20]. IBM pioneered virtualization in
1970 [9, 11 and 12], but it was eventually perfected by others. Hardware level VMs disappeared
during the 1980‟s and 1990‟s [13, 14]. The dilemma of utilizing efficient hardware was faced by
the IT departments where eventually virtualization was applied to x86 architecture in order to
8
improve the issues such as high maintenance, infrastructure costs, management costs and disaster
protection. In 1990, Java VM [15] was developed apart from IBM‟s mainframe computers.
VMware Workstations were first utilized in 1999 [16] which allowed running virtual machines.
VMM became popular in 2005 and 2006 in academia and industry.
In 2005-2006, Xen researchers introduced the term hypervisor [17]. Since then Xen
became the de facto virtualization architecture as open source virtualization. Intel, AMD, Sun
Microsystems and IBM are building different virtualization strategies which would give the
technical world a new plethora.
2.3 Need for Virtualization
The demand for virtualization is growing because of the following advantages [21]:
Underutilized hardware: Many data centers have machines that utilize only 10-15% of
total processing capacity and the remaining 85-90% is unused.
Smaller footprint: Over the last two decades, technology has made a big jump and has
reached a platform where everything is a click away. The rise of internet accelerated this
transformation. Real-time communication and video-conferences are possible because of
the growing internet. This has led to the need for huge servers causing a real estate
problem for organizations. Space is required for such data centers. This is one of the
important reasons to adapt virtualization. Virtualization helps in reducing the cost of
building more data center space.
More system administration: Due to more hardware, every machine needs constant
monitoring, regular updates, OS, application and software installations. This increases the
amount of labor. With the ease of virtualization, the system administration‟s job is
reduced.
9
Ease of testing and development: Testing becomes easy with the deployment of
different OS environments and isolating the applications per VM.
Mobility: Virtual Machines can be migrated from one physical host to another, giving
better system mobility. This also improves the resource utilization.
2.4 Types of Virtualization:
There are different types of virtualization techniques as per the application or services.
Following are the types of virtualization [22]:
2.4.1 Server Virtualization
In Sever Virtualization, the resources of servers are hidden from its users. The users don‟t
need to understand the management of resources. Server Virtualization is done to increase the
utilization and sharing of resources. There are different ways to do Server Virtualization as:
2.4.1.a Operating System (OS) Virtualization
OS virtualization means running an OS on top of an existing one. The main reason for
OS virtualization is to provide the set of libraries that the application might interact with. The
application in hosted OS can‟t see any other applications in another virtual OS. OS virtualization
is offered by Sun and SW Soft [23] with the commercial product Virtuozzo [24]. OpenVZ [25] is
another open source OS virtualization. A typical OS Virtualization is shown in Figure 3.
10
Physical Hardware
Host Operating System
Guest Operating
System
Guest Operating
System Guest Operating
System
Figure 3 OS Virtualization
2.4.1.b Hardware Emulation
A hardware emulated environment (usually hypervisor) is presented to the guest OS in
the hosted environment. This is also referred to as the Virtual Machine Monitor (VMM). The
guest OS and the VMM form a complete consistent package and that can be completely migrated
from one physical host to another. A hypervisor can virtualize the physical host resources such as
CPU and memory, creating a virtual environment. The VMs running in a virtual environment
have an illusion of being in non-virtualized ones. A hypervisor is responsible for allocating
resources to each VM on demand.
2.4.1.c Paravirtualization
The guest OS is modified to run in paravirtualization. It is a subset of Server
Virtualization; it has a thin software interface between the hardware and the VM, not identical to
that of the underlying hardware. This is shown in Figure 4. The time spent in performing the
operations is reduced because of the modified interface. The guest VMs are aware that they are
11
running in a virtualized environment. The virtual devices rely on physical device drivers of the
underlying host. The device interaction is similar to that of full virtualization [26].
Physical Hardware
Hypervisor
Guest Operating
System
Dom 0
Guest Operating
System 1
Dom U
Guest Operating
System 2
Dom U
User Application User ApplicationUser Application
HypervisorDom 0 Interface
Figure 4 Paravirtualization
The abstraction formed with paravirtualization means that OS performs much better than
full virtualization. But the flexibility and security is lost. Since the OS needs to be modified,
flexibility is compromised. The readily available OS can‟t be used for paravirtualization. The
guest OS is close to the hardware and has more control. This creates risk of impacting the lower
hardware level and compromising the entire network.
2.4.2 Storage Virtualization
The amount of data being sent and received is growing enormously. With so many VMs,
the space required to store data is decreasing daily. Hence, there arises Storage Virtualization. If
data is stored on one machine, a bottleneck will be created if the single machine stops operating.
Organizations use specialized hardware and software along with array controllers, and disk
drivers to provide reliable storage for data processing. The storage is virtually centralized and
could be delivered over a Fiber channel, iSCSI, NFS or other storage protocols.
12
2.4.3 Full Virtualization
The guest OS remains unmodified in Full Virtualization. It provides a virtual
environment, namely a complete simulation of the underlying hardware as depicted in Figure 5.
Every feature of the hardware, such as interrupts, I/O operations, memory access, and full
instruction set which runs on the bare machine, should be reflected on the VM. The guest OS is
not aware that it is running in virtual world.
Physical Hardware
Hypervisor
Guest Operating
System
Guest Operating
System
Guest Operating
System
User Application User ApplicationUser Application
Figure 5 Full Virtualization
Full Virtualization helps isolate the users, and emulating the hardware to achieve security
and reliability, improve the efficiency, simplifies migration and portability. The hypervisor [27]
is used to allow the I/O devices to go to the guest OS machines by imitating the physical devices
in the VMM. It does not need any actual hardware to support virtualization which adds the
overhead performance [28]. Full virtualization is slower than any other virtualization techniques.
13
Chapter 3
Literature Survey
Senior Director of Product Management for VMware, Patrick Lin says, “Virtualization
plays a role as an opportunity and as well as a threat” [29]. Security is the most important aspect
every organization would strive for. A detailed discussion on different security threats and
measures are discussed here.
3.1 Security in Virtualization
Organizations have taken a risk with the technology and getting accustomed to compete
in the world considering the aspects of security. It is difficult to maintain a secure environment,
where frequent access to critical business applications is necessary. Going global requires many
data centers, and innovative business practices. Security experts [30] know that detecting threats
and information leaks is an order of magnitude more difficult than stopping malware. To protect
against this kind of threat very few measures have been taken to secure the crucial and sensitive
data.
As per Citrix whitepaper [31], workforce is mobile, and distributed for people working
remotely from home, offices or mobile devices having internet access. Security becomes
necessary and even more elusive because it brings the associated vulnerability. Protecting the
cooperate resources and assets is not easy. A single security breach can cost millions of dollars,
thereby compromising the organizations sensitive information. With iron clad security and
safeguards in place, a failed system should not bring an entire organization to its knees. A proper
strategy should be followed. Data should reside on servers and within data centers and not on the
14
client devices [32]. If only limited access is available over the network, data is secure even if a
laptop or a computer is compromised.
Virtualization is causing a transformation in the traditional way of resource utilizations,
speeding IT response to a new, well-structured business environment [33]. Any computer is
prone to attacks and threats. However, the virtualized computer is more likely to be exposed to
attacks since security is quite weak due to more holes to patch, and too many interconnections.
Common defensive steps to limit user access are in place, such as the Intrusion Detection System
(IDS)/ Intrusion Prevention System (IPS), and firewalls. The attacks possible in virtualization are
compromising the hypervisor or the kernel, a host module attack, VM escape, VM hopping, VM
monitoring from host, Hyperjacking, and VM sprawl [34], etc. Attackers and hackers [35, 36]
have an eye on one weak entry and can insert malware to detect a virtual environment and
change itself accordingly. One of the major attacks is the man-in-the-middle attack during live
VM migration. During live VM migration, data is in clear text, allowing man-in-the-middle
attack on a VM‟s hypervisor [37]. Virtualization and live migration enable new features
benefitting the end user. This combination gives rise to novel security threats. The hypervisor
implementing live migration feature can possibly expose both the OS and the guest VM to
attacks, resulting in integrity loss.
A VMM encapsulates the vivid and volatile states of a VM. When a VM is suspended, its
state is mapped to files in the local system of the host. Transition of a VM from physical host to
another is called live migration. Live migration provides workload balancing, and resource
sharing. Following are the advantages of live VM migration [38, 39]:
15
- Maintenance and Upgrade: When a physical host needs to be restarted, all the VMs
running on that machine are migrated over to another identical physical host without
causing interruptions in the services.
- Load Balancing: Depending on availability of resources VMs can be migrated
depending on the workload.
- Power Efficiency: If a physical host has only a couple of VMs and the workload is
low, they could be migrated over to other devices. This would reduce the power and
resources used.
3.2 Related Work
To perform live VM migration, state of the art uses a pre-copy approach [40]. This
technique transfers the memory pages first and then copies pages modified later Pre-copy is most
common for live VM migration as well as for wide area migration [41]. VM migrations have
been discovered as a tool providing mobility to users working at different times on different
machines. Thus they set an example by transferring an OS instance from one machine to another.
This collective project [42] aims to optimize the links, and reduce service downtime. Other
projects [43 and 44] have concentrated over long time span by pausing, stopping and then
migrating the memory pages during live VM migration.
Process migration moves a process from one physical host to another. Zap [45] uses
partial OS virtualization for migration of process domains (pods) with the help of a Linux kernel.
VMware added VMotion as an advantage for OS migration to their VirtualCenter management
software [46]. Process migration had been another area of interest for some time and more
research was done in the 1980‟s [47, 48, 49, 50, 51, 52, 53, 54, and 55]. However, process
16
migration didn‟t gain popularity because of the residual dependency limitation and mobility. A
better explanation and survey on process migration technique has been done in [56].
In contrast to process migration, OS migration handles all the limitations of the process
migration and still does the VM migration efficiently. The difference between OS migration and
process migration is that OS migration overcomes the problem of residual dependencies with the
help of a narrow interface between a virtualized OS and the hypervisor. The administrator need
not be concerned with what is running within the VM; instead they can migrate the OS and its
associated processes as one unit.
VM migration really means to migrate the control and memory of a VM from one
physical host to another, without causing any service disruption. Extensive research has been
done on how the memory pages are migrated over during live VM migration. In general,
memory migration [57] can be described as:
a) Push phase: During migration, the source VM is not suspended but is running while a
few pages are pushed to the destination machine. To make sure there is consistency
between the memory pages, the pages that are modified during the migration must be
re-sent.
b) Stop-and-copy phase: The VM to be migrated is stopped. The VM is only started
after the memory pages are transferred across the destination host.
c) Pull phase: If the destination host happens to access a page which is not yet
transferred from the source host, those pages are faulted in (“pulled”) across the
network from the source VM.
17
The best example of memory migration is stop-and-copy which involves starting a new
VM only after stopping the original VM and transferring all the pages. The downtime [58] and
total migration time are proportional to the amount of physical allocated memory of the VM, and
this gets affected since the migration of memory pages takes quite a long time. To have the least
amount of downtime, pure demand migration was adopted which uses a short stop-and-copy
phase. The essential kernel data structures are copied to the destination host. The new VM is
started after the complete migration of memory pages and the other pages are fetched across the
network at first use. This leads to less downtime but increased total migration time. Performance
degradation also occurs due to a considerable set of pages being faulted across.
The best option for VM migration is the pre-copy migration [59]. The pre-copy approach
provides a balanced way of migration by combining the bounded iterative push phase along with
a short stop – and – copy phase. The pre-copying of memory pages occurs in rounds hence the
term “iterative”. The pages which are modified during the first round are copied during the next
round. There will be a small set of pages for each VM which are updated often. Thess turn out to
be poor pages for pre-copy migration. The Writable Working Set was designed to calculate the
number of iterations for typical workloads.
Wide research has been carried out on the topic of making live VM migration efficient
and with minimal downtime. The live VM migration feature came into existence quite some time
ago. It was performed from one server to another server located in the same room or from one
rack to another rack which has physical security, and has minimal chances for loss of data. As
the demand grew for virtualization, live VM migration was performed in a LAN, from one data
center to another data center located in different places. Now the question arises: “How secure is
the live VM migration”? Due to wide spread locations, physical security is not possible, invites
18
data sniffing and poses a security threat. It is observed that during live VM migration, the data
can be sniffed and compromised. Any third party can gain access to the network, sniff the
ongoing traffic and visualize the data. Three different threats were explored during live VM
migration [60]. The traffic going over the data plane is sensitive and is not secured. The transit
path is open to any outside attacker to read the data during the transfer causing a man-in-the-
middle attack. This area of security requires more research since it is open for any attack. Using
the tool Xensploit, several ways to attack the live VM migration were evaluated. This includes
control plane, data plane and migration module. This thesis looks at how security can be
provided during live VM migration.
19
Chapter 4
Xen Overview
Xen provides an exceptional feature called live VM migration. To perform live VM
migration certain requirements must be fulfilled. During the migration, an attacker in the middle
can gain control over the VM or capture the traffic since it is easy to decode the sniffed traffic
and recover the data. The traffic between the hypervisors could be intercepted and is more
vulnerable to man-in-the-middle attack or Denial-of-Service (DoS). Confidentiality, data
integrity, and essential credentials are lost easily. Similarly, this could cause killing of the
available resources for that particular VM. After losing vital data, the host machines might have
to shut the VMs preventing them from being compromised. Not much research has been done in
securing the live VM migration. As mentioned in chapter 3, threats during live VM migration are
discussed [60] but no research has been done on how the traffic goes between the hypervisors.
This thesis concentrates on how the traffic flows between the hypervisors. A small experiment
was conducted to show that during live VM migration data goes in clear text and is vulnerable to
attack. The following section gives an overview of how Xen works and supports live VM
migration.
4.1 Xen Hypervisor
Xen is open source industry software for virtualization. The virtual world has been
gaining popularity with the ease of the Xen hypervisor. Xen offers an efficient, strong and secure
feature set for virtualization. The OS supported by Xen are Windows, Linux, Solaris, so it is
neutral. Being independent, Xen allows Domain 0 to be the unique VM and it has control over
all the other VMs.
20
4.1.1 Why Xen?
Xen is popular because it has features which dominate in the virtualization world and
hence is in demand. Xen doesn‟t contain any device drivers [62]. The VMs installed are isolated
from each other. They are not aware of any other VM running on the same physical OS.
4.1.2 Other Features of Xen
Xen is chosen to be implemented because of its following salient features:
- Privileged Access: Domain 0, also called Dom0 has access to communicate with the
hardware and other Guest VMs.
- Small Base Code: The Xen hypervisor layer is thin, and has a tiny code, restricting the
areas for attacks.
- OS Separation: The Xen hypervisor is separated from the physical hardware. There is no
way to attack the actual OS from the hypervisor.
Xen allows paravirtualization. Paravirtualization allows the guest OS to communicate
with the hypervisor in order to improve the memory, CPU, I/O and other resources. Since a VM
is aware of run in a virtual world, varieties of tasks are achieved in accordance with the
hypervisor. The overall maximum performance is achieved since all the VMs are isolated from
each other. The task given to each VM is processed during the normal operation without any OS
overhead.
Xen version 4.0 was used for live VM migration. Xen 4.0 is the fastest and most secure
virtualization software today [63]. The delivery of Xen 4.0 with state of the art of features and
hardware support highlight the strength and commitment of the open source Xen.org community
[63].
21
4.1.3 Working of Xen
The Xen hypervisor is an abstraction layer of software located on the hardware below any
OS. The hypervisor plays an important role in the execution of a VM. CPU Scheduling, memory
allocation, and disk sharing are all controlled by the hypervisor. Xen consists of two domains,
Domain 0 and Domain U [64]. Domain 0, known as Dom0, is modified Linux kernel. Dom0 is
the only VM having access to the physical I/O resources. Dom0 also communicates with the
other VM (Domain U, known as Dom U) running on the physical host. Every machine running
Xen should have Dom0 running before any other VM starts running. Dom0 has two drivers; the
Network Backend Driver and the Block Backend Driver. The Network Backend Driver
processes all the requests coming from the Dom U. The Block Backend Driver checks with the
local storage disk to process the read or write request of the other VMs.
Domain U paravirtualized guests do not have direct access to the OS and are often
referred as having unprivileged access. All paravirtualized guests are known as DomU PV Guest.
The Dom U PV Guests are aware of having no access to the physical hardware and are running
in a virtual world. Fully virtualized VMs running on Xen are known as Dom U HVM Guests. A
Domain U PV Guest consists of two drivers; PV Network Driver and PV Block Driver. PV
Network Driver is not located within a Domain U HVM Guest and instead requires another
special daemon Qemu-dm which is initialized for each VM in Dom0. This supports access to the
disk. Any request made by the Dom U PV Guests must communicate with the Xen hypervisor
via Dom 0. There is a direct link between Dom0 and Dom U PV Guests known as an event
channel running through the Xen hypervisor.
22
4.2 Requirements for Live VM Migration
To perform live VM migration, there are certain requirements for live VM migration. The
following precautions should be taken for a successful VM migration:
- The two hosts should be configured with the same version of Xen
- Both hosts should be in the same layer 2 network and IP subnet.
- The disk image and the configuration file should be stored on the shared storage. Both
hosts should have access to it.
- The processor on which Xen is installed should be the same. It can cause problems for
migration if the hardware is different. Migration works for the Intel Xeon processor with
four cores to an Intel Core2Duo processor with two cores. If the hardware is entirely
different, migration is not supported.
- The Xen configuration for the Xen daemon is stored in the file “xend-conFiguresxp”. It
can be found at /etc.xen/xend-conFiguresxp. This file needs to be modified on both
hosts.
The xend-config file consists of the following lines:
(xend-relocation-server no)
(xend-relocation-port 8002)
(xend-relocation-address “ ”)
(xend-relocation-hosts-allow “ ”)
The above lines need to be modified to
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address “ ”)
(xend-relocation-hosts-allow “ ”)
23
The xend-relocation-server is set to“yes” so that the receiving host can accept the
relocated guests. The second line specifies that xend should listen on the default port 8002. The
third line specifies which IP address the Xen daemon should listen on for migration requests. If
this field is left blank, then it allows listening on all addresses and interfaces. The last line limits
the number of hosts to contact the server for migration. Once the above requirements are done,
live VM migration should be done easily. The different shared storages available are iSCSI,
NFS, ATA over Ethernet (AoE) [65]. We have used NFS (Network File System) as the shared
storage.
24
Chapter 5
Test Bed and Results
Goals and Limitations:
The research focuses on the following main goals:
Tracing the session establishment details
Running any application that leads to data flow in clear text
Loss and misuse of essential credentials
5.1 Hardware
Two Dell Optiplex 780 (x86_64) servers were used for installing openSUSE OS. The processor
used was Intel Core 2 quad cpu q4800@2.66GHz consisting of 4GB Memory. A Cisco Layer 3
Switch 3550 was used for routing between the two host machines.
5.2 Software
5.2.1 openSUSE
To perform live VM migration, a Xen hypervisor was chosen. Xen was installed on
openSUSE 11.3. Novell sponsors Linux based OS openSUSE which includes a command line
interface and a graphical user interface (GUI). It offers various different GUI as such as KDE,
SC, and GNOME. It has the updated Linux kernel 2.6.34. Installation of openSUSE is simple
and it is user friendly [64]. The setup for Xen is simple and creating VMs is fairly easy. [61]
5.3 Test Bed
The physical hosts A and B run Xen hypervisor. A virtual machine VM 1 was created on
host A. The hosts A and B were connected to layer 3 switch in the same vlan 1 as shown in
Figure 6. A Dynamic Host Configuration Protocol (DHCP) was configured on the switch to get
25
the ip addresses automatically. Connectivity between the hosts and the VM was checked with a
ping test. Traffic was sniffed using Wireshark which is a tool used to capture and analyze the
traffic going through the network. Usually Ethernet network is supported on wireshark. To see
the migration data, a Switch Port Analyzer (SPAN) session was configured on the switch. After
the VM was createda a small script was written executing a small known pattern continuously
15,000 times. The pattern was written to a text file, so the script was doing read and write at the
same time. A continuous ping from the host A to VM 1 was started.
Figure 6 Test bed
To start with the migration, host A sent a SYN (synchronize) packet request to initiate a
connection with host B. Host B replied with an ACK (acknowledge) to host A‟s SYN packet
agreeing, upon building a connection and sends its own SYN packet (both messages are
combined to form a SYN + ACK). Host A then sends an ACK to host B‟s SYN thereby agreeing
to establish the connection. Figure 7 shows the pictorial representation of the three way
26
handshake between host A and host B. To perform the capture, wireshark was turned on using a
laptop connected to the switch in the same vlan. The script was executed and at the same time
VM1 was migrated with the command “xm migrate – l 172.16.0.3”. The migration was
completed. The VM1 was now residing on host B and the script was still being executed on the
host B. The script continued on the host B from the point it stopped on the host A. Even though
the VM migrated, connectivity to the VM was not lost and the service was not disrupted. This is
shown by the continuation of the script and no ping packets being lost.
Figure 7 Three-way handshake between host A and host B
5.4 Results
For testing purposes, only the script was running and no other application was running.
This proves that, during the live VM migration, the service will not be lost or it will be lost for a
minimal time. In the wirsehark capture, the ascii value of the string being printed in the script
was seen. It was easy for an attacker to decode the ascii value and see the data being transferred,
hence data is not secure during live VM migration. If any other applications are running or any
important text files containing confidential data are open, then that data is not secure. The below
27
Figure 8 show the wireshark capture where the three way handshake is established between the
source and destination hosts.
Figure 8 Three-way handshake capture
All three packets (SYN, SYN+ACK, ACK) are clearly visible indicating that the
connection has been established between the hosts A and B. Figure 9 shows capture of the ascii
value for the respective string being printed in the script. The string being printed was a known
data pattern of xyxyxyxyxyyxxxxxxyyyyyyyzzzzzzzzz. In the wireshark captures, the ascii value
for the above pattern was seen. Hence it‟s quite easy for anybody to decrypt these values and
recover the data being transmitted. The ascii value for x is 78, y is 79 and z is 7a which is clearly
visible in the wireshark capture. The ascii value is highlighted in red in the wireshark capture.
28
Along with the data, we can also see the source port, and destination port being used for
migration. The destination could also be compromised if too many requests are being sent on the
same destination port from some other host and the migration can never complete.
Figure 9 Wireshark Capture showing the ascii values
The above test shows that live VM migration is not secure and is the biggest security
threat in virtualization. The continuous execution of the string during live VM migration shows
29
that the string was not interrupted when the VM migrated. The wireshark capture shows the
exact ascii value for the string being tested which makes it easy to sniff the data and create man-
in-the-middle attack. This simple experiment proves that data is in clear text and is open for
attacks.
5.5 Proposed Solution
Security measures to protect the data during live VM migration should be designed in
such a way that data integrity, and confidentiality is maintained. There is always a cost to be paid
in order to make a gain some extra benefit. Following are a few ways for a secure live VM which
migration could be implemented without losing data.
5.5.1 IPSec tunnel
One way of securing secure live VM migration is through building an IPSec tunnel. If
the live VM migration is done across the Internet Protocol Security (IPSec) tunnel, then the cost
to be paid would be the downtime of the VM. Since the migration would be done through the
tunnel, extra overhead and processing would be done and the downtime of the VM might
increase. This would cause disruption in the service for a longer duration, but the data would be
secure since it will be encrypted. IPSec is a protocol for securing the Internet Protocol (IP)
traffic. Authentication and encryption of each IP packet in the communication session is done
when passing through the tunnel. In [66], the author explains operation, authentication and
encryption techniques in detail. IPSec tunnel could be used in protecting the data flow at server-
to-server levels or from edge router-to-edge router. When live VM migration is performed
through the IPSec tunnel, data would be encrypted and difficult to trace. Figure 10 shows that an
IPSec tunnel has been created. Live VM migration is done through the tunnel to ensure data is
encrypted.
30
Figure 10 Live VM migration through IPSec tunnel
Figure 11 shows how data would be encrypted and consists of some random characters
which are not easy to decrypt. The red highlighted portion shows that data is encrypted.
Figure 11 Encrypted data after VM migration through IPSec tunnel
While building the tunnel, various parameters have to be considered like Network
Address Translation (NAT), ip route, and application level inspection. All these network
31
parameters add more headers and cause more delay in processing the packet at each stage. In a
network, there could be many devices which process the packet, add headers, and check the route
before forwarding the packet. During this checking, there could be a possibility of losing the
essential packets. If the core network has a low bandwidth, then there can be the following risks:
Loss of ACK packets: If the ACK is lost, the connection could be dropped and VM
migration is not initiated.
Loss of regular packets: There could be intermittent delay or loss of other essential
packets and VM migration can take more time to complete. Also there could be a
chance of the VM crashing and migration failinh. IPSec is an end-to-end protection
scheme at the Internet Layer which can give security by encrypting the packets but
can cause other problems which are not affordable in an IT organization.
5.5.2 Hypervisor Encryption
Another approach is to do encryption at the hypervisor level. If the Xen hypervisor does
the encryption and sends the data during live VM migration, chances of hacking decrease
considerably. Strong encryption keys should be used. Decryption should be done by the Xen
hypervisor itself to maintain the data integrity and confidentiality. If encryption is done at the
hypervisor level, there would be less overhead, less downtime, less migration time. There would
not be a requirement for application inspection through the tunnel. If a host builds a SSH
connection or any secure connection, encryption would be done by the hypervisor. The
hypervisor should maintain the encryption and decryption keys to ensure that the data is secure.
32
Chapter 6
Conclusion and Future Work
6.1 Conclusion
This research addresses the security concerns in live VM migration and proposes
mitigation techniques. With the advent of virtualization and its new features, there are many
security holes. This research conducts a simple experiment to show how data is visible during
live VM migration. If there is an application running on a VM and if that VM is live migrated to
another physical host, then the data being exchanged in that specific application is in clear text
and could be compromised using a tool wireshark. Using wireshark, a man-in-the-middle attack
can be created by sniffing the traffic flowing between the two hypervisors. To have a secure live
VM migration, migration should be done through an IPSec tunnel or using proper encryption
techniques with strong private and public keys.
6.2 Future Work
This thesis brings to attention the less secure feature of virtualization. Performing live
VM migration is beneficial in many ways if done securely. Live VM migration has a limitation
of migrating in the same subnet. To overcome this limitation, a virtual switch should be
configured to note VM‟s new location and routes the packets as per the new default gateway. To
overcome the security issue, IPSec keeps data secure but has the disadvantage of delaying the
migration. This is due to more downtime of the application on the VM, and extra processing of
the packets. One should concentrate on encryption at the hypervisor level. Embedded hypervisor
[67] is a type 1 hypervisor that supports multiple VMs. A very important feature on the
33
embedded hypervisor is secure encapsulation for any subsystem. Embedded hypervisor gives
less exposure to malicious hackers and protects the complete network from being attacked.
34
REFERNCES
35
REFERENCES
[1] Walter Tichy, Technology Review of Mainframe Computer Systems and their
Alternatives
[2] SAN Basics, A Technical White Paper from MetaStor Storage Solutions, 1999
[3] Kara Nance, Brian Hay, Matt Bishop, Virtual Machine Introspection Observation or
Interference, IEEE Computer and Society, 2008
[4] Server Virtualization: A Step Toward Cost Efficiency and Business Agility, Avanade
Perspective, 2009.
[5] G. Somasundaram, Alok Shrivastava, Information Storage and Management, EMC
Education Services, 2009.
[6] SAN Security: A Best Practices Guide. Incorporating SAN security into the enterprise
with the Brocade Secure Fabric OS, Brocade Communications Systems, Inc.
[7] Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hanseny , Eric July,
Limpach C., Pratt I., Wareld A., Live Migration of Virtual Machines
[8] Ma F., Liu F., Liu Z., “Live Virtual Machine Migration based on Improved Pre-copy
Approach,” Network Management Research Centre, Beijing Jiaotong University
Engineering Research Center of Network Management Technology for High Speed
Railway, Ministry of Education, 2010
[9] Rosenblum M., Garfinkel T., "Virtual Machine Monitor: Current Technology and Future
Trends", Published by the IEEE Computer Society, 2005
[10] Al-Rabayah O., Virtualization Concept and History
http://www.remoteitservices.com/content/virtualization-concept-and-history, 2010
[11] Wahlig E., hardware Based Virtualization Technologies
http://www.redhat.com/f/summitfiles/presentation/June2/ECO/Red%20Hat%20Summit%
20Breakout%20A%20Elsie.pdf, 2006
[12] Fleury E., "Virtualization (Wake up Neo, The Matrix got you)",
http://www.labri.fr/perso/fleury/courses/SS07/download/lectures/01-Virtualization.pdf,
2007.
[13] Rosenblum M., "The Reincarnation of Virtual Machines",
http://www.datatrend.com/trendseletter/Issue_09_Articles/ReincarnationoftheVirtualMac
hine.pdf, 2004
[14] Mijat R., Nightingale A., "Virtualization is Coming to a Platform Near You", White
Paper ARM, 2010
36
[15] Veners Bill. 2000, Inside the Java Virtual Machine
http://www.artima.com/insidejvm/ed2/jvm.html
[16] George K. Thiruvathakula, Konstantin Laufer, Konrad Hinsen, and Joe Kaylor,
"Virtualization for Computational Scientists". Computing in Science &
Engineering, Copublished by the IEEE CS and the AIP, 2010
[17] What is Xen hypervisor? , Xen, http://www.xen.org/files/Marketing/WhatisXen.pdf ,
2004
[18] IBM Virtualization, Virtualization on System z, 2009
[19] IBM Virtualization. Virtualization for Linux, 2009
[20] Adair, R.J., R.U. Bayles, L.W. Comeau, and R.J. Creasy,A Virtual Machine System for
the 360/40, Report 320-2007,May 1966, IBM Cambridge Scientific
Center: Cambridge,MA.
[21] Thomas Burger, The advantages of using Virtualization Technology in the Enterprise,
http://software.intel.com/en-us/articles/the-advantages-of-using-
virtualization-technology-in-the-enterprise/, Aug 2010.
[22] Bernard Golden, Clark Scheffy, Virtualization for dummies, Sun and AMD special
Edition, 2009
[23] Virtuozzo,http://www.swsoft.co.uk/, 2001
[24] OS Virtualization, http://www.parallels.com/products/pvc46/info/virtualization/, 2001
[25] Kirill Kolyshkin , Virtualization in Linux, 2006
[26] A. Mann. The pros and cons of virtualization. BTQ, 2007.
http://www.btquarterly.com/?mc=pros-cons-virtualization\&page=virt-view%research
[27] J. Kirch. Virtual machine security guidelines. The center for Internet Security,
September 2007.
http://www.cisecurity.org/tools2/vm/CIS_VM_Benchmark_v1.0.pdf.
[28] Oliver Garraux, Information Technology Fundamentals, How Virtualization Works and
its Effects on IT, Oct 2007.
[29] K. J. Higgins. VM‟s create potential risks. Technical report, darkreading, 2007.
http://www.darkreading.com/document.asp?doc_id=117908.
[30] Roberts Paul, Secure your enterprise data. InfoWorld IT Strategy Guides at
http://www.infoworld.com/ad/sponsored_resources.html, 2007.
37
[31] Citrix , The End of Application Deployment: Virtualized Applications Streamline, Secure
and Manage Your Business. www.citrix.com. 2008
[32] Gruman, Galen ABC: An Introduction to Mobile Security – CIO, March 08, 2007 From:
http://www.cio.com
[33] Dell Virtualize at the Speed of Your Business, 2009.
[34] Doug Hyde, A Survey On the Security of Virtual Machines, April 2009.
[35] Porter, M. E., Competitive strategy: Techniques for analyzing industries and competitors
New York : The Free Press, 1980
[36] Soror, A.A. Aboulnaga, A. Salem, K., Database Virtualization: A New Frontier for
Database Tuning and Physical Design. Data Engineering Workshop,
2007 IEEE 23rd International Conference (April, 2007)
[37] Jim Carr, Two vulnerabilities found in VMware virtualization products,
http://www.scmagazineus.com, February 2008.
[38] A. Ganguly, A. Agrawal. P.O. boykin, and R. Figueiredo, “WOW: Self-Organizaing
Wide Area Overlay Networks of Virtual Work-stations;” Proceedings of the 15th IEEE
International Symposium on High Perfomance Distributed Computing (HPDC), pages
30-41, 2006.
[39] P.Ruth, J.Rhee, D. Xu, R. Kennell and S. Goasguen, “Autonomic Live Adaption of
Virtual COmputational Environment in a Multi-Domain Infrastructure.” IEEE
International COnference on Autonomic Computing (ICAC'06), 2006.
[40] NELSON, M., LIM, B.-H., AND HUTCHINS, G., “Fast transparent migration for virtual
machines.” In Usenix, Anaheim, CA (2005), pp. 25–25.
[41] Bradford, R., Kotsovinos, E., Feldmann, A., And Schi Oberg, H, “Live wide-area
migration of virtual machines including local persistent state,” Proceedings of the
International Conference on Virtual Execution Environments (2007), pp. 169–179.
[42] C. P. Sapuntzakis, R. Chandra, B. Pfaff, J. Chow, M. S. Lam, and M.Rosenblum.
“Optimizing the migration of virtual computers;” Proceedings of the 5th Symposium on
Operating Systems Design and Implementation (OSDI-02), December 2002.
[43] M. Kozuch and M. Satyanarayanan. Internet suspend/ resume. Proceedings of the IEEE
Workshop on Mobile Computing Systems and Applications, 2002.
38
[44] Whitaker, A., Richard S. Cox, Marianne Shaw, and Steven D. Gribble. “ Constructing
services with interposable virtual hardware,” Proceedings of the First Symposium on
Networked Systems Design and Implementation (NSDI '04), 2004.
[45] S. Osman, D. Subhraveti, G. Su, and J. Nieh. “The design and implementation of zap: A
system for migrating computing environments,” Proc. 5th USENIX Symposium on
Operating Systems Design and Implementation (OSDI-02), pages 361-376, December
2002.
[46] VMWare, Inc. VMWare VirtualCenter Version 1.2 User's Manual. 2004.
[47] Michael L. Powell and Barton P. Miller., “Process migration in DEMOS/MP,”
Proceedings of the ninth ACM Symposium on Operating System Principles, pages 110-
119. ACM Press, 1983.
[48] Marvin M. Theimer, Keith A. Lantz, and David R. Cheriton, “Preemptable remote
execution facilities for the V-system.” Proceedings of the tenth ACM Symposium on
Operating System Principles, pages 2 - 12. ACM Press, 1985.
[49] Eric Jul, Henry Levy, Norman Hutchinson, and Andrew Black, “Fine-grained mobility in
the emerald system. ACM Trans,” Comput. Syst., 6(1):109-133, 1988.
[50] Douglis, F., and John K. Ousterhout, “Transparent process migration: Design alternatives
and the Sprite implementation,” Software - Practice and Experience, 21(8):757-785,
1991.
[51] A. Barak and O. La'adan. “The MOSIX multicomputer operating system for high
performance cluster computing,” Journal of Future Generation Computer Systems, 13(4-
5):361-372, March 1998.
[52] D. Milojicic, F. Douglis, Y. Paindaveine, R. Wheeler, and S. Zhou. Process migration.
ACM Computing Surveys, 32(3):241-299, 2000.
[53] Douglis, F., “Transparent process migration in the Sprite operating system,‟ Tech. rep.,
University of California at Berkeley, Berkeley, CA, USA, 1990.
[54] KERRIGHED. http://www.kerrighed.org, 1998.
[55] Thain, D., Tannenbaum, T., And Livny, M., “Distributed computing in practice: the
condor experience,” Concurr. Comput. : Pract. Exper. 17 (2005), 323–356.
[56] Milojicic, D., Douglis, F., Paindaveine, Y., Wheeler, R., And Zhou, S., “Process
migration survey,” ACM Computing Surveys 32(3) (Sep. 2000), 241–299.
[57] Venkatesha S., Sadhu S., Kintali S.Department of Computer Science, University of
California, Santa Barbara, “Survey of Virtual Machine Migration Techniques,” 2009.
39
[58] Nelson M., Beng-Hong L., and Hutchins G., “Fast Transparent Migration for Virtual
Machines,” VMware, Inc. In 2005 USENIX Annual Technical Conference, 2005.
[59] Marvin M. Theimer, Keith A. Lantz, and David R. Cheriton, “Preemptable remote
execution facilities for the V-system,” Proceedings of the tenth ACM Symposium on
Operating System Principles, pages 2 - 12. ACM Press, 1985.
[60] openSUSE 11.1 Start-up,
http://www.novell.com/documentation/opensuse111/pdfdoc/opensuse111_startup/opensu
se111_startup.pdf, March 2009
[61] SUSE Linux Enterprise Server 11SP1, Virtualization with
Xen,http://www.novell.com/documentation/sles11/pdfdoc/book_xen/book_xen.pdf , Aug
2010
[62] Spector ,S., and Xen.org Community, Why Xen?, 2004.
[63] Xen 4.0 Data Sheet, http://www.xen.org/files/Xen_4_0_Datasheet.pdf, 2004.
[64] XEN, How does XEN Work? , http://www.XEN.org, December 2004.
[65] Brantley C., Coraid Inc, ATA Over Ethernet, 2005.
[66] IPSec Tunnel Creation, SANS InfoSec Reading Room, 2003
[67] Robert D., LynuxWorks San Jose, CA Virtualization and hypervisors aid embedded
design
http://www2.electronicproducts.com/Virtualization_and_hypervisors_aid_embedded_des
ign-article-FAJH05_LynuxWorks-Apr2008-html.aspx, 2008
Recommended