Upload
haven
View
66
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Towards a Virtual Cluster Over Multiple Physical Clusters Using Overlay Network. PRAGMA20 2-4 March 2011 Kei Kokubo , Yuki Fujiwara Kohei Ichikawa, Susumu Date Osaka University Adrian Ho, Jason Haga University of California, San Diego. Background. PRAGMA Grid test-bed : - PowerPoint PPT Presentation
Citation preview
Towards a Virtual Cluster Over Multiple Physical Clusters
Using Overlay Network
1
PRAGMA20 2-4 March 2011
Kei Kokubo, Yuki FujiwaraKohei Ichikawa, Susumu Date
Osaka UniversityAdrian Ho, Jason Haga
University of California, San Diego
Background PRAGMA Grid test-bed :
Shares clusters which managed by multiple sitesRealizes a large-scale computational environment.
› Expects as a platform of computational intensive
applications. Highly independent processes which can be
distributed.Ex) Docking simulation.
22http://www.rocksclusters.org/rocks-register/
Site CSite B
Site A
Large-scaleEnvironment
Grid Environment
OS:Debian lib: glibc2.0OS: Redhat lib: glibc3.0
OS: Redhat lib: glibc2.0
Virtual clusterVirtualized cluster which is composed of virtual machines (VMs). › build a private computational environment that
can be customize for users.› relatively easy to deploy on a single physical
cluster by utilizing cluster building tools.
3Computers at a Site
Local network (LAN)
Virtual machines
(VMs)
OS:Debian lib: glibc2.0
Virtual machines
(VMs)
OS: Redhat lib: glibc3.0
Virtual local network
lib: glibc3.0 lib: glibc2.0
Rocks Developed by UCSD
Rocks is installed on clusters at Sites in PRAGMA test-bed.
Rocks virtual cluster :1. A virtual cluster is allocated a VLAN ID and network2. Virtual compute nodes are automatically installed
via network boot technology (PXE boot)
Frontend node
Rocks
Compute nodes
VLAN 2
VLAN 2
Physical NIC
Physical NIC
VLAN 2
VLANConstructi
on
Physical NIC
Physical NIC
VirtualFrontend
nodeeth0eth1
Virtual Compute
nodeeth0
WANLayer 2 communication is needed( LAN)
Virtual Compute
nodeeth0PXE
booting
Issue : It is difficult to build a virtual cluster over multiple clusters at Grid site with Rocks.
Our Goal
Our Approach› Focus on Rocks› Seamlessly integrate N2N overlay network with Rocks
5
Develop a system which can build a virtual cluster over multiple clusters at Grid sites for computational intensive
applications.
Site A Site BPhysical Network
Rocks cluster A Rocks cluster B
N2N Overlay NetworkRocks virtual cluster
N2N : Overlay network technology Developed by ntop project in Italy
1. Creates an encrypted layer 2 overlay network using P2P protocol.
2. Can establishes layer 2 network spanned on multiple sites.› Utilize TAP virtual network interface (VNIC)
3. Divides overlay networks in similar manner to VLAN ID› Community name
6
Site A Site B
PhysicalNIC
N2NVNIC
Physical NIC
N2NVNIC
LAN LANWANN2N Overlay network
Community name ( network ID)
MAC address13:14:15:16:18:26 11:22:33:44:55:66
MAC address
Virtual cluster construction (1/3) MVC Controller (MVC : Multi-site Virtual Cluster)
Rocks
Databese
Physical NIC
Physical NIC
Physical NIC
Rocks
MVCDatabese
Physical NIC
Physical NIC
Physical NIC
WAN
LAN LAN
Frontendnode
Compute nodesCompute nodes
Databese
Overlay network Constructor
Resource ManagerResource Manager
VM ManagerRegisters multiple Rocks cluster as resources for a virtual cluster.
1.
2.
3.
rocks add mvc Site A:Site b
Site A Site B
Site A Site B
Virtual cluster construction (2/3)
Rocks
Physical NIC
Physical NIC
Physical NIC
Rocks
Physical NIC
Physical NIC
Physical NIC
WAN
LAN LAN
Frontendnode
Compute nodesCompute nodes
N2NVNIC
N2NVNIC
N2NVNIC
Resource Manager
N2N Overlay networkCluster name ( Cluster ID)
Builds a Layer 2 overlay network for each virtual cluster.
MVCDatabese
Overlay network ConstructorOverlay network Constructor
VM Manager
1.
2.
3.
MVC Controller (MVC : Multi-site Virtual Cluster)
Site A Site B
Virtual cluster construction (2/3)
Rocks
Physical NIC
Physical NIC
Physical NIC
Rocks
Physical NIC
Physical NIC
Physical NIC
WAN
LAN LAN
Frontendnode
Compute nodesCompute nodes
N2NVNIC
N2NVNIC
N2NVNIC
Resource Manager
N2N Overlay networkCluster name ( Cluster ID)
Builds a Layer 2 overlay network for each virtual cluster.
MVCDatabese
Overlay network ConstructorOverlay network Constructor
VM Manager
1.
2.
3.
N2NVNIC
N2NVNIC
N2NVNIC
MVC Controller (MVC : Multi-site Virtual Cluster)
Site A Site B
Virtual cluster construction (3/3) MVC Controller (MVC : Multi-site Virtual Cluster)
Rocks
Physical NIC
Physical NIC
Physical NIC
Rocks
Physical NIC
Physical NIC
Virtual Compute
nodeeth0
Physical NIC
N2NVNIC
WAN
LAN LAN
N2NVNIC
Frontendnode
Compute nodesCompute nodes
N2N Overlay networkCluster name ( Cluster ID)
Overlay network Constructor
N2NVNIC
Resource Manager
MVCDatabese
VirtualFrontend
nodeeth0 eth1
WAN
PXEブート
VM ManagerVM Manager
1.
2.
3.
rocks start host vm overlay frontend
rocks start host vm overlay compute nodeA Site =A
Seamlessly connects virtual Frontend node and virtual Compute nodes to N2N
overlay network.
Site A Site B
Virtual cluster construction (3/3) MVC Controller (MVC : Multi-site Virtual Cluster)
Rocks
Physical NIC
Physical NIC
Physical NIC
Virtual Compute
nodeeth0Rocks
Physical NIC
Physical NIC
Virtual Compute
nodeeth0
Physical NIC
N2NVNIC
WAN
LAN LAN
N2NVNIC
Frontendnode
Compute nodesCompute nodes
N2N Overlay networkCluster name ( Cluster ID)
Overlay network Constructor
VM Manager
N2NVNIC
Resource Manager
MVCDatabese
VirtualFrontend
nodeeth0 eth1
WAN
PXEブート
1.
2.
3.
rocks start host vm overlay compute nodeB Site =B
Site A Site B
Feather of our virtual cluster solution
Rocks
Physical NIC
Physical NIC
Physical NIC
Virtual Compute
nodeeth0Rocks
Physical NIC
Physical NIC
Virtual Compute
nodeeth0
Physical NIC
N2NVNIC
WAN
LAN LAN
N2NVNIC
Frontendnode
Compute nodesCompute nodes
N2NVNIC
MVCDatabese
VirtualFrontend
nodeeth0 eth1 WA
N
N2N Overlay networkCluster name ( Cluster ID)Virtual LAN
$ qsub -np $NSLOTS app.sh$ mpirun -np 2 app.mpi
Can use as well as a Rocks virtual cluster at local site.
Environment
13
1. Verify the possibility of building virtual cluster over multiple Rocks clusters.
2. Evaluate calculation performance for a computational intensive application.
WAN emulatorFrontend node of Rocks cluster B
Frontend node of Rocks cluster A
Switch(1Gbps)
4 compute nodes of cluster
A
4 compute nodes of cluster
B
OS: CentOS 5.4(Rocks 5.4)CPU: Intel Xeon 2.27G HZ * 2 (16core)Memory: 12GBNetwork: 1Gbps4 of compute nodes:
……
Experiment
Site A Site B
0ms 20ms 60ms 100ms 140ms0
5001000150020002500300035004000
513 578 603 626 648179 787
20252473 2873
Install and configuration time package download time
Insta
ll tim
e fo
r virt
ual c
ompu
te n
odes
(s
)
The latency at WAN emulator (ms)
1. Verified a virtual cluster over cluster A and B can be built through N2N overlay network
2. Verified possibility of building a virtual cluster in WAN environment.› Change the latency at WAN emulator
0ms, 20ms, 60ms, 100ms, 140ms› Calculate install time for 4 of virtual compute nodes
About 1.0GB packages to install
Experiment (Possibility of Building)
6921365
2628
3099
3521
Verified virtual compute nodes can be installed in WAN
A virtual cluster over multiple Rocks clusters can be built even if Rocks clusters are in WAN
environment.
Measure execution time of a computational intensive application.› DOCK 6.2 (sample program)
30 pieces of compounds for a protein divided by 8 processes. There are few communication between 8 processes
› Change the latency and bandwidth at WAN emulator 20ms, 60ms, 100ms, 140ms / 500Mbps, 100Mbps, 30Mbps
15
Experiment (Calculation Performance)
0ms 20ms 60ms 100ms 140ms0
20
40
60
80
100
63 63 64 64 67
Exe
cute
Tim
e (s
)
The latency at WAN emulator (ms)
1000Mbps
500Mbps 100Mbps 30Mbps0
20
40
60
80
100
63 63 63 63
Exe
cute
Tim
e (s
)
The bandwidth at WAN emulator (Mbps)The effect of the performance is small
even if latency is high and bandwidth is narrow
16
• have designed and been prototyping a virtual cluster solution over multiple cluster at Grid sites.
• Integrate N2N with Rocks seamlessly.• Verify the calculation performance for distributed application will be scale even if in WAN. Environment.
Conclusion
1. Manage multiple virtual clusters deployed by multiple users.
2. Make the install time of virtual compute nodes short.• Improve the performance of N2N overlay network.• Set a cache repository per site.
Future work
Conclusion and Future work
Requirements for our Virtual cluster solution
Rocks with Xen Roll N2N
› RPM package installation. Open some port for N2N
› For edge nodes and a supernode Install MVC Controller
› Composed of Some new python scripts› Provide original rocks commands. (we still have been developing.)
17
Thank you for your attention!
Fin18
19