Evaluation of Virtual Clusters Performance on a Cloud Computing Infrastructure

Evaluation ofVirtualClusters

Performanceon a CloudComputing

Infrastructure

NunoCardoso, Tito

Vieira andJorge G.Barbosa

Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Evaluation of Virtual Clusters Performance ona Cloud Computing Infrastructure

Nuno Cardoso, Tito Vieira and Jorge G. Barbosa

Universidade do Porto, Faculdade de Engenharia

November 4, 2011

Nuno Cardoso, Tito Vieira and Jorge G. Barbosa (Universidade do Porto, Faculdade de Engenharia)Evaluation of Virtual Clusters Performance on a Cloud Computing InfrastructureNovember 4, 2011 1 / 26



Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Outline

1 Introduction

2 Hypothesis

3 Goals

4 Methods

5 Results

6 Conclusions




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

User’s perspective

• Increase of computing needs

• Lack of flexibility• SW requirements different from the available• Permissions to the available platforms are restricted• Clusters exist for specific applications




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Provider’s perspective

• High energy cost

• HW aging without proper use

• SW is tightly coupled with HW leading to:• Difficult and lengthy migrations• Inflexibility• Difficult substitution of failing components

• Difficulty in meeting users’ requirements




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Hypothesis

Cloud Computing enables a better usage of the availablehardware without seriously compromising performance.




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Goals

• Explore the creation steps of an IaaS Cloud:• In an academic context• Using both specialized and non-specialized hardware• Using open source technologies

• Evaluate the losses introduced by virtualization

• Analyze the benefits of the Cloud paradigm




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Resources characterization (1)

• More non-specialized HWthan specialized HW

• Specialized HW older thannon-specialized HW

• Inferior performance• Higher energy

consumption• Closer to failure

Specialized: 256 Cores Non-Specialized: 3328 Cores

Figure: Computer resources




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions


• High uptimes• 70% > 40m• 42% > 1h30

• Low loads

• Possibility of running jobsduring those periods

10 < t <= 40: 30% 40 < t <= 90: 28%

90 < t <= 160: 23% 160 < t <= 250: 11%

250 < t <= 360: 5% 360 < t <= 490: 3%

Figure: Non-specialized HWuptimes




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions


• Tier 1 nodes:• Hardware:

• 2x Intel Xeon X3440(4 cores each)

• 24GB RAM• GbE network

• Operating System:Debian GNU/Linux 6

• Tiers 2 and 3 nodes:• Hardware:

• Intel Core 2 QuadQ9300 (4 cores)

• 4GB RAM• GbE network

• Operating System:Ubuntu FEUP

Figure: Multi-tier setup




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Technologies

Infrastructure

• Debian GNU/Linux 6

• OpenNebula 2.0.1

• KVM 0.12.5

• OpenVPN 2.1

• OpenSSH 5.5

Benchmarks

• Phoronix Test Suite 3.2

• HPCC 1.4.1

• OpenMPI 1.4.2

• Condor 7.4.4

• Riak 0.14.1

• Cassandra 0.7.5




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Benchmark Summary

Name Scope

Deployment Infrastructure

Phoronix CPU/RAM, Disk IO

HPCC CPU/RAM, Network, HPC

Condor Infrastructure, HTC

Distributed DBs Infrastructure, Databases

Table: Benchmark Summary




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Deployment

Setup

S1: 1 node → 1 VM

S2, S4 and S6: 1 node at 1x, 2x and 4x capacity (8,16 and 32 VMs)

S3, S5 and S7: 2 nodes at 0.5x, 1x and 2x capacity (8,16 and 32 VMs)

VMs

• 1 Virtual CPU

• 1GB RAM

• 2GB Disk (image: 500MB)

• 4GB Swap




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Deployment

Results

• 1 VM → ∼ 1.8 sec• Elastic capabilities• Enables duplication of

VMs/Clusters

• Transfer bottleneck:Distributed storage

• Processing powerbottleneck:More physical resources

Request Transfer BootAvg Avg Avg

S1 0.6 2.2 216.0S2 0.1 1.9 245.4S3 0.1 1.3 228.5S4 0.1 1.1 466.7S5 0.1 1.3 243.8S6 0.1 1.5 717.4S7 0.1 1.0 418.2

Table: Benchmark results(sec)




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Benchmark Summary

Name Scope










Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Phoronix Test Suite

Setup

• Virtualized vs.Non-Virtualized

• Two Categories:• CPU intensive• IO intensive

Results

• CPU virtualization → Lowoverhead (avg: 1.71%)

• IO virtualization → Highoverhead (avg w/o gains: 22.51%)

• Needing further analysis:Cache

Application Loss

Bullet Physics Engine 2.75 1.29%C-Ray 1.1 0.67%FFmpeg 0.6.1 5.39%FLAC Audio Encoding 1.2.1 1.43%GraphicsMagick 1.3.12 2.33%John The Ripper 1.7.3.1 0.64%LAME MP3 Encoding 3.98.2 4.46%OpenSSL 1.0.0a 0.55%Smallpt 1.0 0.74%

Table: CPU intensive

Application Loss

Dbench 4.0 1 -55.34%Flexible IO Tester 1.21 73.12%PostMark 1.51 -363.33%SQLite 3.7.3 -5.37%7-Zip Compression 9.13 6.68%GnuPG 1.4.10 1.26%Unpacking linux-2.6.32 35.53%x264 2010-11-22 10.95%Compile linux-2.6.32 4.90%

Table: IO intensive




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Benchmark Summary

Name Scope










Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

HPCC - Single Node

Setup

• 8 processes

• 1 Computer (8 cores)

• Several “PxQ” and “N”’s

• Virt. vs. Non-virt.

ResultsSpatial/Temporal Locality

High: Low losses(HPL: 2.2%)

Low: High losses(RandomAccess: 35%)

HPL

Latency

Bandwidth

DGEMM

STREAM

FFTE

RandomAccess

PTRANS

0.2

0.4

0.6

0.8

1

VM Host

Figure: Result comparison

TemporalLow High

SpatialLow Random Access FFT

High PTRANSSTREAM

HPLDGEMM

Table: Data Locality




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

HPCC - Multiple Nodes

Setup

• 16 processes (2x8)

• 2 Computer (16 cores)

• Several “PxQ” and “N”’s

• Virt. vs. Non-virt.

ResultsHuge impact on networking:

Latency: 563% increase

Bandwidth: 70% decrease

HPL

Latency

Bandwidth

DGEMM

STREAM

FFTE

RandomAccess

PTRANS

0.2

0.4

0.6

0.8

1

VM Host

Figure: Result comparison

TemporalLow High

SpatialLow Random Access FFT

High PTRANSSTREAM

HPLDGEMM

Table: Data Locality




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Benchmark Summary

Name Scope










Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Condor

Setup

• S1: Baseline8 VMs → 4 (Tier 1) Computers

• S2: Scalability/Resourceexploitation8 + 15 VMs → 4 (Tier 1) + 15 (Tier 2) Computers

• S3: Elasticity/Resource exploitation15 VMs → 15 (Tier 3, 10a.m. until 19p.m.) Computers

• Job: Subdivided into 10 minutechunks, Embarrassingly Parallel

• VMs: 4 CPUs, 3GB RAM

Figure: Multi-tier setup




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Condor

Results

• Execution times:• 1 VM: 1 month (extrapolated)• S1: 29 hours• S2: 10 hours• S3: 74 hours (extrapolated)

• Nearly linear gain

• Possibility of using HW in both dedicated (S1 and S2) andnon-dedicated (S3) fashions

• Enables using unexploited (Tiers 2 and 3) HW to performtasks with no extra expense




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Benchmark Summary

Name Scope










Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Distributed Databases

Setup

• Baseline6 nodes (2 CPUs, 4GB RAM), 1GB data per node

• S1: ElasticityStart from baseline; add 1 node; measure latencies

• S2: ScalabilityStart from baseline; scale up to 12 nodes while measuring latencies

• Databases:• Riak• Cassandra




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Distributed Databases

Results

• IO was a bottleneck

• S1: The Cloud introducedelasticity in the clusterwith a transient (15 min)overhead

• S2: The cluster scaledwhile maintaining latencyconstant (4 ms variation)

0m

5m

10m

15m

20m

25m

30m

35m

40m

45m

50m

55m

60m

0 ms

500 ms

1000 ms

1500 ms

2000 ms

Cassandra Riak

Figure: S1 Result

6 Nodes

7 Nodes

8 Nodes

9 Nodes

10 Nodes

11 Nodes

12 Nodes

13 ms

14 ms

15 ms

16 ms

17 ms

18 ms

19 ms

20 ms

Cassandra Riak

Figure: S2 Result




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Conclusions

Dynamic architecture enabled:

• Performance scaling

• Cluster duplication

• Redundancy creation

• Increase infrastructure’s capabilities by using non-specialized HW

Performance losses

• IO → Depends on cache’s efficiency, normally high (around 20%)

• Networking → High (Latency: 563% increase, Bandwidth: 70% decrease)

• CPU → Depends on the data locality:

• High → Low (≤ 5%)

• Low → High (≤ 35%)




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Future Work

Evaluate other setups

• Xen, VirtualBox, VMWare, etc.

• Different IO modules/configurations

• OpenStack, Nimbus, Eucalyptus, etc.




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Q&A

?




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Backup

1000 2000 4000 8000 16000 32000 46336

Problem Size

−140 %

−100 %

−60 %

−20 %

20 %

60 %

100 %

140 %

Gain

VM (resources x 2) HOST (resources x 2)

VM (resources x 4) HOST (resources x 4)

Figure: HPL performance gain by increasing processing power




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions

Backup - HPL Benchmark Results(8 processes)

Problem SizePxQ 2K 4K 8K 16K 32K 46K Avg1x8 12.4 23.0 28.4 30.2 29.3 28.9 25.42x4 12.9 20.3 25.5 27.7 28.5 28.6 23.94x2 9.4 16.7 24.0 28.3 32.3 31.9 23.88x1 11.0 12.2 20.5 30.3 36.7 38.2 24.8Avg 11.4 18.0 24.6 29.1 31.7 31.9 24.5

Table: VM (GFlops)

Problem SizePxQ 2K 4K 8K 16K 32K 46K Avg1x8 13.3 23.9 30.3 33.1 33.1 33.3 27.92x4 9.9 20.5 28.3 32.2 32.7 33.0 26.14x2 9.1 17.4 26.2 32.4 35.7 35.3 26.08x1 8.2 8.6 16.8 27.8 36.5 38.8 22.8Avg 10.2 17.6 25.4 31.4 34.5 35.1 25.7

Table: Host (GFlops)




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions


Problem SizePxQ 2K 4K 8K 16K 32K 46K Avg1x8 7.0 3.9 6.3 8.8 11.7 13.2 9.02x4 -29.8 1.1 9.7 14.0 13.0 13.6 3.64x2 -2.5 4.2 8.2 12.6 9.6 9.7 7.08x1 -33.9 -41.6 -22.0 -8.9 -0.6 1.6 -17.6Avg -12.4 -2.4 3.0 7.2 8.2 9.2 2.2

Table: Performance Loss(%)




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions


Problem SizePxQ 2K 4K 8K 16K 32K 46K Avg1x16 1.6 3.0 6.1 12.7 25.3 34.0 13.82x8 3.5 7.3 14.8 27.7 42.3 47.0 23.84x4 5.9 14.4 28.7 41.1 50.9 54.2 32.58x2 7.9 13.5 26.3 43.8 59.4 65.7 36.116x1 1.3 3.0 6.5 12.4 22.4 29.7 12.8Avg 4.0 8.3 16.5 27.6 40.0 46.1 23.8

Table: VM Results(GFlops)

Problem SizePxQ 2K 4K 8K 16K 32K 46K Avg1x16 3.4 6.8 14.7 28.0 43.2 50.3 24.42x8 5.8 13.7 26.9 44.0 55.6 58.6 34.14x4 5.2 15.6 35.2 51.4 61.2 63.5 38.78x2 7.6 11.0 24.3 44.8 63.0 69.8 36.716x1 3.1 6.5 11.4 21.8 37.4 46.9 21.2Avg 5.0 10.7 22.5 38.0 52.0 57.8 31.0

Table: Host Results(GFlops)




Infrastructure

NunoCardoso, Tito


Introduction

Hypothesis

Goals

Methods

Results

Conclusions


Problem SizePxQ 2K 4K 8K 16K 32K 46K Avg1x16 54.1 56.3 58.2 54.5 41.4 32.2 49.52x8 39.4 46.5 44.9 36.9 23.9 19.6 35.24x4 -12.7 7.9 18.5 20.0 16.8 14.6 10.98x2 -4.3 -23.0 -8.1 2.2 5.6 5.9 -3.616x1 56.6 52.6 42.9 42.9 40.2 36.8 45.3Avg 20.6 28.1 31.3 31.3 25.6 21.8 27.4

Table: Performance Loss(%)


Technology

Evaluation of Virtual Clusters Performance on a Cloud Computing Infrastructure