CLOUDSPECS PERFORMANCE REPORT LUNACLOUD, AMAZON EC2, RACKSPACE CLOUD AUTHOR: KENNY LI NOVEMBER 2012

2

EXECUTIVE SUMMARY This publication of the CloudSpecs Performance Report compares cloud servers of Amazon EC2, Rackspace Cloud Servers (not OpenStack Cloud), and Lunacloud. Performance is measured over a period of 30 days to collect data on server performance capabilities as well as consistency in performance. The CloudSpecs system runs open source benchmark tests to measure performance in CPU, RAM, disk, and internal network. The software is set to automatically run 4 times a day for 7 days a week. The report is comprised of:

1 general server performance test 5 CPU performance tests

3 Disk performance tests 2 RAM performance tests

7 network performance tests

Key Findings

In the general server test, Lunacloud outperforms Amazon and Rackspace consistently over a period of 30 days.

Lunacloud disk performance scores up to 8x better than Amazon or Rackspace disk performance. This is an important consideration for database performance.

Lunacloud’s  internal  network  throughput  is  2x  faster  than  Rackspace’s  internal  network  throughput,  and  significantly  more  stable than Amazon in a period of 30 days. With applications requiring more than one server, internal network speed may be considered a bottleneck in many cases if other components of the server are optimized.

Amazon  EC2’s  CPU  performance  generally  scores  slightly  better  than Rackspace and Lunacloud CPU performance. CPU is a vital component of the server, and application performance depends significantly on CPU functionality.

Rackspace’s  RAM  is  more  consistent  in  performance  than  Amazon or Lunacloud, though it scores the lowest in performance over a period of 30 days. Though the processor speed is an important factor in server performance, RAM plays an equally important role. Because modern CPUs can perform many of the applications run today, bottleneck instances may actually occur with RAM.

METHODOLOGY Cloud Spectator sets up two servers on each cloud provider that meets the following minimum requirement:

Primary Server CPU Cores 4 vCPUs RAM in GB 16 Disk in GB 50

Secondary Server CPU Cores 1 vCPU RAM in GB 4 Disk in GB 50

Because Amazon and Rackspace offer packaged cloud servers with preset allocations of CPU, RAM, and disk, the closest package that meets the minimum requirements is selected. AMAZON The servers are set up on Amazon EC2 availability zone US-East-1a. Virtualized server OS is Amazon Linux.

Primary Server CPU Cores 4 vCPUs RAM in GB 15 Disk in GB 1,690

Secondary Server CPU Cores 1 vCPUs RAM in GB 3.75 Disk in GB 410

RACKSPACE The  servers  are  set  up  in  Rackspace’s  Dallas  (DFW)  data  center. Virtualized server OS is Debian 6.

Primary Server CPU Cores 4 vCPUs RAM in GB 15.872 Disk in GB 620

Secondary Server CPU Cores 2 vCPUs RAM in GB 4 Disk in GB 160

3

LUNACLOUD The  servers  are  set  up  in  Lunacloud’s  Virtualized  server  OS  is  Debian  6.

Primary Server CPU Cores 4 vCPUs RAM in GB 16 Disk in GB 50

Secondary Server CPU Cores 1 vCPUs RAM in GB 4 Disk in GB 50

Cloud Spectator maintains root access to all servers. Using SSH, the CloudSpecs application is installed on to the servers. Both Primary and Secondary servers are located within the same virtual private network. Once the CloudSpecs performance application is installed, it automatically runs 4 times a day, 365 days a year. PERFORMANCE TESTS Cloud Spectator runs basic, open source benchmark tests that are precise and accurate for measuring the performance of a cloud server. Tests included come from the Unixbench Suite (https://code.google.com/p/byte-unixbench) and Phoronix Test Suite (http://www.phoronix-test-suite.com). The benchmarks for network tests and the disk benchmark for local file copy may be reproduced using the Linux command line interface. The network tests assume $SERVER is the IP of the secondary server and $CLIENT is the IP of the primary server. Command line instructions for the internal network tests are included below in each of the tests within the network description.

DATA COLLECTION The CloudSpecs test results for this publication run from October 17, 2012 to November 16, 2012 (a total of 30 days). Cloud Spectator obtains cloud servers by either purchasing the server space directly from the client, or the client reimburses Cloud Spectator for data collected relevant to that active project.

CLOUDSPECS SCORE The CloudSpecs Score represents the true value a cloud user receives when using a server. A higher score means better value. The highest value performer receives a score of 100 and other providers are pegged in relativity to value. Value is calculated based on performance over time and the cost of the Primary Server hourly (730 hours per month).

Provider Cost per Hour Amazon EC2 East $0.64

Rackspace $0.90 Lunacloud $0.32

Below  is  the  formula  process  for  calculating  the  CloudSpecs  Score  (Provider’s  CloudSpecs  Score  is  the  calculation for the final CloudSpecs score of each provider). The CloudSpecs score is rounded off to the nearest whole number. The CloudSpecs score in this report were calculated over a period of 30 days:

provider_value(P) = [Provider test score over a period of time] / [Provider price] best_provider_value = max(provider_values) Provider’s  CloudSpecs  Score = 100 * provider_value(P) / best_provider_value

The  Sectional  CloudSpecs  Score  is  calculated  by  averaging  the  CloudSpecs  scores  of  each  section’s  available  tests.

4

PERFORMANCE SUMMARY General Server Performance

UnixBench (Score) Provider 30-day Average CloudSpecs Score Lunacloud 1,305 100 Amazon 1,037 40 Rackspace 955 26 CPU Performance

7-Zip Compression (MIPS) Provider 30-day Average CloudSpecs Score Lunacloud 4,498 100 Amazon 6,777 75 Rackspace 6,152 49

Linux Kernel Compilation (Seconds to Compile) Provider 30-day Average CloudSpecs Score Lunacloud 449 100 Amazon 234 96 Rackspace 251 64

Apache Kernel Compilation (Seconds to Compile) Provider 30-day Average CloudSpecs Score Lunacloud 69 100 Amazon 67 55 Rackspace 63 37

Video Encoding Using x264 (FPS) Provider 30-day Average CloudSpecs Score Lunacloud 29 100 Amazon 52 90 Rackspace 47 58

Audio Encoding Using LAME MP3 (Seconds to Compile) Provider 30-day Average CloudSpecs Score Lunacloud 25 100 Amazon 31 48 Rackspace 32 28

Disk Performance

DBench (MiB/sec) Provider 30-day Average CloudSpecs Score Lunacloud 399 100 Amazon 129 16 Rackspace 95 8

File Copy (MiB/sec) Provider 30-day Average CloudSpecs Score Lunacloud 754 100 Amazon 26 2 Rackspace 39 2

Iozone (MiB/sec) Provider 30-day Average CloudSpecs Score Lunacloud 1,871 100 Amazon 960 26 Rackspace 716 14

RAM Performance

STREAM (MiB/sec) Provider 30-day Average CloudSpecs Score Lunacloud 11,114 100 Amazon 15,497 70 Rackspace 6,648 21

RAMSpeed SMP (MiB/sec) Provider 30-day Average CloudSpecs Score Lunacloud 7,665 100 Amazon 7,676 50 Rackspace 5,501 25

5

Internal Network Performance

Network Throughput with Iperf (Mibit/sec) Provider 30-day Average CloudSpecs Score Lunacloud 767 100 Amazon 384 25 Rackspace 396 18

FTP to RAM (MiBit/sec) Provider 30-day Average CloudSpecs Score Lunacloud 916 100 Amazon 446 24 Rackspace 187 7

FTP to Disk (MiBit/sec) Provider 30-day Average CloudSpecs Score Lunacloud 933 100 Amazon 279 15 Rackspace 187 7

Apache WGET (MiBit/sec) Provider 30-day Average CloudSpecs Score Lunacloud 864 100 Amazon 528 31 Rackspace 188 8

Nginx WGET (MiBit/sec) Provider 30-day Average CloudSpecs Score Lunacloud 891 100 Amazon 517 29 Rackspace 187 7

Network SCP (MiBit/sec) Provider 30-day Average CloudSpecs Score Lunacloud 169 100 Amazon 118 35 Rackspace 184 39

6

GENERAL SERVER PERFORMANCE Cloud Spectator runs the UnixBench benchmark suite to gauge overall performance of a cloud server. Over the test period of 30 days, Lunacloud outperforms Amazon and Rackspace in the UnixBench test. UNIXBENCH

Description The purpose of UnixBench is to provide a basic indicator of the performance of a Unix-like system; hence, multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system.

Why It Matters The UnixBench is a suite of approximately 10 tests that can measure how many instructions per second a CPU can perform. If CPU A performs 2x better than CPU B at only 1.5x the cost, then CPU A provides more performance per unit of cost.

Provider 30-day Average CloudSpecs Score Lunacloud 1,305 100 Amazon 1,037 40 Rackspace 955 26

7

CPU PERFORMANCE 7ZIP COMPRESSION

Description This is a test of 7-Zip using p7zip with its integrated benchmark feature, measured in million instructions per second (MIPS). The test consists of compressing a file with random data using the 7-zip program and then dividing the number of CPU instructions executed during the compression by the number of seconds. The result is then divided by 1 million to calculate the value in MIPS.

Why It Matters The 7-zip compression test results are important in comparison because the 7-zip compression algorithm uses many kinds of CPU instructions, resulting in an accurate overall score reflecting the performance of the CPU. The fact that it is a compression program is irrelevant in the test.

Provider 30-day Average CloudSpecs Score Lunacloud 4,498 100 Amazon 6,777 75 Rackspace 6,152 49

TIMED LINUX KERNEL COMPILATION

Description This test times how long (in seconds) it takes to build the Linux kernel. Unlike the 7-zip compression, this is a compilation source code. Though many compilation tests exist, an advantage of the Linux Kernel compilation is the portability and compatibility with a majority of Linux distributions.

Why It Matters Because of the variety of CPU instructions and the length of time for the test, the Linux kernel compilation can accurately measure  a  CPU’s  performance.

Provider 30-day Average CloudSpecs Score Lunacloud 449 100 Amazon 234 96 Rackspace 251 64

8

TIMED APACHE COMPILATION

Description This test times how long (in seconds) it takes to build the Apache HTTP Server. Though many compilation tests exist, an advantage of the Apache compilation is the portability and compatibility with a majority of Linux distributions.

Why It Matters Because of the variety of CPU instructions and the length of time for the test, the Apache compilation can accurately measure  a  CPU’s  performance.

Provider 30-day Average CloudSpecs Score Lunacloud 69 100 Amazon 67 55 Rackspace 63 37

X264 VIDEO ENCODING

Description The x264 test benchmarks the speed it takes a server to encode a movie file using the x264 H.264/AVC encoder.

Why It Matters When results are shown alone, they have little meaning, but the process requires a large amount of computations, so when compared side-by-side using results from several cloud providers, the test provides an accurate insight of CPU performance.

Provider 30-day Average CloudSpecs Score Lunacloud 29 100 Amazon 52 90 Rackspace 47 58

9

LAME MP3 ENCODING

Description LAME is an MP3 encoder licensed under the LGPL. This test measures the time (in seconds) required to encode a WAV file to MP3 format.

Why It Matters Similar to the x264 test, results alone have little meaning, but the process requires a large amount of computations, so when compared side-by-side using results from several cloud providers, the test provides accurate insight of CPU performance.

Provider 30-day Average CloudSpecs Score Lunacloud 25 100 Amazon 31 48 Rackspace 32 28

10

DISK PERFORMANCE DBENCH

Description DBench is a benchmark designed by the Samba project as a free alternative to netbench, but dbench contains only file-system calls for testing the disk performance.

Why It Matters DBench can be used to stress a filesystem or a server to see which workload it becomes saturated and can also be used for prediction analysis to determine how many concurrent clients/applications performing this workload the server can handle before response starts to lag.

Provider 30-day Average CloudSpecs Score Lunacloud 399 100 Amazon 129 16 Rackspace 95 8

FILE COPY

Description This measures the performance of the operating system's 'cp' command, followed by a 'sync' (to flush any cache), executed on the local storage device. The file size is 10GB, and is copied to the same disk measured in mebibytes per second.

Why It Matters File copy measures read and write performance of the disk, which is important when processing massive amounts of data.

Provider 30-day Average CloudSpecs Score Lunacloud 754 100 Amazon 26 2 Rackspace 39 2

11

IOZONE FILESYSTEM BENCHMARK

Description Iozone benchmarks file IO performance of a server for the operations: Read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread ,mmap, aio_read, aio_write Why It Matters Iozone can collect data on a broad range of file system performance to locate hot or cold spots in the file systems.

Provider 30-day Average CloudSpecs Score Lunacloud 1,871 100 Amazon 960 26 Rackspace 716 14

12

RAM TESTS STREAM

Description The STREAM benchmark is a simple synthetic benchmark program that measures sustainable memory bandwidth (in MB/s) and the corresponding computation rate for simple vector kernels. The STREAM benchmark is specifically designed to work with datasets much larger than the available cache on any given system, so that the results are (presumably) more indicative of the performance of very large, vector style applications.

Why It Matters STREAM measures memory bandwidth to see how the RAM accommodates to the processes running on the server.

Provider 30-day Average CloudSpecs Score Lunacloud 11,114 100 Amazon 15,497 70 Rackspace 6,648 21

RAMSPEED SMP

Description This benchmark tests the system memory (RAM) performance.

Why It Matters RAMspeed results correlate highly with STREAM results. Though the algorithms are different, RAMspeed SMP also measures the RAM accommodation to processes running on the server.

Provider 30-day Average CloudSpecs Score Lunacloud 7,665 100 Amazon 7,676 50 Rackspace 5,501 25

13

NETWORK PERFORMANCE TESTS NETWORK THROUGHPUT WITH IPERF

Description Iperf is a commonly used network testing tool that can create TCP and UDP data streams and measure the throughput of a network that is carrying them. Our test consists of transferring as much data as possible through the local network (where possible, private network IPs were used) for 120 seconds over TCP port 5001.

Why It Matters Iperf tests the quality of the network. Using this test between two cloud servers in the same data center of the same provider, results can be collected on how efficient the internal network speed is, which is important for servers inside the same virtual private network that may be required to transfer data between or among each other quickly and efficiently to effectively run applications.

Code wget 'http://downloads.sourceforge.net/project/iperf/iperf-2.0.5.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fiperf%2F&ts=1341586993&use_mirror=ignum' -O iperf.tgz tar zxvf iperf.tgz cd iperf-2.0.5 ./configure make ./src/iperf -s -p 5001 wget 'http://downloads.sourceforge.net/project/iperf/iperf-2.0.5.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fiperf%2F&ts=1341586993&use_mirror=ignum' -O iperf.tgz tar zxvf iperf.tgz cd iperf-2.0.5 ./configure make ./src/iperf -c $SERVER -p 5001 -t 120

Provider 30-day Average CloudSpecs Score Lunacloud 767 100 Amazon 384 25 Rackspace 396 18

14

NETWORK FTP TO RAMDISK

Description Using vsftpd (Very Secure FTP Daemon) on Server and lftp on Client, the test transfer a 1GB file from RAM drive to RAM drive.

Why It Matters Transmitting from a RAM disk bypasses the disk in the network test, preventing a possible bottleneck at the disk level and enabling a more accurate measurement of the internal network transfer speed.

Code mkdir -p /mnt/ramfs chmod 0555 /mnt/ramfs mount -t ramfs -o size=1050mb ramfs /mnt/ramfs dd if=/dev/zero of=/mnt/ramfs/file.big bs=1MiB count=1024 time lftpget -d ftp://$SERVER/file.big

Provider 30-day Average CloudSpecs Score Lunacloud 916 100 Amazon 446 24 Rackspace 187 7

15

NETWORK FTP TO HARD DISK

Description Using vsftpd (Very Secure FTP Daemon) on Server and lftp on Client, the test transfer a 5GB file from hard disk to hard disk.

Why It Matters The File Transfer Speed of the Disk is similar to the FTP speed of the RAM disk, but in a manner that measures the speed efficiency of a disk-to-disk transfer, rather than RAM-to-RAM.

Code dd if=/dev/zero of=/tmp/file.big bs=5MiB count=1024 time lftpget -d ftp://$SERVER/file.big

Provider 30-day Average CloudSpecs Score Lunacloud 933 100 Amazon 279 15 Rackspace 187 7

16

NETWORK HTTP Apache

Description Using apache2 as Server and wget as Client, transfer of a 1 GB file.

Why It Matters Performing a more natural test, the Apache Network HTTP, Nginx Network HTTP, and Network SCP tests all combine the internal network with CPU usage to see how fast servers perform when they have added CPU usage.

Code dd if=/dev/zero of=/tmp/file.big bs=1MiB count=1024 time wget http://$SERVER/file.big

Provider 30-day Average CloudSpecs Score Lunacloud 864 100 Amazon 528 31 Rackspace 188 8

17

NETWORK HTTP NGINX

Description Using nginx (a web server application) as server and wget as Client, transfer of a 1 GB file.

Why It Matters Performing a more natural test, the Apache Network HTTP, Nginx Network HTTP, and Network SCP tests all combine the internal network with CPU usage to see how fast servers perform when they have added CPU usage.

Code dd if=/dev/zero of=/tmp/file.big bs=1MiB count=1024 time wget http://$SERVER/file.big

Provider 30-day Average CloudSpecs Score Lunacloud 891 100 Amazon 517 29 Rackspace 187 7

18

NETWORK SCP

Description Copy of a 5GB file using Secure Copy (SCP).

Why It Matters Performing a more natural test, the Apache Network HTTP, Nginx Network HTTP, and Network SCP tests all combine the internal network with CPU usage to see how fast servers perform when they have added CPU usage.

Code dd if=/dev/zero of=/home/guest/file.big bs=5MiB count=1024 time scp guest@$SERVER:file.big .

Provider 30-day Average CloudSpecs Score Lunacloud 169 100 Amazon 118 35 Rackspace 184 39

About Cloud Spectator Cloud Spectator is the premier cloud analyst group focused on infrastructure pricing and server performance. Since 2011, Cloud Spectator has monitored the cloud Infrastructure industry on a global scale and continues to produce research reports for businesses to make informed purchase decisions by leveraging its CloudSpecs utility, an application that automates live server performance tests 4 times a day, 365 days a year with use of open source benchmark tests. The data is compared relatively with pricing from each provider to allow business decision-makers to smart decisions. Cloud Spectator 800 Boylston Street 16TH Floor Boston, MA 02199 Website: www.cloudspectator.com Phone: (USA) 1-617-300-0711