OpenIO SDS on ARM€¦ · OpenIO SDS is a next-generation object storage solution with a modern, lightweight design that associates flexibility, eﬀiciency, and ease of use. It is

�1

OpenIO SDS on ARM A practical and cost-effective object storage infrastructure based on SoYouStart dedicated ARM servers.

30 September 2017Copyright © 2017 OpenIO SAS All Rights Reserved. Restriction on Disclosure and Use of Data

OpenIO OpenIO SDS on ARM 30/09/2017

Table of Contents

Introduction 3

Benchmark Description 4

1. Architecture 4

2. Methodology 5

3. Benchmark Tool 5

Results 7

1. 128KB objects 7

Disk and CPU metrics (on 48 nodes) 8

2. 1MB objects 9


5. 10MB objects 11


Cluster Scalability 13

Total disks IOps 13

Conclusion 15

2


Introduction In this white paper, OpenIO will demonstrate how to use its SDS Object Storage platform with dedicated SoYouStart ARM servers to build a flexible private cloud. This S3-compatible storage infrastructure is ideal for a wide range of uses, offering full control over data, but without the complexity found in other solutions.

OpenIO SDS is a next-generation object storage solution with a modern, lightweight design that associates flexibility, efficiency, and ease of use. It is open source software, and it can be installed on ARM and x86 servers, making it possible to build a hyper scalable storage and compute platform without the risk of lock-in. It offers excellent TCO for the highest and fastest ROI.

Object storage is generally associated with large capacities, and its benefits are usually only visible in large installations. But thanks to characteristics of OpenIO SDS, this next-generation object storage solution can be cost

effective even with the smallest of installations. And it can easily grow from a simple setup to a full-sized data center, depending on users’ storage needs: one node at a time, with a linear increase in performance and capacity.

OpenIO SDS’s simple, lightweight design allows it to be installed on very small nodes. The minimal requirements for a node on ARM are 512MB RAM and 1CPU core, with packages available for Raspbian and Ubuntu Linux (supporting Raspberry Pi computers). The software’s true flexibility is powered by Conscience technology, a set of algorithms and mechanisms that continuously monitors the cluster, computing quality scores for all nodes, and choosing the

best node for each operation. Thanks to Conscience, all operations are dynamically distributed and there is no need to rebalance the cluster when new resources are added.

By partnering with SoYouStart, a dedicated server provider built on OVH’s infrastructure around the world, we have been able to build a cluster and run a complete set of benchmarks to demonstrate the benefits of an ARM-based storage solution that can quickly scale from three nodes to an unlimited number of nodes at a reasonable cost. Thanks to its very competitive offer, SoYouStart enables small and medium-sized organizations to build private

infrastructures and making them available on high-speed public networks without the hassle of managing physical hardware purchasing and maintenance.

For this benchmark, the OpenIO team worked on a 48-node configuration to take advantage of erasure coding and parallelism, but the minimal configuration supported in production starts at three nodes, allowing end users with the smallest storage needs to take advantage of object storage at reasonable prices.

SoYouStart offers ARM-based servers in European and North American datacenters, and the configuration described in the following pages is replicable in those countries, allowing end users to be compliant with local laws and regulations. (https://www.soyoustart.com/en/server-storage/)

3

https://www.soyoustart.com/en/server-storage/


Benchmark Description 1. Architecture

For our test, we chose to work on a two-tier architecture. This allowed us to install the necessary software to run the benchmark testing suite on two X86 nodes, which also acted as Swift gateways, while the storage layer was built around 48 ARM servers (2 CPU cores, 2GB RAM, 2TB HDD, unlimited traffic, and a public IP address: https://www.soyoustart.com/fr/offres/162armada1.xml).

Each node was configured to reflect the minimum requirements for OpenIO SDS. FileSystem:

• Rootvol (/): EXT4, 8 GB • Datavol for OpenIO SDS data (/var/lib/oio): XFS, 1.9 TB

The object store was configured using a dynamic data protection policy which enabled erasure coding for

objects larger than 250 KB and three-way replication for smaller ones.

OpenIO SDS is easy to deploy and scale thanks to the available Ansible role (available at https://

github.com/open-io/ansible-role-openio-sds) and the OVH/SoYouStart APIs.

4

https://www.soyoustart.com/fr/offres/162armada1.xml

https://github.com/open-io/ansible-role-openio-sds

https://github.com/open-io/ansible-role-openio-sds


2. Methodology

For the benchmark, the platform was populated with 500 containers and 50 objects in each container. Sizes of objects were 128KB, 1MB, or 10MB distributed in equal parts.

A first type of test (80% read / 20% write), which is close to a real use case scenario, was performed for each object size, and we launched seven different runs based on different levels of parallelism (5, 10, 20, 40, 80, 160, and 320 workers).

A second test was designed to test the linear scalability of the solution. In this case, a 100% read run was launched against 10MB objects on three different cluster configurations: 12, 24, and 48 nodes.

Each runs lasted 5 minutes (long enough to see any performance issues).

3. Benchmark Tool

We ran the tests using COSbench, a tool developed by Intel to test object storage solutions. It is open source, so results can be easily compared and verified. In this case, we chose to use the Swift API, but

OpenIO SDS is also compatible with the S3 API.

COSbench (https://github.com/intel-cloud/cosbench) features include:

- Easy to use via a web interface or on the command line - Exports significant metrics for comparative usage - All metrics are saved in CVS format

5

https://github.com/intel-cloud/cosbench


The benchmark was organized in five phases: init: container creation prepare: container population with objects main: bench scenario (with all possible read/write/delete combinations)

cleanup: object deletion dispose: container deletion

Here is an example of a result page.

6


Results 1. 128KB objects

7

128KB objects 80% read

Thro

ughp

ut (o

p/s)

0

40

80

120

160

Res

pons

e tim

e (m

s)

0

200

400

600

800

Number of workers5 10 20 40 80 160 320

Response time (ms) Throughput (op/s)

144,59141,23139,67138,9

98,46

70,46

34,13

128KB objects 20% writeTh

roug

hput

(op/

s)

0

10

20

30

40

Res

pons

e tim

e (m

s)

0

450

900

1350

1800



35,2334,8234,4634,23

25,01

17,94

8,72


Disk and CPU metrics (on 48 nodes)

8


2. 1MB objects

9

1MB objects 80% read

Thro

ughp

ut (o

p/s)

0

35

70

105

140

Res

pons

e tim

e (m

s)

0

200

400

600

800



137,59136,84130,81126,25

84,06

48,26

29,49

1MB objects 20% write

Thro

ughp

ut (o

p/s)

0

10

20

30

40

Res

pons

e tim

e (m

s)

0

600

1200

1800

2400



34,4333,683331,23

20,55

11,83

7,15



10


5. 10MB objects

11


Thro

ughp

ut (o

p/s)

0

17,5

35

52,5

70

Res

pons

e tim

e (m

s)

0

1000

2000

3000

4000



69,68

63,41

53,08

43,99

32,09

21,17

11,38

10MB objects 20% write

Thro

ughp

ut (o

p/s)

0

4

8

12

16

Res

pons

e tim

e (m

s)

0

1500

3000

4500

6000



15,8915,69

13,22

10,75

8,13

5,14

2,95



12


Cluster Scalability To demonstrate the linear scalability of OpenIO SDS, as well as its ability to scale quickly when needed, we simulated a cluster of 12 nodes as it was expanded to 24, then 48 nodes. We ran three benchmarks using 80 workers configured to perform 100% read operations on 10MB objects with the three configurations.

Total disks IOps

Each disk delivers between 180 IOps and 210 IOps, which is very good for SATA disks (and is also the limit as we reach 100% disk utilization).

13


Thro

ughp

ut (o

p/s)

0

25

50

75

100

Band

wid

th (M

B/s)

0

275

550

825

1100

Number of nodes12 24 48

Bandwidth (MB/s) Throughput (op/s)

99,42

54,05

22,96

1.044,48

553,45

235,15235,15

553,45

1.044,48

80 workers

12 nodes 2,5 KIOps

24 nodes 5 KIOps

48 nodes 8,7 KIOps


On the 48-node cluster, we configured the number of workers to increase progressively from 20 to 40, then 80, 160, and 320. After the COSbench preparation phase, we found that the highest bandwidth was achieved with 80 workers. After that, disks reached saturation and performance decreased.

Maximum bandwidth is 1.02 GB/s, with 8.16 Gbps of data delivered to the client application.

14


Conclusion The limitation of this infrastructure comes from the SATA drives, with the exception of the first benchmark run with small 128KB objects.

With OpenIO SDS, SoYouStart ARM servers can achieve the full performance of the single drives they host, without being limited by their ARM CPU. Using the default configuration, cluster performance grows linearly as the number of nodes increases. The more nodes there are in a cluster, the better the performance.

At peak performance, using a 100% read benchmark scenario, and with objects of 10MB in size, we achieved 8.16 Gbps (1GB/s) on the 48-node cluster. This performance is what could be expected from the SATA drives handling both data (sequential IOps) and metadata (random IOps).

The principle of a single-drive ARM server is very appealing, as long as it is able to deliver the full performance of its drive. It reduces the failure domain to its smallest form: one drive. Nowadays, as x86 storage boxes get larger and larger, a failed server can mean that up to 90 disks (900 TB raw) can be taken offline simultaneously. On the contrary, with this type of ARM node, only one drive is affected by a server failure. It also simplifies maintenance as drives are not easy to replace within a large x86 server. This operation requires a lot of care, considering the risk of losing the rest of the server (or switching the wrong drive). In the case of a single-drive server, drives are replaced alongside the rest of the server components. This operation is equivalent to the simple task of adding a drive (and its server) to the cluster, very much like adding additional capacity when needed.

Even though this test was meant to demonstrate OpenIO SDS’s capabilities, it also highlights the fact that SoYouStart ARM servers are inexpensive and can be the base of an interesting hardware infrastructure to build next generation private cloud services. These can allow end users to maintain full control of data while avoiding lock-in, and provide services at a reasonable price. This is a compelling solution, especially for development and testing scenarios.

15


16

FR2 bis avenue Antoine PinayParc d’Activité des 4 Vents59510 Hem

US180 Sansome Street, FI4San Francisco, 94104 CA

openio.io

Next-generation Object Storage and Serverless Computing

JP1-35-2 Grains Bldg. #61 Nihonbashi-Kakigara-cho, Chuo-ku, Tokyo , Japan 103-0014

http://www.openio.io

http://www.openio.io

Documents

OpenIO SDS on ARM€¦ · OpenIO SDS is a next-generation object storage solution with a modern, lightweight design that associates flexibility, eﬀiciency, and ease of use. It is