High-performance cluster storage for IOPS-intensive workloads · Red Hat Ceph Storage on Samsung NVMe SSDs The Reference Architecture configurations described inthis document are

White Paper

High-performance cluster storage for IOPS-intensive workloads

Optimize Ceph cluster performance by combining Red Hat Ceph Storage on Samsung NVMe SSDs

White Paper

2

Table of contents

Executive summary 4

Document purpose 4

Introduction 4

Samsung NVMe Reference Design 5

Ceph distributed architecture overview 6

Reference Architecture elements 7

Red Hat® Ceph Storage 7

Red Hat Enterprise Linux® 7

Samsung NVMe SSDs 7


Networking 9

Operational planning considerations 10

Optimized configurations 10

IdentifyingtargetworkloadI/Oprofiles 10

Genericoptimizations 10

IOPSoptimizedconfiguration 12

Throughputoptimizedconfiguration 13

Benchmark results 13

Referencetestconfigurations 13

SequentialIO 14

RandomIO 16

Testing methodology and details 18

Baselinestoragenodeperformance 18

Networkbandwidth 19

Systemtunables 19

CBTtests 20

Summary 20

Appendix 21

Referenceceph.conf 21

SampleCBTtest.yamlfile 29

Billofmaterials 30

White Paper

3

Table of contents

List of Figures

Figure1:SamsungNVMeReferenceDesign 5

Figure2:Cepharchitecture 6

Figure3:HighperformanceCephReferenceArchitectureelements 7

Figure4:SymmetricConfigurationinCeph 11

Figure5:PerformancescalingwithCPUcorecount 12

Figure6:Samsung/RedHatCephReferencetestcluster 13

Figure7:3-nodeclusterreadthroughput–SSDscaling 15

Figure8:3-nodeclusterwritethroughput–SSDscaling 15

Figure9:3-nodeclusterreadthroughput–Capacitysizing 15

Figure10:3-nodeclusterwritethroughput–Capacitysizing 16

Figure11:3-nodeclusterreadthroughput–NICscaling 16

Figure12:3-nodeclusterwritethroughput–NICscaling 16

Figure13:3-nodeclusterreadIOPS–SSD/OSDscaling 17

Figure14:3-nodeclusterwriteIOPS–SSD/OSDscaling 17

Figure15:3-nodeclusterreadIOPS–4KBvs.8KB 17

Figure16:3-nodeclusterwriteIOPS–4KBvs.8KB 18

Figure17:3-nodeclusterreadIOPS–Capacitysizing 18

Figure18:3-nodeclusterwriteIOPS–Capacitysizing 18

Figure19:TestmethodologystepsinReferenceArchitecture 18

List of Tables

Table1:Commonworkloadcharacteristics 4

Table2:PM953specificationsummary 8

Table3:PM1725specificationssummary 8

Table4:All-FlashNVMestoragesystemspecifications 9

Table5:IOPSperformancecomparison 12

Table6:IOPSperformance–4KBvs.8KB 12

Table7:Throughputperformancecomparison 13

Table8:OSDnodeconfiguration 13

Table9:Clientnodeconfiguration 14

Table10:Defaultdebugvaluesintestcluster 14

Table11:ReferenceArchitecturesoftwareversions 14

Table12:SamsungNVMeSSDPM953960GB 31

Table13:SamsungNVMeReferencePlatform(OSDnode)configuration 31

Table14:Genericdual-socketx86server(Clientandmonitornodes)configuration 31

White Paper

4

Red Hat Ceph Storage on Samsung NVMe SSDs

Executive summaryCeph users frequently request simple, optimized cluster configurationsfordifferentworkloadtypes.AsCephtakesonhighperformanceintensiveworkloads,SSDsbecomeacriticalcomponent of Ceph clusters. To address the needs of Ceph userstoeffectivelydeployAll-FlashCephclustersoptimizedfor performance, Samsung Semiconductor Inc. and Red Hat haveperformedextensivetestingtocharacterizeoptimizedconfigurationsfordeployingRedHat® Ceph Storage on Samsung NVMe SSDs deployed in a Samsung NVMe Reference

Document purposeThisdocumentpresentsReferenceArchitecturefordeployingahigh-performanceRedHatCephStorageclusterusingSamsungNVMe SSDs and Samsung NVMe Reference Design. This document details hardware and software building blocks used inperformancecharacterization.ItcoversCephclusterandLinuxoperatingsystemconfiguration,hardwareconfigurationincluding Samsung NVMe Reference Design, network and Samsung NVMe SSDs. The test methodologies to characterize CephclusterperformanceusedCephBenchmarkingTool(CBT)1 for benchmarking.

The targeted audience for this document is system administrators, solution architects and IT planners.

1. Ceph Benchmarking Tool (CBT) - https://github.com/ceph/cbt

IntroductionStorage infrastructure is undergoing tremendous change, particularly as organizations deploy storage to support big data andprivateclouds.Twomajorleadingcausesforthischangeare:

• Adoption of high-performance flash based storage media

NVMehasemergedasanindustrystandardbasedlow-latencystorageinterface.NVMeSSDsinstandard2.5-inchU.2driveformfactorarewidelyavailableinthemarkettoday.Samsunghasdevelopedax86serverreferenceplatformwithdrivebaystosupport24x2.5-inchNVMeSSDs.

• Software-defined storage infrastructure based on clustered storage servers

Cephhasemergedasaleadingsolutiontodeploycost-effectiveandmanageablestorageatscale.Asamodern

storage system for cloud deployments, Red Hat Ceph Storage offers mature interfaces for enterprise block and objectstorage,makingitwellsuitedforactivearchive,richmedia, and cloud infrastructures like OpenStack®.

WhileNVMeSSDsprovidehighrawperformanceandCephisextremelyflexible,deploymentsshouldbecarefullydesignedtodeliverhighperformancewhilemeetingdesiredfaultdomainrisktolerance.OrganizationsneedflexibleandscalableconfigurationsthatscalefromaclusterbuiltforIOPSintensiveworkloadswith100sTBstoaclusterbuiltforthroughputintensiveworkloadswith10sofPBscapacity.

Table1providesthecriteriausedtoidentifyoptimalRedHatCephStorageclusterconfigurationsusingSamsungNVMeSSDs on Samsung NVMe reference platforms that can take upto24x2.5-inchNVMeSSDs.Thesecriteriaareprovidedasgeneralguidelinesforhardwarepurchaseandconfigurationdecisionsthatcanbeadjustedtosatisfyuniqueworkloadblendsofdifferentoperators.Astheworkloadmixvariesfromorganizationtoorganization,actualhardwareconfigurationschosenwillvary.

IOPS optimized

• Lowest cost per IOP

• Highest IOPS

• Meets minimum fault domain recommendation (singleserverislessthanorequalto10%ofthecluster)

• MySQL®/MariaDB®-basedapps on OpenStack

• Block storage

Throughput optimized

• Lowest cost per MBps*

• Highest MBps

• HighestMBpsperBTU

• Highest MBps per watt

• Meets minimum fault domain recommendation (singleserverislessthanorequalto10%ofthecluster) * MBps = Mbytes per sec

• Digitalmediaserverworkloads

• Blockorobjectstorage

Table 1: Common workload characteristics

https://github.com/ceph/cbt

White Paper

5


TheReferenceArchitectureconfigurationsdescribedinthisdocumentaretheresultofextensivetestingbySamsungandRedHattoevaluatetheperformanceofRedHatCephStorage cluster using Samsung NVMe SSDs within a Samsung NVMeReferenceDesign.Thegoalsweretoprovideoptimized,repeatableconfigurationsfortheIOPSandthroughput-optimizedcriterialistedinTable1.

• TestswererunonCephHammer(0.94.5)releaseandLinuxkernel3.10.

• Testswererunona3-nodeCephOSDclusterbasedonaSamsungNVMereferenceplatformusingSamsungPM9532.5-inchNVMeSSDs.

• TCP/IP stack running on Mellanox®40GbEwasusedforcluster messaging.

• TestresultswereproducedviatheCBTframework.

⸰librbdfiotestsuiteisusedforcharacterizingrandomIOperformance

⸰radosbenchtestsuiteisusedforcharacterizingsequentialIO performance

• Areplicationfactorof2wasusedfordataprotection.

• The performance measurements listed in this document areaveragemeasurementsacrossmultipleclients.

• Thetestrundurationsinthisstudyarebasedon5minutetest runs.

Theresultingoptimizedconfigurationscanbeusedasstartingpoints to build a range of cluster sizes, from hundreds of terabytestomultiplepetabytesinsize.Whiletheconfigurationlisted in this document may work with other software combinationsthanoutlined,furtherfine-tuningmayberequiredtomaximizetheperformanceinsuchconfigurations.


SamsungNVMeReferenceDesignisengineeredtoprovideawell-balancedstorageservernodethatincludesmatchingCPUs,networking,storageandPCIeconnectivitytodeploylarge amounts of NVMe SSDs and maximize the performance ofsoftwaredefinedstoragestackssuchasCeph.

SamsungNVMeReferenceDesignisaHighPerformanceall-FlashNVMescale-outstorageServerwithupto24x2.5-inch

hot-pluggableSamsungadvancedNVMeSSDstoprovideextremely high capacity in a small footprint. It is based on PCIe Gen3NVMeSSDstoofferthelowestlatencyintheindustrywithanoptimizeddatapathfromtheCPUtotheSSDs.EachSSDslotprovidespowerandcoolingforupto25WperSSDto enable the support of current and future generation large capacity SSDs as well as SSDs with different endurance and performancelevels.UsingthePM953,themaxcapacitypersystemis46TB.WithnextgenerationPM963SSDs,themaxcapacitypersystemis92TB,whileusingthehigh-endurancePM1725a,themaxcapacitypersystemis153TB.Thisisadual-socketXeon-basedsystemandEIAcompliant2RUchassis.Italsouses4x40GB/snetworkingconnectivitywithRDMA.

TheSamsungNVMeReferenceDesignisavailablethroughStackVelocity(asubsidiaryofJabilSystems)astheGreyguardplatform.

2. Samsung NVMe Reference Design - http://www.samsung.com/semiconductor/support/tools-utilities/All-Flash-Array-Reference-Design/

Figure 1: Samsung NVMe Reference Design

http://www.samsung.com/semiconductor/support/tools-utilities/All-Flash-Array-Reference-Design/

http://www.samsung.com/semiconductor/support/tools-utilities/All-Flash-Array-Reference-Design/

White Paper

6


Ceph distributed architecture overviewACephstorageclusterisbuiltfromlargenumbersofCephnodesforscalability,fault-tolerance,andperformance.Eachnode is based on commodity hardware and uses intelligent Cephdaemonsthatcommunicatewitheachotherto:

• Storeandretrievedata

• Replicate data

• Monitor and report on cluster health

• Redistributedatadynamically(remapandbackfill)

• Ensuredataintegrity(scrubbing)

• Detectandrecoverfromfaultsandfailures

To the Ceph client interface that reads and writes data, a Ceph storage cluster looks like a simple pool where data is stored.However,thestorageclusterperformsmanycomplexoperations in a manner that is completely transparent to the clientinterface.CephclientsandCephobjectstoragedaemons(CephOSDdaemons,orOSDs)bothusetheCRUSH(controlledreplicationunderscalablehashing)algorithmforstorageandretrievalofobjects.

ForaCephclient,thestorageclusterisverysimple.WhenaCephclientreadsorwritesdata(referredtoasanI/Ocontext),it connects to a logical storage pool in the Ceph cluster. Figure 1illustratestheoverallCepharchitecture,withconceptsthatare described in the sections that follow.

ClientswritetoCephstoragepoolswhiletheCRUSHrulesetdetermines how placement groups are distributed across objectstoragedaemons(OSDs).

• Pools:ACephstorageclusterstoresdataobjectsinlogicaldynamic partitions called pools. Pools can be created for particulardatatypes,suchasforblockdevices,objectgateways, or simply to separate user groups. The Ceph poolconfigurationdictatesthenumberofobjectreplicasandthenumberofplacementgroups(PGs)inthepool.Ceph storage pools can be either replicated or erasure coded, as appropriate for the application and cost model. Additionally,poolscan“takeroot”atanypositionintheCRUSHhierarchy,allowingplacementongroupsofserverswith differing performance characteristics—allowing storage to be optimized for different workloads.

• Placement groups:Cephmapsobjectstoplacementgroups(PGs).PGsareshardsorfragmentsofalogicalobjectpoolthat are composed of a group of Ceph OSD daemons that areinapeeringrelationship.Placementgroupsprovidea means of creating replication or erasure coding groups ofcoarsergranularitythanonaperobjectbasis.Alargernumberofplacementgroups(e.g.,200perOSDormore)leads to better balancing.

• CRUSH ruleset:TheCRUSHalgorithmprovidescontrolled,scalable, and declustered placement of replicated or erasure-codeddatawithinCephanddetermineshowtostoreandretrievedatabycomputingdatastoragelocations.CRUSHempowersCephclientstocommunicatewith OSDs directly, rather than through a centralized serverorbroker.Bydeterminingamethodofstoringandretrievingdatabyanalgorithm,Cephavoidsasinglepointof failure, a performance bottleneck, and a physical limit to scalability.

• Ceph monitors (MONs): Before Ceph clients can read or write data, they must contact a Ceph MON to obtain the currentclustermap.ACephstorageclustercanoperatewith a single monitor, but this introduces a single point of failure. For added reliability and fault tolerance, Ceph supportsanoddnumberofmonitorsinaquorum(typicallythreeorfiveforsmalltomid-sizedclusters).Consensusamongvariousmonitorinstancesensuresconsistentknowledge about the state of the cluster.

• Ceph OSD daemons: In a Ceph cluster, Ceph OSD daemons storedataandhandledatareplication,recovery,backfilling,andrebalancing.Theyalsoprovidesomeclusterstate information to Ceph monitors by checking other Ceph

Figure 2: Ceph Architecture

White Paper

7


OSDdaemonswithaheartbeatmechanism.ACephstorageclusterconfiguredtokeepthreereplicasofeveryobjectrequires a minimum of three Ceph OSD daemons, two of which need to be operational to successfully process write requests.CephOSDdaemonsroughlycorrespondtoafilesystemonaphysicalharddiskdrive.

Reference Architecture elementsEachnodeintheCephclusterusedinthisReferenceArchitecturehasfiveelementsasshowninFigure3.

Figure 3: High Performance Ceph Reference Architecture elements

The following sections detail each of the elements in the ReferenceArchitecture.

Red Hat® Ceph StorageDesigned for the cloud, Red Hat®CephStoragesignificantlylowers the cost of storing enterprise data and helps to manage exponentialdatagrowth—efficientlyandautomatically.Deliveredinaself-healing,self-managingplatformwithnosingle point of failure, Red Hat Ceph Storage handles data managementsoadministratorscanfocusonimprovingdataavailabilityforbusiness.

Thekeybenefitsare:

• Value Significantlylowerthecostofstoringdata.Layafoundationfor managing its exponential growth at a low cost per gigabyte.

• Enterprise readiness Integrate tightly with OpenStack®andenjoyadvancedblockstoragecapabilitiesthatworkjustlikeatraditionalblock

storagedevicebutwithhardwareflexibilityandmassivescalability.

• Longevity

Startwithblockstorageandgrowintoobjectstorage,orviceversa.Integratewithexistingstorageinfrastructureeasily.

• Expert backed

TakeadvantageoftheexpertiseofCeph'screatorsandprimarysponsorsthroughbest-in-classprofessionalservicesandtraining.

ThisReferenceArchitectureisbasedonRedHatCephStorage1.3.

Red Hat Enterprise LinuxWith Red Hat® Enterprise Linux®, a platform with unparalleled stabilityandflexibility,businessescanreallocateinfrastructureresourcestowardmeetingthenextchallengesinsteadofjustmaintaining the status quo.

Thekeybenefitsare:

• Freedom through stability

Businessapplicationsrequireatested,proven,predictableplatform. Red Hat Enterprise Linux frees IT personnel todelivermeaningfulbusinessresultsbyprovidingexceptionalreliabilityandmilitary-gradesecurity.

• An ecosystem of solutions and support

With a Red Hat Enterprise Linux subscription, administratorsareconnectedtotheindustry'slargestecosystem of partners, customers, and experts that support and accelerate organizations’ success.

• Confidence through flexibility

RedHatEnterpriseLinuxgivesyoutheflexibilitytotailoryour infrastructure for business needs now and in the future.Asmarketsshiftandtechnologiesevolve,you'llhavetheagility,adaptability,andperformancetosucceed.

ThisReferenceArchitectureusesRedHatEnterpriseLinux7.2withLinuxkernel3.10.x.

Samsung NVMe SSDsSamsungEnterpriseSolidStateDrives(SSDs)arebeingused increasingly as data storage media in computing, communication,andmultimediadevices.MostSSDsuse

Samsung NVMe Reference Design

RAM

RAM

RAM

Red Hat Ceph Storage

Red Hat Enterprise Linux®

x86 CPU

x86 CPU

40 GbE NIC

White Paper

8


NANDflashmemoryasthestoragemedia,whichiscapableofretaining data without an external power supply.

SSDs offer superior reliability compared to traditional hard diskdrives(HDDs).AdvancesinsemiconductorflashmemorytechnologieshaveenabledthedevelopmentofSSDsthatare much larger in capacities compared to HDDs and can be usedasdirectreplacements.SSDsalsoprovetobehighlycosteffectivein-useduetotheirmuchlowerpowerconsumptionandmaintenancecosts.Astheworldleaderinsemiconductormemorytechnology,SamsungrevolutionizedthestorageindustrybyshiftingtheplanarNANDtoaverticalstructure.SamsungV-NANDtechnologyfeaturesauniquedesignthatstacks48layersontopofoneanotherinsteadoftryingtodecreasethecells’pitchsize.SamsungoffersacomprehensiverangeofSSDsfordeploymentinawiderangeofdevicesacrosseveryindustrysegment.SamsungNVMeReferenceDesignsupportsthefollowingSamsungNVMeSSDs:

• PM953:PM953presentsoutstandingperformancewithinstantresponsivenesstothehostsystem,byapplyingPCIeandNVMewithcosteffectiveness.ThroughV-NAND,italsoprovidesDWPDof1.3over3years.Thesedrivessupportpowerfailsupport.StandardLinuxNVMedriversworkwithSamsungPM953drives.

Table2providessummaryofPM953specifications3.

• PM1725:PM1725presentsthehighestlevelswithunsurpassedrandomreadspeedsandanultra-lowlatencyrateusingSamsung’shighlyinnovative3Dvertical-NAND(V-NAND)flashmemoryandanoptimizedcontroller.ItalsoprovidesDWPDof5over5years.Thesedrivessupportpowerfailsupport.StandardLinuxNVMedriversworkwithSamsungPM1725drives.

Table3providessummaryofPM1725specifications.

Model PM953 Interface PCIeGen3x4

Form factor 2.5inch Capacity Upto1.92TB

Sequential read (128 KB)

Upto1,000MB/s Sequential write (128 KB)

Upto870MB/s

Random read IOPS (4 KB)

Upto240KIOPS Random write IOPS (4 KB)

Upto19KIOPS

DWPD 1.3DWPD Production status

Mass production





Upto2,000MB/s



Upto120KIOPS

DWPD 5DWPD Production status

Mass production

3. Samsung SSD PM953 - http://www.samsung.com/semiconductor/products/flash-storage/enterprise-ssd/MZQLV1T9HCJM?ia=832

4. Samsung SSD PM1725 -http://www.samsung.com/semiconductor/global/file/insight/2015/11/pm1725-ProdOverview-2015-0.pdf

Table 2: PM953 specification summary

Table 3: PM1725 specifications summary

http://www.samsung.com/semiconductor/products/flash-storage/enterprise-ssd/MZQLV1T9HCJM?ia=832

http://www.samsung.com/semiconductor/products/flash-storage/enterprise-ssd/MZQLV1T9HCJM?ia=832

http://www.samsung.com/semiconductor/global/file/insight/2015/11/pm1725-ProdOverview-2015-0.pdf

http://www.samsung.com/semiconductor/global/file/insight/2015/11/pm1725-ProdOverview-2015-0.pdf

White Paper

9


Samsung NVMe Reference Design ThisReferenceArchitectureusestheSamsung NVMe Reference DesignasOSDnodesintheCephcluster.Table4providesasummaryoftheReferencesystemspecifications.

Table 4: All-Flash NVMe Storage System specifications

Thekeybenefitsare:

• Quick Time-to-market: FullytestedandvalidatedplatformavailablefromStackVelocity

• Open Reference Design Standardx-86baseddesign Customizable to support different combinations depending on market requirements

NetworkingThisReferenceArchitectureusesMellanoxConnectX®-4ENadapters5in2x40GbEconfiguration.Thekeyfeaturesoftheadaptersare:

• 100GB/sEthernetperport

• 1/10/20/25/40/50/56/100GB/sspeeds

• Singleanddual-portoptionsavailable

• ErasureCodingoffload

• T10-DIFSignatureHandover

• Power8CAPIsupport

• CPUoffloadingoftransportoperations

• Applicationoffloading

• Mellanox PeerDirect™ communication acceleration

• HardwareoffloadsforNVGREandVXLANencapsulatedtraffic

• End-to-endQoSandcongestioncontrol

• Hardware-basedI/Ovirtualization

• Ethernetencapsulation(EoIB)

• RoHS-R6

ThisReferenceArchitecturealsousesMellanoxSX10366,a36-portNon-blocking40/56GbEOpenEthernetSpineSwitchSystem.TheReferenceArchitectureconfigurestheswitchportsin40GbEthernetmode.

5. Mellanox ConnectX®-4 EN - http://www.mellanox.com/page/products_dyn?product_family=204&mtag=connectx_4_en_card

6. Mellanox SX1036 - http://www.mellanox.com/page/products_dyn?product_family=115&mtag=sx1036

Chassis

Form factor 2Ux17.2-inchx30.4-inch

Capacity

20x2.5-inchSamsungNVMeSSDslots(hotswap,front) 4x2.5-inchSamsungNVMe/SATA/SASSSDslots(hotswap,front) 2x2.5-inchSamsungSAS/SATASSDslots(hotswap,rearbulkhead)

Temperature management

4+1redundantfans

Power 1,200W1+1redundantGold

Compute and networking

CPU 2 x Intel®E5-2600v3upto145W

Memory 16xDDR42133MHzL/RDIMM

Solid state drive

20x2.5-inchSamsungNVMeSSDsw/3DV-NAND(hotswap,front) 4x2.5-inchSamsungNVMe/SATA/SASSSDs(hotswap,front) 2x2.5-inchSamsungSAS/SATASSDslots(hotswap,rearbulkhead)

BMC AST2400withdedicated1GbEMACforIPMI2.0

PCIe expansion

2xPCIe3.0x16slots(forhostnetworkcards) 1xPCIe3.0x8slot(usedforoptionalRAIDcard)

LAN 4x1GbE,10GbESFP+,40GbEQSFP,orFDRBalancedbetweenCPUsforquickdatatransfersbetweenthenetworkandthedrive

Rear IO 2xUSB3.02x10GbE VGA1x1GbE(IPMIdedicated)

Front IO Buttons:Pwr,RstUSB2.0(x2) LEDs:Pwr,SSD

Enclosure management

I2C Internal Communication

http://www.mellanox.com/page/products_dyn?product_family=204&mtag=connectx_4_en_card

http://www.mellanox.com/page/products_dyn?product_family=204&mtag=connectx_4_en_card

http://www.mellanox.com/page/products_dyn?product_family=115&mtag=sx1036

http://www.mellanox.com/page/products_dyn?product_family=115&mtag=sx1036

White Paper

10


Operational planning considerationsThis section presents general guidance on operational planning for deploying high performance Ceph Storage clusters using Samsung NVMe SSDs.

• MinimumnumberofCephStoragenodes:3(anoddnumberofmonitornodesismandatory);itisrecommendedtohaveatleast3storagenodesinaCephclustertobecomeeligible for Red Hat technical support.

• Monitornodes:3;itisrecommendedtoconfiguremonitornodesonseparatenodes;whilethesenodesdonotneedtohavehighperformanceCPUs,theywouldbenefitfromhighperformance SSDs to store monitor map data.

• Capacityof1storageserver:<=10%Cephclustercapacity;this recommendation may be relaxed for less critical pilot projects.

• Replicationfactor:2;giventhebetterMTBFandMTTRofflash-basedmedia,manyCephcustomershavechosentorun 2x replication in production when deploying OSDs on flash,vs.the3xreplicationcommonwithmagneticmediadeployments.

• Reservedcapacityforself-healing:Cephhasself-healingcapabilitiestoautomaticallyrecoverthedatafromdrive/node failures. Care should be taken to operate the Ceph clustertoleaveenoughunusedcapacitytorecoverfromfailures.Clusterscomprisedof>10nodesarefrequentlyrunat70-75%capacityinproduction.

• Networking:atleasta10GbEclusterwillberequiredtoleveragetheperformancebenefitsofNVMeSSDbasedCephcluster.Forthroughput-orientedworkloads,atleast40GbEperserverisrecommended.

• Thermalmanagement:Careshouldbetakentomaintaincoolingasspecifiedinthehardwarespecifications.

Optimized configurationsDespitetheflexibilityofCeph,nooneclusterorpoolconfigurationfitsallapplicationsorsituations.Instead,successfulconfigurationofaCephclusterrequiresansweringkey questions about how the cluster will be used and the applicationsitwillserve.

IdentifyingtargetworkloadI/OprofilesAccommodatingthetargetworkloadI/Oprofileisperhapsthemostcrucialdesignconsideration.Asafirstapproximation,

organizations need to understand if they are simply deploying low-costarchivestorageoriftheirstorageneedstomeetspecificperformancerequirements.Forperformance-orientedCeph clusters, IOPS, throughput, and latency requirements mustbeclearlydefined.Ontheotherhand,ifthelowestcostperterabyteistheoverridingneed,Cephclusterarchitecturecan be designed at dramatically lower costs. For example, Cephobjectarchiveswitherasure-codedpoolsandwithoutdedicatedSSDwritejournalscanbedramaticallylowerincostthanCephblockdeviceson3x-replicatedpoolswithdedicatedflashwritejournals.

Formoreperformance-orientedneeds,IOPSandthroughputtargets are often established. Historically, Ceph has performed verywellwithhigh-throughputworkloads,andhasbeenwidelydeployed for these use cases. These use cases are frequently characterizedbylarge-block,asynchronous,sequentialI/O(e.g.,digitalmediaperformancenodes).Incontrast,highIOPSworkloadsarefrequentlycharacterizedbysmall-blocksynchronousrandomI/O(e.g.,4KBrandomI/O).Theuseof Ceph for high IOPS open source database workloads is emerging(e.g.,MySQL,MariaDB,andPostgreSQL)7.Moreover,when Ceph is deployed as Cinder block storage for OpenStack virtualmachine(VM)instances,ittypicallyservesamixofIOPs-andthroughput-intensiveI/Opatterns.

Table1summarizesgenericpropertiesofthesetwoworkloadcategories–IOPSOptimizedandThroughputOptimized.

7. MySQL on Ceph - http://www.slideshare.net/Red_Hat_Storage/my-sql-on-ceph

Generic optimizationsThissectionpresentsconfigurationoptimizationsthatareapplicable to both IOPS optimized and throughput optimized deployments of a Ceph cluster on Samsung NVMe reference platforms.

System tunables CephIOpathtraversesthroughseveralkernelmodulesintheLinuxstack.DefaultvaluesoftheserespectivemoduleswillnotbethebestfitforaCephconfigurationoptimizedforperformance.Section9.3liststhevarioussystemparametersthat are tuned and used in Ceph reference test clusters used forthisReferenceArchitecture.

http://www.slideshare.net/Red_Hat_Storage/my-sql-on-ceph

http://www.slideshare.net/Red_Hat_Storage/my-sql-on-ceph

White Paper

11


Symmetric configuration Inahigh-performanceserverplatformsuchasaSamsungNVMereferenceplatform,NVMeSSDsandhigh-speedNICscompeteforcommonCPUandmemorysystemresources.MaintainingCPUsocketaffinityacrossSSDsandNICportsservingCephOSDdaemonsiscriticaltoachievehigherperformance.Thisis one of the key design features of the Samsung NVMe Reference Design.Figure4showsadual-socketconfigurationwherecoresonCPU0serviceinterruptsfromonesetofNVMeSSDsandNICports,whilecoresonCPU1serviceinterruptsfromadifferentset of NVMe SSDs and NIC ports.

Figure 4: Symmetric configuration in Ceph

This kind of assignment of system resources results in better cachelocalityandavoidstheoverheadassociatedwithcrossingQPI in Intel® Xeon®CPUbasedx86systems.InmultipleNICporttestconfigurationsusedinthisReferenceArchitecture,SamsungNVMereferencenodesareconfiguredinsymmetricmodesasshowninFigure4.

CPU sizing Cephprovidesrichdatamanagementfeatures;thesefeatures,notlistedexhaustively,includedynamicspaceallocationtoobjectsatthetimeofobjectcreationratherthanvolume/imagecreation,flexibilitytodynamicallyupdatetheclusterconfiguration,protectionacrossmultiplefailuredomains,periodic checking to maintain data integrity, etc. In order toprovidesuchrichfunctionalityontopofoff-the-shelfhardware, OSD daemons in Ceph perform a lot of processing.

AhigherCPUcorecountresultsinhigherperformanceinCephforIOintensiveworkloads.ToreachhighestIOPStargets,aratioof10Xeon®coresperNVMedriveisused.Forthroughput-optimized workloads characterized by large sequential IO patterns, this ratio is relaxed.

NIC configuration It’sacommonpracticetobondlow-speednetworkinterfacesto create a higher speed single network interface, which can thenbeusedtoconfigurethepublicandclusternetworksforceph-osddaemons.However,bondingconfigurationsmaynot be required when deploying high performance network interfaces,therebyreducingoperationalcomplexity.ASamsungNVMereferenceplatformsupportstwoPCIeGen3x16slotsforexternalnetworkconnectivity;eachoftheseslotsisattachedtooneCPUsocket.Eachslotallows4x25GbEOR2x40GbEOR1x100GbENICcard.Todeliverhighperformance,it is recommended both the PCIe slots are populated with high-performanceNICcardsbasedontheclusterinterconnectconfiguration.

NICportsonthesecardscanbeconfiguredinstand-alonemode without any bonding. This allows maintaining symmetric configurationasshowninFigure4.Toavoidinter-OSDcommunication within the same node on different NIC ports, ceph.confneedstobeconfiguredasfollows.

osd_crush_chooseleaf_type = 0

In addition, to allow a higher number of simultaneous connectionsfromeachOSD,ceph.confcanbeconfiguredtohaveitsownTCPportrange.Anexamplesettingisshownbelow.

ms_bind_port_min = 7000 ms_bind_port_max = 8000

PCIeGen3|40lanes upto320GB/s

PCIeGen3|40lanes upto320GB/s

QPI Upto9.6GT/s

QPI path

CPU0 CPU1

x4NVMedrives x4NVMedrives40GbENICs 40GbENICs

NVMe NVMeNIC NIC

DRAM DRAM

White Paper

12


Journals InahybridCephconfiguration,itiscommontouseseparateSSDsforOSDjournals.However,inanall-flashCephclusterusing Samsung NVMe SSDs on Samsung NVMe reference platforms,separatingthejournalfromtheOSDdatastoreusuallydoesnotproduceadditionalbenefits.Intheseall-flashconfigurations,aCephjournalisfrequentlyco-locatedonthesameNVMedriveinadifferentpartitionfromtheOSDdata.Thismaintainsasimpletouseconfigurationandalsolimitsthescopeoftheanydrivefailuresinanoperationalcluster.

IOPSoptimizedconfigurationWhendeployinghigherperformanceNVMeSSDstodeliverhigherrandomIOperformance,theOSDnodeconfigurationintheclusterneedstosupportahighernumberofCPUcores.InasingleOSDnodewith4NVMeSSDs,asshowninFigure5,Cephdeliversa24%performanceincreasein100%4KrandomreadIOPSasthecorecountonanOSDnodeisincreasedfrom24coresto36cores.

SamsungNVMereferenceplatformsofferflexibilityinprovisioninghardwaresoastomeetthedeploymentrequirements in terms of random IOPS performance and capacity. Different ceph.conf settings can be used to deploy aCephclustertomeettheneeds.Table5summarizestherandomIOperformancefor4KBIOsobservedwithdifferentSSDsina3-nodeCephreferencetestcluster.Notethatallperformanceismeasuredfromtheclientperspective.Assuch,thewriteperformancenumbersmaskthewriteamplificationrequiredforOSDjournalwritesaswellas2xreplication.Performance from a higher number of SSDs in the cluster is cappedbyCPUbottlenecks.

ToachievedesiredIOPSperformancelevelsinaCephcluster,therearefourkeyattributesthatshouldbeconsidered:

• NumberofOSDnodes–DeliversaggregateIOPSacrossmany nodes

• NumberofSSDsperOSDnode–ProvidesrawdeviceIOPS

• NumberofCPUcoresperSSDs–DrivesrawdeviceIOPS.

• NumberofOSDsperSSD–InsmallerSSDconfigurations,upto4OSDsperSSDwillgivehigherperformance;butinlargerSSDconfigurations,multipleOSDsperSSDmayhaveanegativeperformanceimpactduetoincreasedcontextswitching costs.

While sizing IOPS optimized Ceph clusters, care should also be taken to the IOPS size that cluster is being optimized for. Table 6summarizestherandomIOperformancefor4KBand8KBIOsobservedwiththesamenumberofSSDsina3-nodeCephreference test cluster. The write performance accounts for OSD journalwritesaswellas2xreplication.

AsshowninTable6,anodewithonly4SSDscandeliver80%asmanyIOPSasa24SSDnode.Assuch,toachieveoptimalprice-performance,workloadsrequiringhighestsmall,randomIO performance should be deployed on Samsung NVMe ReferenceDesignbasedsystemswith4NVMedrivesanddual-socket Xeon®E5-2699v3processors.

Figure 5: Performance scaling with CPU core count

Table 5: IOPS performance comparison

Table 6: IOPS Performance – 4 KB vs. 8 KB

# of SSDs per OSD

node

# of OSDs per SSD

Effective cluster capacity

(replication factor = 2)

100% random

read (4 KB)

100% random

write (4 KB)

4 4 5.76TB 579K 70.3K

24 1 34.56TB 693K 87.8K

# of SSDs per OSD node

# of OSDs per SSD

IO size 100% random

read

100% random

write

4 4 4KB 579K 70.3K

4 4 8KB 484K 60K

24 1 4KB 693K 87.8K

24 1 8KB 529K 82.7K

100% 4K random read performance

White Paper

13


ThroughputoptimizedconfigurationSamsung NVMe Reference Design based systems offer flexibilityinprovisioninghardwaresoastomeetavarietyof deployment requirements across IOPS, throughput, and capacity targets.

SamsungNVMeReferenceDesignuniquelyleveragesthehigh throughput capabilities of NVMe SSDs by matching them with high throughput capable NICs, with a balanced NVMe bus design. Samsung NVMe Reference Designs enable Ceph clusterstodeliververyhighthroughputrateswithasmallserverfootprintandloweroperationalcosts.

Different ceph.conf settings can be used to deploy a Ceph cluster to meet these throughput needs. Table 7 summarizes thethroughputperformancefor1MBIOsobservedwithdifferentSSDsin3-nodeCephreferencetestcluster.Notethatallperformanceismeasuredfromtheclientperspective.Assuch,thewriteperformancenumbersmaskthewriteamplificationrequiredforOSDjournalwritesaswellas2xreplication. Performance from a higher number of SSDs in the clusteriscappedbyCPUbottlenecks.

Benchmark resultsThis section presents the detailed results gathered on a Ceph reference test cluster.

ReferencetestconfigurationsThe sizing guidelines are based on the performance benchmarking studies performed on Samsung NVMe Reference DesignusingNVMeSSDsina3-nodeCephclusterasshowninFigure6.

Table8summarizesthesystemdetailsforaSamsungNVMereference platform used in the reference test cluster.

UnlikeforrandomIOPS,increasednumberofOSDsperSSDshowed minimal impact on the throughput performance.

ToachievedesiredthroughputperformancelevelsinaCeph cluster, there are three key attributes that should be considered:

• NumberofOSDnodes–Deliversaggregatethroughputacross nodes.

• NumberofSSDsperOSDnode–Providesrawdevicethroughput.

• NumberofNICsperOSDnode–Providesnetworkthroughputtomatchthedrivethroughput.

Table 7: Throughput performance comparison

Figure 6: Samsung/Red Hat Ceph Reference test cluster

Table 8: OSD node configuration

# of SSDs per OSD

node

# of OSDs per SSD

Effective cluster capacity

(replication factor = 2)

100% random

read (1 MB)

100% random

write (1 MB)

4 4 5.76TB 11.6GB/s 2.1GB/s

24 1 34.56TB 28.5GB/s 6.25GB/sComponent Configuration details

2 x Intel® Xeon® [email protected]

With18corespersocket,thiswillhave36corespersystem. With HT on, OS will list 72 hardware threads.

256GB Toavoidmemorylimitationsinusinghigh-performanceSSDs and NICs.

Upto24xPM953 Eachdrivewith960GBrawcapacity.

Mellanox ConnectX®-4ENadapters

1xto4x40GbEports;toavoidanynetworkbottlenecksindeliveringhighIOPSinaclusterwithreplicationorrecoverytraffic.

9 client nodes Eachnodewith2x40GbE

40 G

bE N

etw

ork

1monitornode

3OSDnodes Eachnodewith4x40GbE

Samsung NVMe Reference Design

White Paper

14

Table 9 summarizes the system details for client nodes used in the reference test cluster.

ThisReferenceArchitectureusedCephBenchmarkingTool1(CBT)tocharacterizetheperformanceofCephclusters.CBTisatestingharnesswritteninpythonthatcanautomateavarietyof tasks related to testing the performance of Ceph clusters. CBTrecordssystemmetricswithcollectl.ceph-osddaemonsareconfiguredwiththesameNICportforbothpublicandcluster networks.

CBT benchmark modules that were used in this Reference Architecturearepresentedbelow:

• radosbench for sequential IO RADOSbenchtestingusestheradosbinarythatcomeswiththeceph-commonpackage.Itcontainsabenchmarkingfacility that exercises the cluster by way of librados, the low levelnativeobjectstorageAPIprovidedbyCeph.Currently,theRADOSbenchmodulecreatesapoolforeachclient.

• librbdfio for random IO Thelibrbdfiobenchmarkmoduleisthesimplestwayoftesting block storage performance of a Ceph cluster. Recent releasesoftheflexibleIOtester(fio)provideaCephRBDioengine.ThisallowsfiototestblockstorageperformanceofRBDvolumeswithoutKVM/QEMUconfiguration,throughtheuser-spacelibrbdlibraries.Theselibrariesarethesameonesusedbythe,QEMUbackend,soitprovidesanapproximationtoKVM/QEMUperformance.

The following sections present performance results of the abovebenchmarkrunson3-nodeCephclusterasshowninFigure6.

Table10liststhedebugvaluesthatwereusedwhengatheringthedata,asdocumentedinthedefaultceph.conffiletomakethe results closer to actual real life deployments.

Table11providesthesoftwareversionsusedintheReferenceArchitecture.

Sequential IOSequentialperformancewasevaluatedforsequentialIOworkloadsoflargeIOsizes;associatedCPU,memoryandnetworkutilizationwerealsoobserved.Allthetestswererunwith2xreplication,andbothjournalandOSDdataarestored in different partitions on the same SSD. Throughput was measuredusingradosbenchunderCBTtestframework;128concurrentopsand8KPGsperpoolwereused.

Throughput Scaling with SSDs ForsequentialreadsoflargeIOsizes(1MB),the3-nodeCephclusterrunningonaSamsungNVMereferenceplatformdelivers28.5GB/s.AsshowninFigure7,thethroughputperformancescalesfrom11.6GB/sto28.5GB/sastheSSDcountincreasedinthe cluster. In clusters with a lower number of SSDs per node (4perOSDnode),thethroughputperformanceisprettyclosetotherawdriveperformance,whereaswithahighernumberofSSDspernode(24perOSDnode),CPUsontheOSDnodessaturatebeforeachievingtherawdriveperformance.However,theincreasednumberofSSDsdeliver~2.4xthroughput.


Table 9: Client node configuration

Table 11: Reference Architecture software versions

Table 10: Default debug values in test cluster

Component Configuration details

2x Intel® Xeon® [email protected]

With12corespersocket,thiswillhave24corespersystem.WithHTon,OSwilllist48hardwarethreads.

128GB Toavoidmemorylimitationsinusinghigh-performanceNICs.

Mellanox ConnectX®-3ENadapters

2x40GbEport

Software Version

Linux distribution CentOS 7.2

Linux kernel 3.10.0-327.13.1.el7.x86_64

Ceph HammerLTS(0.94.5)

debug_crush = 1/1

debug_auth = 1/5

debug_finisher = 1/5

debug_heartbeatmap = 1/5

debug_perfcounter = 1/5

debug_rgw = 1/5

debug_asok = 1/5

debug_throttle = 1/1

White Paper

15


ForIOPS-optimizedworkloads,multipleOSDsmappedtoasingleSSDhaveshowntoimproveperformanceofsmallblockrandomIO.Forthroughput-optimizedworkloadswithlargeblocksequentialIO,however,asshowninFigure7,thisisnotthecase.ForagivennumberofSSDspernode,thesequentialreadperformanceforlargeIOsisrelativelysameasthenumberofOSDsmappedtoeachSSDincreasedfrom1to4.Forthroughputintensiveworkloads,therecommendedconfigurationisa1:1mappingbetweenOSD:SSD.

ForsequentialwritesoflargeIOsizes(1MB),a3-nodeCephcluster running on a Samsung NVMe reference platform deliverseffectivethroughputof6.2GB/s.Thiswriteperformanceismeasuredfromtheclientperspective,andmasksthewriteclusterwriteamplificationfromCephOSDjournalwritesand2xreplicationfordataprotection.AsshowninFigure8,thethroughputperformancescalesfrom2.1GB/sto6.2GB/sastheSSDcountincreasedinthecluster.InclusterswithalowernumberofSSDspernode(4perOSDnode),thethroughputperformanceis~80%oftherawdriveperformance, whereas with a higher number of SSDs per node (24perOSDnode),CPUsontheOSDnodessaturatebeforeachievingtherawdriveperformance.However,theincreasednumberofSSDsdeliver~2.9xthroughput.

AsshowninFigure8,foragivennumberofSSDspernode,thesequentialwriteperformanceforlargeIOsisrelativelysameasthenumberofOSDsmappedtoeachSSDincreasedfrom1to4.Insomecases, due to increase in context switches, lower performance is observedwhenmoreOSDsaremappedtoasingleSSD.Asdiscussedabove,forthroughputintensiveworkloads,therecommendedconfigurationisthemorecommon1:1mappingbetweenOSD:SSD.

Capacity sizing SamsungPM953NVMeSSDsof960GBcapacityareusedinthereferencetestcluster.AsthenumberofSSDsareincreasedfrom4perOSDnodeto24perOSDnode,theeffectivecapacityoftheclusteraccountingfor2xreplicationincreasesfrom5.76TBto34.56TB.

Figure9showstheincreasedsequentialreadforlargeIOs(1MB)throughputtoscalefrom11.6GB/sto28.5GB/sastheclustercapacityincreases.ThedottedlineinthefigureshowsreadthroughputinMB/sperTBofeffectivestoragecapacityinthecluster.Figure10showstheincreasedsequentialwriteforlargeIOs(1MB)throughputtoscalefrom2.1GB/sto6.2GB/sastheclustercapacityincreases;thisperformanceaccountsforOSDjournalwritesand2xreplicationfordataprotection.ThedottedlineinthefigureshowsreadthroughputinMB/sperTBofeffectivestoragecapacityinthecluster.

Figure 7: 3-Node Cluster Read Throughput – SSD scaling

Figure 8: 3-Node Cluster Write Throughput – SSD scaling

Figure 9: 3-node cluster read throughput – Capacity sizing

<OSDs, SSD> vs. 100% sequential read (IO size: 1 MB, replication factor = 2)

Top: OSDs per SSD Bottom: SSDs per OSD node

<OSDs, SSD> vs. 100% sequential write (IO size: 1 MB, replication factor = 2)

Cluster capacity vs. 100% sequential write (IO size: 1 MB, replication factor = 2)


Top: Number of SSDs per OSD node Bottom: Effective storage in cluster (TB)

Thro

ughp

ut (G

B/s)

Thro

ughp

ut (G

B/s)

Thro

ughp

ut (G

B/s)

White Paper

16

Throughput Scaling with NICs TheCephreferencetestclusterisbuiltwith40GbEfabric.Each of the Samsung NVMe reference nodes in the cluster can supportupto4x40GbEports.TheNICportsarenotbondedinthe test cluster.

Figure11andFigure12showthebenefitsofbalancingSSDthroughputwithanumberof40GbENICportsoneachOSDnode.AsthenumberofSSDsisincreasedperOSD,theincreasednumberof40GbENICportsreflectsthethroughputgainsforboth sequential read and write operations of large IOs.

Random IOIOPSperformancewasevaluatedforrandomIOworkloadsofsmallIOsizes;associatedCPU,memoryandnetworkutilizationwerealsoobserved.Allthetestswererunwith2xreplication,andbothjournalandOSDdataarestoredinseparatepartitionson the same SSD.

IOPSperformancewasmeasuredusinglibrbdfiounderaCBTtestframework;onlyoneOSDpoolwith16KPGswasusedduringthetests.Thedatasetsizewas5.8TB.

IOPS Scaling with SSDs ForrandomreadsofsmallIOsizes(4KB)3-nodeCephclusterrunningonaSamsungNVMereferenceplatformdelivers693KIOPS.AsshowninFigure13,theIOPSperformancescalesfrom394KIOPSto693KIOPSastheSSDcountisincreasedinthecluster.ForrandomIOPstests,CPUsontheOSDnodessaturatebeforeachievingtherawdriveperformance.

ForIOPS-optimizedworkloads,multipleOSDsmappedtoasingleSSDhaveshowntoimproveperformanceofsmallblockrandomIO.AsshowninFigure13,inaconfigurationwith4SSDsperOSDnode,therandomreadperformanceforsmall IOs increases as the number of OSDs mapped to each SSDincreasedfrom1to4.However,athighernumberofOSDsperOSDnodesuchmappingdoesnotalwaysdeliverhigherperformance and in some cases reduces performance due toincreasedoverheadassociatedwithcontextswitching.Asillustrated,optimalprice-performanceforsmallblockrandomIOconfigurationsisachievedwith4SSDsperOSDnode(with4OSDsmappedtoeachSSD).


Figure 10: 3-node cluster write throughput – Capacity sizing

Figure 11: 3-node cluster read throughput – NIC scaling

Figure 12: 3-node cluster write throughput – NIC scaling

Thro

ughp

ut (G

B/s)

Thro

ughp

ut (G

B/s)

Thro

ughp

ut (G

B/s)

Cluster capacity vs. 100% sequential write (IO size: 1 MB, replication factor = 2)

<NIC, SSD> vs. 100% sequential write (IO size: 1 MB, replication factor = 2)

<NIC, SSD> vs. 100% sequential write (IO size: 1 MB, replication factor = 2)


Top: SSDs per OSD node Bottom: 40 GbE ports per OSD node

Top: SSDs per OSD node Bottom: 40 GbE ports per OSD node

White Paper

17


ForrandomwritesofsmallIOsizes(4KB)3-nodeCephclusterrunningonaSamsungNVMereferenceplatformdelivers87.8KIOPS;thisaccountsforOSDjournalwritesand2xreplicationfordataprotection.AsshowninFigure14,theIOPSperformancescalesfrom34.4KIOPSto87.8KIOPSastheSSDcountisincreasedinthecluster.ForrandomIOPstests,CPUsontheOSDnodessaturatebeforeachievingtherawdriveperformance.

ForIOPS-optimizedworkloads,multipleOSDsmappedtoasingleSSDhaveshowntoimproveperformanceofsmallblockrandomIO.AsshowninFigure14,inaconfigurationwith4SSDsperOSDnode,therandomwriteperformancefor small IOs increases as the number of OSDs mapped to eachSSDincreasedfrom1to4.However,suchmappingdoesnotalwaysdeliverhigherperformanceandinsomecasesreducesperformanceduetoincreasedoverheadduetocontextswitchingathighernumberofSSDsperOSDnode.Asillustrated,optimalprice-performanceforsmallblockrandomIOconfigurationsisachievedwith4SSDsperOSDnode(with4OSDsmappedtoeachSSD).

IOPS for 8 KB IOs WhilerandomIOPSperformanceisoftenpresentedfor4KBIOsaslistedabove,databasesandotherIOintensiveapplicationsuse8KBandlargerIOsizesastransactionsizeincreases.

AsshowninFigure15andFigure16randomreadandwriteIOPSperformancefor8KBIOsis529Kand82.7Krespectively.8KBIOrandomreadperformanceis~23%lowerthan4KBIOrandomreadperformance,and8KBIOrandomwriteperformanceis~5%lowerthan4KBIOrandomwriteperformance.

Figure 13: 3-node cluster read IOPS – SSD/OSD scaling Figure 14: 3-node cluster write IOPS – SSD/OSD scaling

Figure 15: 3-node cluster read IOPS – 4 KB vs. 8 KB

<OSDs, SSD> vs. 100% random read (IO size: 4 KB, replication factor = 2)

<OSDs, SSD> vs. 100% random write (IO size: 4 KB, replication factor = 2)

100% random read: 4 KB and 8 KB (replication factor = 2)




White Paper

18

Capacity sizing SamsungPM953NVMeSSDsof960GBcapacityareusedinthereferencetestcluster.AsthenumberofSSDsareincreasedfrom4perOSDnodeto24perOSDnode,theeffectivecapacityoftheclusteraccountingfor2xreplicationincreasesfrom5.76TBto34.56TB.

Figure17showstheincreasedrandomreadIOPSforsmallIOs(4KB)toscalefrom488.9K(averageIOPSacross1to4OSDsmappedtoasingleSSD)to693K(averageIOPSwith1OSDmapped1SSD)astheclustercapacityincreases.ThedottedlineinthefigureshowsreadIOPSperGBofeffectivestoragecapacityinthecluster.Figure18showstheincreasedrandomwriteIOPSforsmallIOs(4KB)toscalefrom54.4K(averageIOPSacross1to4OSDsmappedtoasingleSSD)to87.8K(averageIOPSwith1OSDmapped1SSD)astheclustercapacityincreases.ThedottedlineinthefigureshowswriteIOPSperGBofeffectivestoragecapacityinthecluster.

Testing methodology and detailsThissectionprovidesdetailsonthetestingmethodologyusedinestablishingtheoptimizedconfigurationsinthisReferenceArchitecture.

Figure19liststhestepstestingfortheReferenceArchitectureprogressed through.

Baseline storage node performanceThe purpose of these test runs is to establish a single node raw storage IO performance of the storage media present in the node.Stepsfollowedtorunthetestsarelistedbelow:

• Each of the SSDs in the storage node is mounted using XFS and with the same options as they would be mounted in a Cephconfiguration.

o mkfs.xfs -f -i size=2048 <SSD device> o mount -t xfs <SSD device> -o noatime,inode64,lo gbsize=256k,delaylog /var/lib/ceph/osd/ceph- <osd#>

Step1: Baseline storage node performance

Step2: Network bandwidth consistency across nodes

Step3: RADOSbaseline:CBT/radosbenchover2xreplicatedpool

Step4: RBDbaseline:CBT/librbdfioover2xreplicatedpool

Figure 16: 3-node cluster write IOPS – 4 KB vs. 8 KB Figure 18: 3-node cluster write IOPS – Capacity sizing

Figure 19: Test methodology steps in Reference Architecture

Figure 17: 3-node cluster read IOPS – Capacity sizing


100% random write: 4 KB and 8 KB (replication factor = 2)

Cluster capacity vs. 100% random write (IO size: 4 KB, replication factor = 2)



Cluster capacity vs. 100% random read (IO size: 4 KB, replication factor = 2)


White Paper

19

• SSDsarepre-conditionedasappropriatetothedevice.

• FIO®testtoolisusedtoexercisethedrive;typically,with

• 4FIOjobswithaqueuedepthof32• IOSizesof4KB,8KB,128KB,and4MB• Readwriteworkloads:100%randomread,100%randomwrite

forsmallIOs(4KBand8KB),and100%sequentialread,100%sequentialwriteforlargeIOs(128KBand4MB).

• IOPS for small IO sizes, throughput for large IO sizes and latency across the tests are measured.

• Aggregatedriveperformanceacrossalldrivesinthestorage node is also measured.

Network bandwidthCeph uses a message based protocol with all participants in a Ceph cluster exchanging messages with each other across the cluster. It is critical that any potential network performance issuesareidentifiedandresolvedtoensureahighperformanceCeph cluster.

Stepsfollowedtorunthetestsarelistedbelow:

• Runbi-directionaliperftestsacrosstwoendpointsandensure that the bandwidth rates are close to link rate.

• Verifythenetworkconnectivityperformancefromeachclient to each storage node in the cluster on the public network.

• Verifythenetworkconnectivityandperformancefromeachclient to each of the monitor nodes in the cluster on the public network.

• Verifythenetworkconnectivityandperformancefromeachstorage node in the cluster to the each of the monitor nodes in the cluster on the public network.

• Verifythenetworkconnectivityandperformancefromeachstorage node in the cluster to all other storage nodes in the cluster on the cluster network.

JumboframesofMTU=9000BareenabledontheNICinterfaces in the network.

System tunablesInaddition,thefollowingNICandsystemtunablesareset:

• /etc/sysctl.conf

# System default settings live in /usr/lib/sysctl.d/00-system.conf.

# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file

#

# For more information, see sysctl.conf(5) and sysctl.d(5).

kernel.pid_max=4194303

fs.file-max=4194303

# VM Settings

vm.swappiness = 1

vm.vfs_cache_pressure = 1

# Increase Linux autotuning TCP buffer limits

# Set max to 16MB for 1GE and 32M (33554432) or 54M (56623104) for 10GE

# Don't set tcp_mem itself! Let the kernel scale it based on RAM.

# Use 128M buffers

net.core.rmem_max=268435456

net.core.wmem_max=268435456

net.core.rmem_default=67108864

net.core.wmem_default=67108864

net.core.netdev_budget=1200

net.core.optmem_max=134217728

net.ipv4.tcp_rmem=67108864 134217728 268435456

net.ipv4.tcp_wmem=67108864 134217728 268435456

net.ipv4.tcp_low_latency=1

net.ipv4.tcp_adv_win_scale=1

#

# Make room for more TIME_WAIT sockets due to more clients,

# and allow them to be reused if we run out of sockets

# Also increase the max packet backlog

net.core.somaxconn=32768

# Increase the length of the processor input queue

net.core.netdev_max_backlog=250000

net.ipv4.tcp_max_syn_backlog=30000

net.ipv4.tcp_max_tw_buckets=2000000

net.ipv4.tcp_tw_reuse=1

net.ipv4.tcp_tw_recycle=1

net.ipv4.tcp_fin_timeout=5

#

# # Disable TCP slow start on idle connections

net.ipv4.tcp_slow_start_after_idle=0

#

# # If your servers talk UDP, also up these limits

net.ipv4.udp_rmem_min=8192

net.ipv4.udp_wmem_min=8192

#

# # Disable source routing and redirects

net.ipv4.conf.all.send_redirects=0

net.ipv4.conf.all.accept_redirects=0


White Paper

20

net.ipv4.conf.all.accept_source_route=0

#

# # Recommended when jumbo frames are enabled

net.ipv4.tcp_mtu_probing=1

• Disablefirewallandstartntpdserviceinallnodes.

# rpm -qa firewalld | grep firewalld && sudo systemctl stop firewalld && sudo systemctl disable firewalld

# systemctl start ntpd.service

# systemctl enable ntpd.service

• ForallCPUsinthesystem,setcpuscalinggovernoras‘performance’.

For e.g., for CPU 0,

# echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

• ForeachNVMedevice,enablerq_affinity,

# echo 2 > /sys/block/nvme0/queue/rq_affinity

• ForeachNVMedevice,enableread_ahead,

For radosbench tests, set read_ahead value to 128,

# echo 128 > /sys/block/nvme0/queue/read_ahead_kb

For librbdfio tests, set read_ahead to 0,

# echo 0 > /sys/block/nvme0/queue/read_ahead_kb

CBT testsCBTframeworkprovidesradosbenchandlibrbdfiothatwillbeused to perform benchmark runs on the Ceph cluster. Make sure size of /tmp is adequate to hold the results of CBT tests. Foreachtestlikeread,random-read,random-write,etc.,theresultsmaytakeupto40MBofdrivespace.

radosbench is used for sequential workloads for large IO sizes andlibrbdfioisusedforrandomworkloadsforsmallIOsizes.

Appendix10.1presentsceph.conffilewiththeCephconfigurationtunablesusedintheconfiguration.

Appendix10.2presentsCBTtest.yamlfilewiththetestconfigurationsusedinthetestruns.

SummarySamsungNVMeReferenceDesignprovidesahighlyflexiblehigh-performanceplatformthatcanbeconfiguredforservingboth high IOPS and high throughput Ceph cluster deployments usingSamsungNVMeSSDs.CombinedwithCephflexibility,ascalableCephclustercanbebuilttodeliveranoperationalclusterthatcandeliverpernode:

• Upto11.52TBofeffectivecapacityaccountingfor2xreplication

• Upto231K100%randomreadof4KBIOs

• Upto29.2K100%randomwriteof4KBIOsaccountingforOSDjournaland2xreplicationwrites

• Upto9.5GB/s100%sequentialreadof1MBIOs

• Upto2.06GB/s100%sequentialwriteof1MBIOsaccountingforOSDjournaland2xreplicationwrites


White Paper

21

AppendixReference ceph. conf[global]

auth_cluster_required = none

auth_service_required = none

auth_client_required = none

crushtool = /usr/local/bin/crushtool

debug_lockdep = 0/1

debug_context = 0/1

debug_crush = 1/1

debug_buffer = 0/1

debug_timer = 0/0

debug_filer = 0/1

debug_objecter = 0/1

debug_rados = 0/5

debug_rbd = 0/5

debug_ms = 0/5

debug_monc = 0/5

debug_tp = 0/5

debug_auth = 1/5

debug_finisher = 1/5

debug_heartbeatmap = 1/5

debug_perfcounter = 1/5

debug_rgw = 1/5

debug_asok = 1/5

debug_throttle = 1/1

debug_journaler = 0/0

debug_objectcatcher = 0/0

debug_client = 0/0

debug_osd = 0/0

debug_optracker = 0/0

debug_objclass = 0/0

debug_filestore = 0/0

debug_journal = 0/0

debug_mon = 0/0

debug_paxos = 0/0

osd_crush_chooseleaf_type = 0

filestore_xattr_use_omap = true

osd_pool_default_size = 1

osd_pool_default_min_size = 1

mon_pg_warn_max_object_skew = 10000

mon_pg_warn_min_per_osd = 0

mon_pg_warn_max_per_osd = 32768

rbd_cache = true

mon_compact_on_trim = false

log_to_syslog = false

log_file = /var/log/ceph/$name.log

perf = true

mutex_perf_counter = true

throttler_perf_counter = false

ms_nocrc = true

[mon]

mon data = /tmp/cbt/ceph/mon.$id

mon_max_pool_pg_num = 166496

mon_osd_max_split_count = 10000

[mon.a]

host = mon-host-1

mon_addr = 40.10.10.150:6789

[client]

rbd_cache = true

rbd_cache_writethrough_until_flush = false

admin_socket = /var/run/ceph/$cluster-

$type.$id.$pid.$cctid.asok

log_file = /var/log/ceph/

[osd]

filestore_wbthrottle_enable = false

filestore_queue_max_bytes = 1048576000

filestore_queue_committing_max_bytes =

1048576000

filestore_queue_max_ops = 5000 filestore_queue_committing_max_ops = 5000

filestore_max_sync_interval = 10

filestore_fd_cache_size = 64

filestore_fd_cache_shards = 32

filestore_op_threads = 6

filestore flusher = false

osd_crush_update_on_start = false

osd_max_backfills = 1

osd_recovery_priority = 1

osd_client_op_priority = 63

osd_recovery_max_active = 1

osd_recovery_max_start = 1

osd_mount_options_xfs = "rw,noatime,inode64,

logbsize=256k,delaylog"


White Paper

22

osd_mkfs_options_xfs = "-f -i size=2048"

journal_max_write_entries = 2000

journal_queue_max_ops = 3000

journal_max_write_bytes = 4194304000

journal_queue_max_bytes = 4194304000

osd_enable_op_tracker = false

osd_client_message_size_cap = 0

osd_client_message_cap = 0

objecter_inflight_ops = 102400

objecter_inflight_op_bytes = 1048576000

ms_dispatch_throttle_bytes = 1048576000

osd_op_threads = 32

osd_op_num_shards = 5

osd_op_num_threads_per_shard = 2

[osd.0]

host = osd-host-1

public_addr = 42.10.10.111

cluster_addr = 42.10.10.111

osd_data = /tmp/cbt/mnt/osd-device-0-data

osd_journal = /dev/disk/by-partlabel/osd-device-0-journal

ms_bind_port_min = 7000

ms_bind_port_max = 8000

[osd.1]

host = osd-host-1

public_addr = 42.10.10.111



osd_journal = /dev/disk/by-partlabel/osd-

device-1-journal



[osd.2]

host = osd-host-1

public_addr = 42.10.10.111




device-2-journal



[osd.3]

host = osd-host-1

public_addr = 42.10.10.111




device-3-journal



[osd.4]

host = osd-host-1

public_addr = 42.10.10.111




device-4-journal



[osd.5]

host = osd-host-1

public_addr = 42.10.10.111




device-5-journal



[osd.6]

host = osd-host-1

public_addr = 43.10.10.111




device-6-journal



[osd.7]

host = osd-host-1

public_addr = 43.10.10.111



White Paper

23





[osd.8]

host = osd-host-1

public_addr = 43.10.10.111






[osd.9]

host = osd-host-1

public_addr = 43.10.10.111




device-9-journal



[osd.10]

host = osd-host-1

public_addr = 43.10.10.111






[osd.11]

host = osd-host-1

public_addr = 43.10.10.111






[osd.12]

host = osd-host-1

public_addr = 40.10.10.111






[osd.13]

host = osd-host-1

public_addr = 40.10.10.111






[osd.14]

host = osd-host-1

public_addr = 40.10.10.111






[osd.15]

host = osd-host-1

public_addr = 40.10.10.111






[osd.16]

host = osd-host-1

public_addr = 40.10.10.111






[osd.17]

host = osd-host-1


White Paper

24

public_addr = 40.10.10.111






[osd.18]

host = osd-host-1

public_addr = 41.10.10.111






[osd.19]

host = osd-host-1

public_addr = 41.10.10.111






[osd.20]

host = osd-host-1

public_addr = 41.10.10.111






[osd.21]

host = osd-host-1

public_addr = 41.10.10.111






[osd.22]

host = osd-host-1

public_addr = 41.10.10.111






[osd.23]

host = osd-host-1

public_addr = 41.10.10.111






[osd.24]

host = osd-host-2

public_addr = 42.10.10.112






[osd.25]

host = osd-host-2

public_addr = 42.10.10.112






[osd.26]

host = osd-host-2

public_addr = 42.10.10.112







White Paper

25

[osd.27]

host = osd-host-2

public_addr = 42.10.10.112






[osd.28]

host = osd-host-2

public_addr = 42.10.10.112






[osd.29]

host = osd-host-2

public_addr = 42.10.10.112






[osd.30]

host = osd-host-2

public_addr = 43.10.10.112






[osd.31]

host = osd-host-2

public_addr = 43.10.10.112






[osd.32]

host = osd-host-2

public_addr = 43.10.10.112






[osd.33]

host = osd-host-2

public_addr = 43.10.10.112






[osd.34]

host = osd-host-2

public_addr = 43.10.10.112






[osd.35]

host = osd-host-2

public_addr = 43.10.10.112






[osd.36]

host = osd-host-2

public_addr = 40.10.10.112





White Paper

26

device-12-journal



[osd.37]

host = osd-host-2

public_addr = 40.10.10.112






[osd.38]

host = osd-host-2

public_addr = 40.10.10.112






[osd.39]

host = osd-host-2

public_addr = 40.10.10.112






[osd.40]

host = osd-host-2

public_addr = 40.10.10.112






[osd.41]

host = osd-host-2

public_addr = 40.10.10.112






[osd.42]

host = osd-host-2

public_addr = 41.10.10.112






[osd.43]

host = osd-host-2

public_addr = 41.10.10.112






[osd.44]

host = osd-host-2

public_addr = 41.10.10.112






[osd.45]

host = osd-host-2

public_addr = 41.10.10.112






[osd.46]

host = osd-host-2


White Paper

27

public_addr = 41.10.10.112






[osd.47]

host = osd-host-2

public_addr = 41.10.10.112






[osd.48]

host = osd-host-3

public_addr = 42.10.10.115






[osd.49]

host = osd-host-3

public_addr = 42.10.10.115






[osd.50]

host = osd-host-3

public_addr = 42.10.10.115






[osd.51]

host = osd-host-3

public_addr = 42.10.10.115






[osd.52]

host = osd-host-3

public_addr = 42.10.10.115






[osd.53]

host = osd-host-3

public_addr = 42.10.10.115






[osd.54]

host = osd-host-3

public_addr = 43.10.10.115






[osd.55]

host = osd-host-3

public_addr = 43.10.10.115






White Paper

28


[osd.56]

host = osd-host-3

public_addr = 43.10.10.115






[osd.57]

host = osd-host-3

public_addr = 43.10.10.115






[osd.58]

host = osd-host-3

public_addr = 43.10.10.115






[osd.59]

host = osd-host-3

public_addr = 43.10.10.115






[osd.60]

host = osd-host-3

public_addr = 40.10.10.115




device-12-journal



[osd.61]

host = osd-host-3

public_addr = 40.10.10.115






[osd.62]

host = osd-host-3

public_addr = 40.10.10.115






[osd.63]

host = osd-host-3

public_addr = 40.10.10.115






[osd.64]

host = osd-host-3

public_addr = 40.10.10.115






[osd.65]

host = osd-host-3

public_addr = 40.10.10.115



White Paper

29





[osd.66]

host = osd-host-3

public_addr = 41.10.10.115






[osd.67]

host = osd-host-3

public_addr = 41.10.10.115






[osd.68]

host = osd-host-3

public_addr = 41.10.10.115






[osd.69]

host = osd-host-3

public_addr = 41.10.10.115






[osd.70]

host = osd-host-3

public_addr = 41.10.10.115






[osd.71]

host = osd-host-3

public_addr = 41.10.10.115






Sample CBT test.yaml filecluster:

user: 'root'

head: "cbt-host-1"

clients: ["cbt-client-1","cbt-client-2","cbt-client-3","cbt-client-4","cbt-client-5","cbt-client-6","cbt-client-7","cbt-client-8","cbt-client-9"]

osds: ["osd-host-1", "osd-host-2", "osd-host-3"]

mons:

mon-host-1:

a: "40.10.10.150:6789"

osds_per_node: 24

fs: 'xfs'

mkfs_opts: '-f -i size=2048 -n size=64k'

mount_opts: '-o inode64,noatime,nodiratime,logbufs=8,logbsize=256k'

conf_file: '/usr/local/ceph-cbt/ceph.conf'

iterations: 1

use_existing: False

clusterid: "ceph"

tmp_dir: "/tmp/cbt"

pool_profiles:

rbd:

pg_size: 16384

pgp_size: 16384

replication: 2

radosbench:

pg_size: 8192

pgp_size: 8192


White Paper

30

replication: 2

benchmarks:

librbdfio:

time: 300

ramp: 100

vol_size: 65536

mode: ['randread', 'randwrite', 'randrw']

rwmixread: [70]

numjobs: 4

use_existing_volumes: False

procs_per_volume: [1]

volumes_per_client: [10]

op_size: [4096, 8192, 16384]

concurrent_procs: [1]

iodepth: [768]

osd_ra: [9000]

cmd_path: '/usr/local/bin/fio'

log_avg_msec: 20000

pool_profile: 'rbd'

radosbench:

op_size: [131072, 4194304, 1048576]

write_only: False

time: 300

pool_per_proc: True

concurrent_procs: 4

concurrent_ops: [ 32, 64, 128 ]

osd_ra: [1108]

pool_profile: 'radosbench'


White Paper

31

Bill of materialsTable12,Table13,andTable14presentthehardwareconfigurationsusedinthisCephReferenceArchitecture.

Samsung NVMe SSD PM953





Upto870MB/s



Upto19KIOPS

DWPD 1.3DWPD Production status

Mass Production

Samsung NVMe reference platform


2x Intel® Xeon® CPU E5-2699 v3 @ 2.30 GHz

With18corespersocket,thiswillhave36corespersystem. With HT on, OS will list 72 hardware threads.

256 GB Toavoidmemorylimitationsinusinghigh-performanceSSDs and NICs.

Up to 24x PM953

Eachdrivewith960GBrawcapacity.

Mellanox ConnectX®-4 EN adapters

1xto4x40GbEports;toavoidanynetworkbottlenecksindeliveringhighIOPSinaclusterwithreplicationorrecoverytraffic.

Client and monitor nodes


2 x Intel® Xeon® CPU E5-2670 v3 @ 2.30 GHz

With12corespersocket,thiswillhave24corespersystem.WithHTon,OSwilllist48hardwarethreads.

128 GB Toavoidmemorylimitationsinusinghigh-performanceNICs.

Mellanox ConnectX®-3 EN adapters

2x40GbEport

Table 12: Samsung NVMe SSD PM953 960 GB

Table 13: Samsung NVMe Reference Platform (OSD node) configuration

Table 14: Generic dual-socket x86 Server (client and monitor nodes) configuration


White Paper

32

About Samsung Electronics Co., Ltd.Samsung Electronics Co., Ltd. inspires the world and shapes thefuturewithtransformativeideasandtechnologies.ThecompanyisredefiningtheworldsofTVs,smartphones,wearabledevices,tablets,cameras,digitalappliances,printers,medical equipment, network systems, and semiconductor and LEDsolutions.Forthelatestnews,pleasevisittheSamsungNewsroom at news.samsung.com.

About Red HatRedHatistheworld’sleadingproviderofopensourcesoftwaresolutions,usingacommunitypoweredapproachtoprovidereliableandhigh-performingcloud,Linux,middleware,storage,andvirtualizationtechnologies.RedHatalsooffersaward-winningsupport,training,andconsultingservices.Asaconnectivehubinaglobalnetworkofenterprises,partners,andopensourcecommunities,RedHathelpscreaterelevant,innovativetechnologiesthatliberateresourcesforgrowthandprepare customers for the future of IT.

For more informationFor more information about the Samsung NVMe Reference Design,pleasevisitatsamsung.com/semiconductor/afard/.

Copyright©2016SamsungElectronicsCo.,Ltd.Allrightsreserved. SamsungandAutoCachearetrademarksorregisteredtrademarksofSamsungElectronicsCo.,Ltd.Specificationsanddesignsaresubjecttochangewithoutnotice.Nonmetricweightsandmeasurementsareapproximate.Alldatawere deemed correct at time of creation. Samsung is not liable for errors or omissions.Allbrand,product,servicenamesandlogosaretrademarksand/orregisteredtrademarksoftheirrespectiveownersandareherebyrecognizedandacknowledged.

CinderisatrademarkofCinderAlpha,Inc.

Fio is a registered trademark of Fio Corporation.

IntelandXeonaretrademarksofIntelCorporationintheU.S.and/orothercountries.

LinuxisaregisteredtrademarkofLinusTorvalds.

TheMariaDBtrademarkiswhollyownedbyMariaDBCorporationAbandisaregisteredtrademarkintheUnitedStatesofAmericaandothercountries.

Mellanox and ConnectX are registered trademarks of Mellanox Technologies, Ltd. Mellanox PeerDirect is a trademark of Mellanox Technologies, Ltd.

MySQLisaregisteredtrademarkofOracleand/oritsaffiliates.

The OpenStack Word Mark and the OpenStack logos are trademarks of the OpenStack Foundation.

PostgreSQLisaregisteredtrademarkofthePostgreSQLCommunityAssociationof Canada.

Red Hat, Inc., Red Hat, Red Hat Enterprise Linux, are trademarks or registered trademarksofRedHat,Inc.oritssubsidiariesintheUnitedStatesandothercountries.

StackVelocityisaregisteredtrademarkofJabilCircuit,Inc.

Samsung Electronics Co., Ltd.

129Samsung-ro,

Yeongtong-gu,

Suwon-si,Gyeonggi-do16677,

Korea

www.samsung.com

2016-10

Legal and additional information

http://news.samsung.com

http://www.samsung.com/semiconductor/afard/

http://www.samsung.com

Documents

High-performance cluster storage for IOPS-intensive workloads · Red Hat Ceph Storage on Samsung NVMe SSDs The Reference Architecture configurations described inthis document are