Embed Size (px)
Citation preview
1
UnleashthePowerofPetabyteObject
StoragewithStorReduceandActiveScaleÔ
StorReducehigh-speeddeduplicationbringsscale,simplicity,and
dramaticallylowercoststohybridandprivatecloudbackup.
March2018
2
TableofContentsIntroduction.................................................................................................................................................3
ActiveScalewithStorReduceSoftware....................................................................................................4
StorReduceOverview...................................................................................................................................4
StorReduceArchitecture..............................................................................................................................7
S3ClientSoftware....................................................................................................................................8
CIFS/NFSClientSoftware.....................................................................................................................8
StorReduceServers..................................................................................................................................8
LocalSSDStorage.....................................................................................................................................9
ObjectStorage........................................................................................................................................10
Performance...............................................................................................................................................10
StorReduceDemonstrationCluster........................................................................................................11
UsingMultipleStorReduceServers............................................................................................................12
Scale-outClusters...................................................................................................................................12
MakingDataAvailableinMultipleLocationswithRead-OnlyEndpoints..............................................13
CreatingMultipleCopiesoftheDatawithDataReplication..................................................................13
ObjectClone...............................................................................................................................................14
Security.......................................................................................................................................................15
UseCases...................................................................................................................................................15
UsingStorReduceforPrimaryBackupsStraighttoObjectStorage:RemovingPurpose-BuiltBackupAppliances..............................................................................................................................................16
Backup/TapeMigrationtoCloud...........................................................................................................16
CloningLargeDataSetsonCloudorinObjectStores............................................................................17
ReplacingTraditionalStorageSolutionswithObjectStorage................................................................18
On-CloudDeduplicationforBackups.....................................................................................................18
MovingorReplicatingDataBetweenClouds.........................................................................................19
OptimizingBigData/HadoopStorage....................................................................................................20
IsYourStorageKeepingUpwithYourBusiness?.......................................................................................20
3
IntroductionNaturaldisasters.Securitybreaches.Fatfingers.Catastrophicdisruptionstocriticalbusinessservicescancomeinmanyforms.That’swhy,whetherduetoregulatoryrequirementsorjustcommonsense,everyresponsibleenterprisethinkslongandhardaboutdataprotection.Unfortunately,thecurrentapproachesonwhichmanyorganizationsrely—storingprimarybackupsonpurpose-builtbackupappliancesandusingmagnetictapeforlong-termretention—arefarfromideal.
Conventionalbackupappliancescarryhighpricetags.And,asorganizationsstoremoreandmoredata,theyconstantlyneedtoaddappliances—lockingthemselvesintoanever-endingcapitalinvestmentcyclewithever-growingpower,cooling,andmanagementcosts.Theseappliancesalsotypicallybecome“hardwaresilos,”offeringlimitedflexibilitytomoveandaccessstoreddata.Evenworse:theydon’tfullyaddresstheproblemthey’reintendedtosolve.Anill-timedsystemoutagecaneasilyleadtodataloss,underminingtherationaleforinvestinginbackupappliancesinthefirstplace.
Meanwhile,tapebackups,whiletheoreticallylessexpensive,carrytheirownhiddencostsintheongoingneedtohandle,transport,andstorephysicaltapes.They’revulnerabletohumanerror—likebeinglost,stolen,oraccidentallyrewritten.And,intheeventanorganizationneedstousethemtorecoverfromanoutage,theycanexpectslowrestoretimesandsignificantdowntimewhiletheywait.
Now,there’sabetterwaytostoremassiveamountsofbackupdata.Bycombiningadvanceddeduplicationwithhigh-densityobjectstorage,youcanslashdataprotectioncapitalandoperatingcostsandbetterprotectyourbusiness.Andyoucangetridofbackupappliancesandtapeforever.
4
ActiveScalewithStorReduceSoftwareBycombiningtwopowerfultechnologies,ActiveScale™objectstorageandStorReducededuplication,organizationscannoweasilyandaffordablytransitionfrompurpose-builtbackupappliancesandtapetohigh-performanceclouddataprotection.
ActiveScalebuildsonaheritageofinnovationthatgoesbacktotheworld’sfirstharddiskdrive,providingahigh-densityobjectstoragesolutiondesignedforlargeenterpriseshandlingmassiveamountsofdataforlongperiods.DeployedwithStorReduceinlinededuplicationsoftware,organizationscanquicklymovehugevolumesofdatatoActiveScaleprivateorhybridcloud,eliminatingredundantdataalongtheway.
WithStorReducescale-outdeduplicationsoftware,youcanimplementcloud-basedobjectstorageforprimaryandsecondarybackupsatpetabytescales.And,youcanslashstoragecostsbyupto70%byreplacingbackupappliancesandtapewithprivateorhybridcloud.UsingStorReducewithActiveScalegivesyoubest-of-breedscalability,durability,andrecoverabilityforyourdata.Atthesametime,youcannowactuallyusethatbackupdataforsearching,analytics,andevenmachinelearningviastandardcloudAPIs.
ThispaperprovidesatechnicaloverviewoftheStorReducesolution.Ithighlightskeyfeaturesandcapabilities,anddetailstheStorReducearchitecture.AnditpresentsseveralusecasestoillustratehoworganizationscanuseStorReducewithActiveScaleobjectstoragetorealizeconcretebenefits.(FormoreinformationonActiveScaleitself,visithttps://www.hgst.com/storreduce)
StorReduceOverviewStorReduceisaspecializedclouddeduplicationsolutiondesignedtomeettheuniquerequirementsofcompaniesusingobjectstorage(publiccloudorprivateobjectstore,orahybridcombination)forhugevolumesofdata.StorReducesitsbetweenyourapplicationsandActiveScaleobjectstorage,transparentlyperforminginlinedatadeduplication.Itreducesstorageandbandwidthrequirementsbyasmuchas97%andcanreducedatatransfertimesbyjustasmuch.
StorReduceisfastandmassivelyscalable.Whenusingmultipleserversdeployedasascale-outcluster,itcanachievesustainedwritespeedsof10Gigabytespersecondorhigherandmanagehundredsofpetabytesofdata.Alldataismanagedinasinglededuplicationpool.And,datadeduplicatedwithStorReducecanalwaysbeaccessedon-cloudoron-premisesusingtheAmazonSimpleStorageService™(S3)RESTAPI.
5
DIAGRAM1:STORREDUCEARCHITECTURE
Unlikeotherdatabackupanddeduplicationsolutions,StorReducedelivers:
• Massivescalabilitywithanarchitecturedesignedforhighthroughputandlowlatencyatpetabytescale
• Upto70%storagecostsavingsbymigratingdataoffexpensive,complexbackupappliancesandlegacytape
• Improvedresiliencywithasolutionthatrunsoncommodityhardwarewithnosinglepointoffailure
• Fasterperformancethroughinlinededuplicationthatremovestheriskofdatalosswithoutslowingbackuptimes
• Nomorestoragesiloswithasinglenamespaceacrossyourhybridoron-premisescloudstorage• Abilitytoactuallyusemigratedbackupdatabylookinginsidebackuparchivefiles,and
providingacloud-nativeinterfacethatenablesyoutouseserviceslikeSearch(tosearchinsideallmigratedbackups)anddatamining
• Increasedflexibilitywiththeabilitytoreplicatededuplicateddatabetweencloudregionsandcloud-basedinfrastructures,orbetweencloud-basedandon-premisesenvironments
6
• Abigdata-readysolutionthatoffersscalabilityforlargedatasetsandveryfastthroughput,makingitidealforcloud-hostedworkloadsthatnowrequirelesscloudstorageandcanutilizeHadoopandservices
• Highavailabilityviaclustering,withtheabilitytodeployclustersinmultipleRegionstoensurethatdataaccessismaintainedevenintheunlikelyeventofacompletefailureofacloud-baseddatacenter
• Morevaluefromyourbackupdata,withtheabilitytocloneitontoActiveScaleobjectstorage,whereitcanbeusedforadditionalworkloadssuchasdev/testandcompliance
Additionally,StorReducedeliversthefollowingkeybenefits:
• FastSustainedThroughput:Achievewritesspeedsof10gigabytespersecondormoreusingascale-outcluster,whileaddinglessthan50msoflatency.EachindividualStorReduceservercandeduplicatedataatspeedsof2gigabytespersecondorfaster.And,StorReducecandeliversustainedthroughput24/7—unlikesomesolutions,itdoesnotstagedatainbuffersthatcanfillup.
• Cloud-nativeStorage:MakealldeduplicateddataimmediatelyaccessibletocloudservicesviaStorReduce’sS3RESTAPI.Storedataonallmajorpubliccloudproviders—AmazonS3orS3IA,MicrosoftAzureBlobStorage,orGoogleCloudStorage.
• Resiliencybydesign:Thesystemstoresindexinformationrequiredtore-hydratethedatainmultiplelocations,bothinobjectstorageandoneachStorReduceserver.Extensiveuseofchecksumsandhashesensuresdataintegrityduringbothdeduplicationandre-hydration.
AlloftheseadvantagesderivefromStorReduce’suniquearchitecture—designedfromthegroundupforperformance,scale,andflexibility.KeyStorReducearchitecturalfeaturesinclude:
• Scale-outClusters:IncreasedatastoragecapacityandwritethroughputalmostlinearlybysimplyaddingmoreStorReduceserverstoacluster.Atthesametime,youmaintainasinglededuplicationpoolandasinglenamespaceforthedata.
• BackupSoftwareIntegration/DataManagement:StorReduceworkswithanyexistingdatamanagementorbackupsoftwarethatiscompatiblewithAmazonS3,includingVeritasNetBackup,BackupExec,Commvault,andEMCNetworker.Veeamissupportedvia“gateways”untiltheyprovidenativeS3support.
• ObjectClone:StorReducecancreatewritable“virtualclones”ofallthedatainastoragebucket,withoutusingadditionalstoragespace.Createunlimitedvirtualclonesfortestingordistributionofdatatodifferentteamsinanorganization.Copy-on-writesemanticsalloweachindividualcloneofthedatatodivergeasnewversionsofindividualobjectsarewritten.
• Read-OnlyEndpoints:YoucandeployadditionalStorReduceserversorclusterson-cloudoron-premisesasread-onlyendpoints.Uploadeddataisonlystoredoncebutisimmediatelyavailableateachendpointlocation.Withthiscapability,yourmigratedbackupworkloadsarenolongerjustacostcenter;youcannowrepurposethemonthecloudfordevelopment,test,QA,disasterrecovery,andforusebyothercloud-basedservices.
7
• DataReplication:StorReducecanreplicatedatabetweenregions,betweencloudvendors,orbetweenpublicandprivatecloudtoincreasedataresiliency.Thesystemonlytransfersuniquedata,providingupto30timesfastertransmissionandupto30timeslowerbandwidthandstoragecosts.
• HighAvailability:Whenconfiguredasascale-outclusterStorReduceprovidesautomaticfailoverintheeventofserverfailure.Youcansetupthesystemwithanydegreeofredundancyyouchoosebetweenservers.
• WriteSpeedThrottling:Evenmassivedatabackupdoesn’thavetoimpedetheperformanceofotherbusinessapplications.SetamaximumwritespeedtopreventStorReducefromusingtoomuchbandwidthwhensharinganInternetconnectionwithotherinfrastructure.
StorReduceisalsobuilttoensurethatsensitivedatastaysprotected—inmotionandatrest.Keysecurityfeaturesinclude:
• Encryption:StorReducesupportsencryptionofdatabeforeitlandsintheobjectstore.ItintegrateswithkeymanagementsystemslikeAmazonKMSorKMIP-compatiblehardwaresecuritymodulesforstorageofthecryptographickeys.Datacanbeencryptedon-premisesbeforebeingsenttothecloud,oryoucanusecloudencryption-at-restservices.StorReducealwaysencryptsdatain-transit.
• SecureUserAccountandKeyManagement:YoucanassignusersorserversindividualuseraccountswithinStorReducetorestrictdataaccess.Multipleaccesskeyscanbecreatedandmanagedasneededforeachuseraccount.
• SecurePolicy-basedAccessControl:YoucanexpressenterprisesecuritypoliciesusingtheStorReducepolicyengine,usingAmazon’sIdentity&AccessManagement(IAM)policylanguage.
StorReduceArchitectureFromanarchitecturalperspective,StorReduceactsasanintelligent,transparentlayerbetweenyourclientapplicationsandActiveScaleobjectstorageandpubliccloud,enablingapplicationstouseyourstoragemoreefficientlyandcost-effectively.(SeeDiagram1.)Itoperatestransparentlytoapplications,allowingthemtostoreandretrievedataastheynormallywould,butwithbest-in-classdatadeduplicationandhighthroughput.
StorReducecanbedeployedaseitherasingleserverorascale-outclusterofservers.Asingleservercanbeexpandedtobecomeaclusterbyaddingmoreserversatanytime.Ineithercase,clientapplicationsseeasinglelogicalserver.
StorReduceserversoftwarecanbedeployedinthepubliccloud(suchasanAmazonEC2instance),oron-premisesinavirtualmachine(VM)orDockercontainer,orinstallednativelyonLinux.
8
TheStorReduceserverprovidessimilarfunctionalitytoobjectstoreorcloudstoragesolutions,includingobjectmanagement,useraccounts,accesskeys,accesscontrolpoliciesandaWeb-basedmanagementinterface(theStorReduceDashboard).
ThefollowingsectionsdescribeStorReducearchitecturalcomponentsandfeaturesinmoredetail.
S3ClientSoftwareStorReduceisdesignedtobeeasytoimplementinyourexistingdatabackupinfrastructure,withouthavingtoreengineerthewayapplicationsinteractwithstorage.ItworkswithclientsoftwarethatsupportsAmazon’sS3RESTinterfaceforobjectstorage.ThisincludesclientsdesignedtoworkwithAmazonS3,aswellasthosedesignedtoworkwithGoogleCloudStorageviatheXMLAPI(whichisS3-compatible).Foryourapplications,theonlydifferencefromconventionalS3objectstorageisthatclientsoftwareisconfiguredtotalktoStorReduceinsteadofdirectlytoobjectstorage,usingaccesskeysprovidedbytheStorReduceserver.
PracticallyanyS3clientsoftwareusedinconventionalobjectstorageenvironmentscanbeusedwithStorReduce.Thisincludeson-premisesbackupsoftware(includingVeritasNetBackup)aswellascustomsoftwarewrittentousetheS3RESTinterface.Youcanalsouseanycloud-basedservicesdesignedtoworkwithAmazonS3orGoogleCloudStoragewithStorReduce,asthesealsoactasS3clients.
StorReduceautomaticallytranslatesS3clientrequestsintowhicheverprotocolisneededbytheunderlyingobjectstorageproviders—includingMicrosoftAzure.ItprovidesanS3-compatibleinterfaceontoAzureBlobStorage,allowingbackupproductsandothersoftwaredesignedtoworkwithS3tobeusedwithAzureBlobStoragewithoutmodification.
CIFS/NFSClientSoftwareStorReducealsosupportsfile-sharinginterfacessuchasCommonInternetFileSystem(CIFS)andNetworkFileSystem(NFS)viagatewaysoftwareandappliances.Inthesescenarios,thegatewayexposesafileshareinterfaceandconvertsrequestsintocallstoAmazon’sS3protocol.
StorReduceServersEachStorReduceserverrunsonitsownphysicalorvirtualmachine,withlocalSSDstoragerecommended.EachStorReduceservercanhandletensofpetabytes(tensofmillionsofGigabytes)ofrawdata,dependingonthededuplicationratioachievedandtheamountofSSDstorageavailableforindexinformation.(Seethesection“LocalSSD”below.)
WithStorReduce,youcancreatemultiplestoragebucketsandperformglobaldeduplicationacrossallofthem.
Whenusingpubliccloud,theStorReduceserverrunsonanAmazonEC2instance,AzureVirtualMachine,orGoogleComputeEngine.VMimagesareavailablethroughtheAWSMarketplace,AzureMarketplace,andGoogleCloudMarketplacewiththeserverpre-installed,allowingquickandeasysetup.YoucanalsoeasilyinstallStorReduceonanyLinuxvirtualmachineusingRedHatPackageManager(RPM).
9
StorReducealsosupportson-premisesdeployments.Formigrationofon-premisesdatatothecloud,orforprivateobjectstoredeployments,theStorReduceservercanberunon-premisesonaphysicalorvirtualmachine,orunderDocker.Pre-builtvirtualappliance(OVAfile)andDockercontainerimagesareavailable.
StorReduceisdesignedfromthegroundupforresiliencyandflexibility.Thearchitectureallowsmultiple
StorReduceserverstoberunagainstthesameback-endobjectstorageservice,forredundancy,load-sharing,andincreasedstoragevolume.ThiscapabilityformsthebasisforStorReducescale-outclusters,aswellasallowingmultipleserverstoactasendpointsprovidingaccesstothesamedatafromdifferentlocations.Forexample,anon-premisesStorReduceclustermightbeusedtodeduplicateanduploadbackupdata,withasecondin-cloudStorReduceclusterprovidingimmediateaccesstothisdataforcloudservicesasthedataisuploaded.
StorReduceserversfeaturetwointerfaces:
• S3Interface:TheStorReduceserverexposesanS3-compatibleRESTinterfaceforobjectstorage.ThishighlyscalableinterfacesupportsmostS3interfacecalls,including:
o ObjectGET/PUT/POST/DELETE(includingmultiple-objectdelete)o Multipartuploads(includinglistinganddeletinguploads)o Digitalsignatureverificationo Bucketcreate/delete/renameo Setting/readingbucketpoliciesforaccesscontrol.
• AdminInterface:StorReduceserversexposeaseparateRESTinterfaceforusebytheWeb-
baseddashboard.ThisadminAPIisservedonaseparateporttoallowfirewallstorestrictnetwork-levelaccess,andcanbeservedoverHTTPS.TheadminAPIisavailableforusebyotherclientapplicationsaswellastheStorReducedashboard,andsupportsmanipulationofuseraccounts,accesspoliciesandindexsnapshots,aswellasprovidingareplicaoftheS3APIforusebymanagementtools.
LocalSSDIndexStorageStorReducestoresindexinformationonlocalSSDstorage.TheamountofrawdataaStorReduceservercanhandledependsontheamountoffastlocalstorageavailableandthededuplicationratioachievedforthedata.
TheuseoflocalSSDstorageratherthanRAMforindexinformationallowstensofpetabytesofdatatobemanagedbyasingleStorReduceserver.Typically,theamountofSSDstoragerequiredislessthan0.2%oftheamountofdataputthroughStorReduce,forastandaloneserver.
TheStorReduceservertreatslocalSSDstorageasephemeral.Allinformationstoredinlocalstorageisalsosenttoobjectstorageandcanberecoveredlaterifrequired(seelatersection).
10
ObjectStorageTheStorReduceserverusesActiveScaleobjectstorageinprivateorhybridcloudscenariosforallpersistentdata.StorReduceactsasanS3client,makinguseoftheobjectstorageAPItostoreallitsdatainasingleaccount.StorReduceisarchitectedtoworkwithanyS3-compatiblepublicobjectstoragesolution,includingAmazonS3,aswellaswithAzureBlobStorage.
TheStorReduceservermakesuseofObjectStoragetostorethefollowingtypesofdata:
• Deduplicateduserdata:Rawdataisdeduplicatedusingstate-of-the-artalgorithmsandthencompressed.Typically,thisrequiresaslittleas3%oftheobjectstoragespacethattherawdatawouldhaverequired,dependingonthetypeofdatastored.
• SystemData:Informationaboutbuckets,users,accesscontrolpolicies,andaccesskeysisalsostoredinbackendobjectstorage,makingitavailabletoallStorReduceserversinagivendeployment.
• Indexsnapshots:Dataforrapidlyreconstructingindexinformationonlocalstoragecanalsobestoredinbackendobjectstorage.
PerformanceStorReduceistheproductofgroundbreakingcloudstorageinnovationsthatdeliverunprecedentedscalability,highthroughput,andlowlatency.Theinternalarchitectureandcodearefinelytunedfordatadeduplicationandoptimizedtoensuresuperiorperformanceevenwhenrunninginapubliccloudenvironment.
AsingleStorReduceserveriscapableofsustainedwritespeedsof2Gigabytespersecondormore,dependingonthededuplicationratio,CPUcoresavailable,andnetworkconnectionavailable.Thesereal-worldspeedshavebeenachievedonAmazonEC2instancesandwithincustomerdatacentersusingtheStorReducevirtualappliance.
AStorReduceclusteriscapableofthroughputthatscalesalmostlinearlywiththenumberofserversinthecluster.Distributingtheloadoverallserversintheclusters,thesystemcanachievethroughputsof10Gigabytespersecondormore,whilemaintainingasinglededuplicationpool.BydeployingaStorReduceclusterinfrontofanActiveScaleprivateobjectstoreinstallation,youcandramaticallyreduceyourstoragerequirementswhilemaintainingveryhighthroughputrates.
Thebestwaytosubstantiallyspeedupthroughputanddecreasetransferbandwidthtocloud-basedstorageistorunaStorReduceserverorclusteron-premises,deduplicatingdatapriortosendingitintothecloud,andreadingdeduplicateddatafromthecloudandreconstitutingitlocally.BytakingadvantageofStorReduce’sfastthroughput,youcanreducemigrationtimesfromyearstoweeksandgreatlyreducemigrationcosts.(ForadirectindependentcomparisonofStorReducevs.awell-knownmigrationvendor,visithttp://www.storreduce.com/case-studies/apn-spectrumdata/.)
11
StorReducealsokeepslatencytoaminimum,typicallyaddinglessthan50msoflatencyevenwhenrunninginthecloud.Formostsituations,thisaddedlatencymakesnodifferenceatalltoendusersanddoesnotaffectthroughput.
StorReducemaintainsanindexofuserdataonfastlocalstorage.EachStorReduceserverkeepsitsownindependentindex,typicallyonlocallyattachedSSD.Inascale-outcluster,indexdataisspreadovertheserversintheclusterandcanbeconfiguredwithanyrequireddegreeofredundancy.
Allindexdatacanberebuiltfromthelogoftransactionsstoredinobjectstorage.Forlargedatasets,however,itcantakealongtimetorebuildtheindexfromscratch.Tospeedthisup,theserverorclusterperiodicallytakesasnapshotoftheindexinformationandstoresthisinobjectstorage.
WhenaStorReduceserverstartsup,ifanindexneedstoberebuilt,theserverwill:
1. Loadthelastindexsnapshotfromobjectstorage2. Replaysubsequenttransactionstobringtheindexuptodate.
Note:WhenstoppingaStorReduceserverrunningonanAmazonEC2instance,Amazonwilldeletealldataonthatmachine’sSSDinstancestorage.Whenthemachineisstartedagain,theindexmustberebuiltasdescribedabove.Forthisreason,itisrecommendedtoleaveproductionEC2StorReduceserversrunningratherthanstoppingandstartingthem.NotethatwhenrestartinganEC2instance,theinstancestorageispreserved.
StorReduceDemonstrationClusterIfyou’dliketoseehowStorReduceperformsinareal-worlddeployment,StorReduceoperatesanine-serverclusteronAmazonEC2instancestoprovidereal-timeperformancedemonstrations.Thisclusteriscontinuously(24x7)ingestingdataat¾ofapetabyteperday(8.8Gigabytespersecond).Thisclusterissimultaneouslyabletorecoverdataat4.8Gigabytespersecond(doublethespeedofthelargestDataDomain).StorReduceclusterscanscaleto31servers,withspeedsscalingalmostlinearly.Asimilar-sizedclusteronphysicalhardwarewilloperateatevenfasterspeeds,asphysicalhardwarecantypicallyprovidemoreCPUresourcesthanpubliccloudinstances.
12
DIAGRAM2:STORREDUCEDEMONSTRATIONCLUSTERDASHBOARD
ThescreengrababoveshowstheStorReducedemonstrationclusteringestingdataat8.8Gigabytespersecondwhileachievinga95%deduplicationrate.StorReducecanprovidelivedemonstrationsusingthissystemonrequest.
UsingMultipleStorReduceServersStorReduceoffersanumberofadvantageswhenusingmultipleStorReduceserversandisdesignedtomakeaddingserversaquick,easyprocess.BecauseStorReducemaintainsalogofalltransactionsonobjectstorage,multipleserverscanwatchthistransactionlogtokeeptheirindependentindicesuptodate.ThismeansthatyoucansetupnewserverstotalktoanexistingObjectStorageservice,andtheywillautomaticallypopulatetheirlocalindexdatafromObjectStorage.
ThesectionsbelowdetailsomeofthecapabilitiesandadvantagesthatorganizationscanrealizewhenusingmultipleStorReduceservers.
Scale-outClustersOneofthebestwaystoimprovetheredundancyandresiliencyofyourdatabackupinfrastructureisthroughStorReduce’sscale-outclustercapability.Ascale-outclusterusesmultipleStorReduceserverstodistributeloadandprovideautomaticfailoverwithinasingledatacenterorregion.
13
Inascale-outcluster,dataisdividedintoshards,witheachshardassignedtoaprimary“write”serverand,optionally,oneormoresecondary“read”servers.Foreachshard,thesecondaryserver(s)standreadytotakeoverasprimaryserverforthatshardintheeventofaproblem.Failoverhappensautomatically,coordinatedviaanopen-sourcedistributedkey-valuestore.
AnyS3protocolrequestcanbeprocessedbyanyStorReduceserver,soloadcanbedistributedbyDNSround-robinoraloadbalancerdeployedinfrontofthecluster.Theserverprocessingagivenrequesthandlesanyrequiredcoordinationwithotherserversinthecluster.
Thisarchitectureallowstheamountofdatamanagedandthethroughputtoscaleupalmostlinearlyasmoreserversareaddedtothecluster.Intra-clustercommunicationsarehighlyoptimizedandcanbehandledonaseparatenetworkinterfacetomaximizethroughput.
Formoreinformationondeploymentoptionsforscale-outclusters,pleasecontactStorReduce.
MakingDataAvailableinMultipleLocationswithRead-OnlyEndpointsLargeenterprisesneedtobeabletoaccessdatafrommultiplelocations—andtheyshouldbeabletodoitwithouthavingtonavigatemultiplecomplexdatasilos.StorReducewasdesignedtoallowStorReduceserversorscale-outclusterstobedeployedasread-onlyendpoints,eachofwhichcanreadandre-hydratedata.Inthisway,thesamecontentcanbefetchedfrommultipleStorReduceserversindifferentlocations,eachhavingthesameviewofthecontent,andalllocationsupdatedinreal-time.
Toimplementthis,onecommondeploymentscenarioistohaveaStorReduceserverrunningon-premises,deduplicatingdataasitissenttothecloud.AsecondStorReduceserverrunsinthecloudasaread-onlyendpoint,providingreal-timeaccesstothedata(inre-hydratedform)tocloud-basedapplicationsandservicesviaitsS3interface.Thismodelworksparticularlywellformovingbackupstocloud.
CreatingMultipleCopiesoftheDatawithDataReplicationStorReducealsomakesiteasyandcost-effectivetocreateandstoremultiplecopiesofdata.Similartotheread-onlyendpointscenario,aStorReduceservercanbeconfiguredtoautomaticallyreplicatedatafromitsprimaryobjectstoretooneormoreotherobjectstorelocations.WhenevertheStorReduceserverdetectsanychanges,itautomaticallycopiesthemtotheotherlocation(s).And,sincetheserveronlyreplicatesdeduplicateddata,thecostofthatdatatransferisatinyfractionofconventionaldatatransfers.
AStorReducereplicationservercanbeusedtoreplicatedata:
• Acrossmultipleregionsinthesamecloud• Acrossmultiplecloudproviders(includingAmazonS3,AzureBlobStorage,andGoogleCloud
Storage)• Fromprivatetopubliccloud,orviceversa(e.g.ActiveScalehybridcloudsolutions).
14
ObjectCloneEnterprisesfacemanyscenarioswhereit’snecessaryforusersorgroupstohavetheirown,separatecopiesofadataset.StorReducemakesiteasyandefficienttocreatesuchcopiesthroughobjectcloning.
AStorReduceservercan“clone”allobjectsinastoragebucket,creatinganynumberofvirtualcopiesoflargedatasets.Eachcloneiswritableonanobject-by-objectbasis.StorReduceusescopy-on-writesemantics,soclonesdonottakeanyextraobjectstoragespaceuntildataiswrittentotheclone.Eventhen,dataisfullydeduplicatedbetweenclones.
BycombiningStorReduceuseraccountsandaccesscontrolpolicieswithobjectcloning,teamswithinanorganizationcaneachbeprovidedwiththeirownwritablecopyofadataset,withoutbeingabletoseeeachother’schangesornewdata.Andenterprisescandothiswhilestillmaintainingasinglededuplicationpoolacrosstheentiresystem.
ObjectcloningalsoenablessimplerandmoreflexibletestingandQA,sincesoftwarecanbetestedagainstaclonedbucket.Thesoftwaremaintainsfullcontroloveralldatainitsbucket,withoutbeingabletointerferewithanyothercloneortheoriginaldata.
DIAGRAM3:CLONINGOFOBJECTDATA
15
SecuritySecurityisessentialtoanyviablecloudstoragesolution,andStorReduceisdesignedfromthegrounduptoprotectdataatrestandinmotion.Inadditiontousingthesecurityofyourunderlyingobjectstorageservice,StorReduceprovidesthefollowingsecuritycapabilities:
• Client-SideDataEncryption:StorReducesupportsclient-sidedataencryption,performedinreal-timeasdataispassedthroughtheStorReduceserver.Thisallowsdatatobeencryptedon-premisesbeforebeingsenttothecloud.KeymanagementcanbeperformedusingAmazon’sKMS(KeyManagementService)orviaaKMIPcompatiblekeyserver.
• CloudEncryption-at-Rest:Asanalternativetoclient-sideencryption,StorReducesupportstheuseofencryption-at-restservicesfromcloudserviceproviders,suchasAmazon’sS3Server-SideEncryptionforDataatRest.
• In-transitDataEncryption:TheStorReduceserverrequiresHTTPSencryptionfordashboardrequestsbydefaultandcanbeconfiguredtorequireHTTPSencryptionforallS3APIrequestsandobjectstoragerequests.YoucanuploadandsetservercertificatesthroughtheStorReducedashboard.
• UserAccountManagement:StorReducemaintainsasetofuseraccountsforeachStorReducedeployment.YoucanusetheseuseraccountstoprovidepeoplewithlimitedaccesstotheStorReducedashboard,ortoprovidepeopleorprogramswithlimitedaccesstotheS3API.Youcanalwaysrevokeindividualuseraccountstoinstantaneouslycutoffaccess.
• ClientAccessKeys:EachuseraccountcanhavemultipleaccesskeysusedforaccessingtheS3API.TheseworkinthesamewayasaccesskeysmanagedbyAmazon’sIAMservice.IndividualaccesskeyscanberevokedusingtheStorReducedashboard.
• DigitalSignatures:AllrequestsfromS3clientsmustbedigitallysignedusingasecretaccesskeytiedtoaStorReduceuseraccount.StorReduceacceptsAWSversion2andversion4signatures.
• DataSegregation:StorReducesupportsthecreationofmultiplestoragebucketsfordatasegregation,withdifferentaccessrightsforeachbucket.
• Policy-BasedAccessControl:EnterprisesecuritypoliciescanbeexpressedusingtheStorReducepolicyengine,whichsupportsAmazon’sIAMpolicylanguage.Youcanapplyaccesscontroltobucketsusingbucketpolicies(compatiblewithAmazonS3IAMbucketpolicies).
• AmazonAWSRoleCredentials:AStorReduceserverrunninginanEC2instancecanmakeuseofAWSIAMRolestosecurelyobtaincredentialsforaccessingitsunderlyingobjectstorage,enablingautomatickeyrotation.
UseCasesStorReducecandeliversignificantbusinessandtechnicaladvantagesinanumberofdifferentusecases.ThefollowingsectionsdetailsomeofthewaysthatenterprisescanuseStorReducetoimprovedatabackup,simplifyoperations,unlocknewvaluefromtheirexistingdata,andmore.
16
UsingStorReduceforPrimaryBackupsStraighttoObjectStorage:RemovingPurpose-BuiltBackupAppliancesOneofthemostimportantstepsanenterprisecantaketoslashcapitalandoperationalcostsiseliminatingpurpose-builtbackupappliancesandbackingupdatadirectlytocloud.StorReduce’sscale-outarchitectureandhighlyoptimizeddesign,alongwithleadingobjectstoragesolutionslikeActiveScale,makeiteasy.BycombiningStorReducewithsolutionslikeActiveScale,organizationscanachievefasterspeedsforbothingestandrecoverythanthelargestpurpose-builtbackupappliances.
UsingStorReducefromanon-premisesinstallationofaprivateobjectstore,LAN-speedrecoveryofprimarybackupsbecomesfastandsimple.StorReducealsooptimizesrecoveryofprimarybackupsfromapubliccloudsuchasAmazonWebServices,GoogleCloud,orMicrosoftAzure.Forthesescenarios,enterprisescanrecoverinfullinthecloudorrecoverbacktoon-premises.ThelattercaserequiresaLAN-speedconnectionsuchasAWSDirectConnectoritsequivalent.Or,enterprisescanuseahybridcloudinstallationwitharchivaldatastoringtopubliccloud,whileprimarybackupsstoretoprivateobjectstorage.
StorReducecandeployintheheadnodesofprivateobjectstores,furtherreducingdatacenterfootprintandcosts.
Backup/TapeMigrationtoCloudShiftingfromtapetocloudstoragecandrivemajorcostsavings.Accordingto451Research’sEnterpriseStorageSurvey2016,“Movingdataofftapeto(mainly)publiccloudobjectstorage...forbackupanddisasterrecovery...isthelargestswinginmostenterprisesITbudgetsfor2017,withanestimated12%dropinon-premisesITspendoverthisperiod.”Inthesamesurvey,35%ofenterprisessaidtheyplantoredesigntheirbackupand/ordisasterrecoveryinfrastructurestoincludecloudoverthenextyear.
Deduplicationplaysabigroleinthosesavings.Tapearchivesgenerallycontainperiodicfullbackupswithmultiplecopiesofthesamedatasets,whichcanbereducedtoasinglecopywithdeduplication.Byshiftingtocloudstoragewithdeduplication,enterprisescanreducetheamountofdatatheystoredowntoaslittleas1/30thofcurrentdatavolumes.
StorReducemakesmigratingtapeordisk-basedbackupsimple.ByinstallingStorReducesoftwareon-premises,enterprisescaninitiateaveryfastmigrationoftheirlargetapearchivesandbackupappliancedatatocloud,whileminimizingbandwidthduringthetransfer.
Onceorganizationsareusingcloud,StorReducealsomakesitpossibletorealizemuchmorevaluefrombackupdata.EnterprisescandeployaStorReduceread-onlyendpointtomakethedataavailableon-cloudfordevelopment,testing,qualityassurance,anddisasterrecovery.
17
DIAGRAM4:ON-PREMISESBACKUPTOCLOUDMIGRATION
CloningLargeDataSetsonCloudorinObjectStoresStorReduceObjectClonemakesitsimpleforenterprisestorepeatedlyproducefullyisolated,copy-on-write(COW)clonesofbucketscontainingmillionsofobjectsandpetabytesofdata,atvirtuallyzerocost.Clonedbucketscontainacomprehensivesnapshotofthedataatasinglepointintime.Enterprisescanusethatdataforawiderangeofpurposeswithoutworryingthatchangesanddeletionsfromdevelopers,researchers,andexperimentswillcausedatacorruption.
ObjectClonebenefitsanykindofdatathatcanbestoredinthecloud,whethergeneratedfromInternetofThingsappliances,financialtrading,lifesciences,multimedia(images,videos,music/audioetc.),orevenbackupworkloads.ObjectClonebenefitsdatathatdeduplicates,aswellasdatathatdoesnotdeduplicate.Itbringsstate-of-the-artvolumecloningcapabilitiestoobjectstorageandextendstheutilityofanorganization’sobjectstorageinvestmentwellbeyondbasicdatabackup.
WithObjectClone,organizationscan:
18
• ProtectAgainst“FatFingers”orMaliciousDeletionofDatainObjectStore:Administratorscannowefficientlymaketime-basedclonesofentirebucketsandassignread-onlyaccessfor“lockingin”protecteddataatcriticalpointsintime.
• CapitalizeonBigData,InternetofThings,andResearchUseCases:Enterprisescanclonepetabyte-scaledatasetssothatresearcherscanworkindependentlyofeachotherintheirownscratchareas.
• StreamlineITOperations:ObjectCloneallowsorganizationstotestnewversionsofsoftwareagainsttheirdatasetinisolation.
• OptimizeDevelopmentandSoftwareTesting:Organizationscanclonetestdatasetssothatdevelopersandtesterscanworkwiththewholedataset,notjustasmallsubset,inisolation.Theycanrollthestatebackafterfailuresandretestrapidly.
• ImproveSoftwareQualityAssurance:Softwareteamscantakeacloneofanentiresystematthepointthatitfailsatestandhanditbacktothedeveloper.
• SimplifyHadoopandBigDataImplementations:OrganizationscanuseObjectClonetotakepoint-in-timesnapshotsofthestateofaclusterandrollbacktoapreviousstateafteraproblem.
ReplacingTraditionalStorageSolutionswithObjectStorageObjectstoragewithinthedatacentercanprovidesignificantcostandscalabilityadvantagesovertraditionalstoragesolutions.PrivateobjectstoreproductssuchasActiveScalecanprovideprotected,redundantstoragespreadovermultiplelocationsforafractionofthecostoftraditionalsolutions.
Datadeduplicationisanessentialpartofanymodernstoragesolution,andStorReducecanhandlethehighthroughputanddatavolumesrequiredofmodernprivateobjectstoresolutions.ByusingaStorReducescale-outclusterinfrontofActiveScaleobjectstore,enterprisescanfurtherreducestoragecostswhilestillmaintainingveryhighthroughputrates(10Gigabytespersecondormore).
On-CloudDeduplicationforBackupsManyorganizationswouldliketomovetheirentireITinfrastructurestocloudandeliminatetheexpenseandhardwaresilosofon-premisespurpose-builtbackupappliances.Bydoingso,theycanreducetotalcostofownership(TCO),improvedisasterrecovery,eliminatehardwarerefreshcycles,andimprovethescalabilityofrecovery.StorReducecanplayasignificantroleinthesecostsavings,reducingthecostofstoringprimarybackupson-clouddowntoaslittleas3%.
Toenablededuplicationofbackupson-cloud,StorReducesoftwarecanbeinstalleddirectlyfromleadingCloudMarketplaces(Amazon,Azure,GoogleCloud),andintegratewithmoston-cloudbackupofferings,includingVeritasNetBackup.Bydeduplicatingbackupdatainline,StorReduceremovesupto97%ofthedatathatorganizationsarecurrentlypayingtostore.
19
DIAGRAM5:ON-CLOUDDEDUPLICATION
MovingorReplicatingDataBetweenCloudsManyorganizationsusemorethanonepubliccloudorhavedatainahybridcloudsystem.Ifamulti-cloudstrategyisn’tgoingtoaddhugenewlayersofcomplexity,however,organizationsneedtobeabletoquicklyandaffordablyreplicateormovetheirdatafromonecloudtoanotherwhileminimizingstoragecosts.
StorReducemakesiteasytousemultiplecloudswithautomatedcloud-to-cloudreplication.AnyorganizationwithdatainmorethanonecloudcaninstallStorReducesoftwareoneachcloud,andthenreplicatedata.Bydoingso,theycanreducedatasizetoaslittleas3%andreducebandwidthanddatatransfercostsbyupto30times.Thedatatheymoveisaccessiblebyanycloudservice.Inaddition,byreducingdatavolume,StorReduceenablesorganizationstoaffordablykeepdataintwocloudsortwocloudregionstosatisfyredundancyorcompliancerequirements.Evenwithinasinglepubliccloud,organizationscanusethesamecapabilitytoreplicatedatafromregiontoregion.
20
DIAGRAM6:CLOUD-TO-CLOUDDATAREPLICATION
OptimizingBigData/HadoopStorageMoreorganizationsareusingbigdataandHadoopeveryday.Withstate-of-the-artdeduplicationandstorageefficiencies,StorReducecanmakebigdataanalysistoolsmuchmoreefficientandaffordable.
EnterprisescanuseStorReduceinconjunctionwithproductslikeWANdiscoFusionforbackups,snapshots,andcloningofstorageforHadoopclustersandotherbigdatasolutions.
IsYourStorageKeepingUpwithYourBusiness?Asounddataprotectionstrategyhasneverbeenmoreimportant.Buttoomanyenterprisesarewrestlingwithsoaringstoragecapitalcosts,complexhardwareappliances,andsiloeddata.Now,StorReduceinlinededuplicationsoftware,combinedwithhigh-densityActiveScaleobjectstorage,makesiteasytocapitalizeonthepowerofcloudstorage.Betterprotectyourbusiness.Efficientlymanagemassiveamountsofdata.Andslashyourstoragecostsbyupto70%.
Tolearnmore,visitwww.hgst.com/storreduce
