How to Measure Energy Consumption in Your Data Center

Gartner RAS Core Research Note G00205428, Rakesh Kumar, 8 September 2010, R3506 02112011

This research outlines a pragmatic approach that should be used to get an overall measurement of how much energy a data center is consuming. Without a practical measurement methodology, users will waste money and operate the center at suboptimal levels.

Key Findings

• Measuringdatacenterenergyconsumptionatthebuildinglevelcanbeacomplexexercise(dependingontheage,designanduseofthebuilding);itrequiresacarefulbreakdownofthecoolingcomponent.

• Continuouspowerutilizationefficiency(PUE)readingswillbecomethenormformostnewlargedatacenters.

• Rack-levelenergymeasurementisrelativelystraightforward,andallusersshouldbedoingthatnow.

Recommendations

• MeasureenergyacrossIThardware(server,networkingorstoragebox),racksandelectricalfacilitiestogetanaccurateapproximationofthetotalenergyusedinthedatacenter.

• Accuratelymeasurethedata-center-only,building-levelcooling-energyusage,becauseitaccountsformorethan40%ofthedatacenter’senergyconsumption.

• Usetheenergymanagementsoftwaretoolsavailableonthehardwarecomponents(servers,storageboxes,networkingequipment,etc.)tomeasureandmodelITcomponentenergyuse.

• FeedtheenergyusagedatafromITcomponents(servers,storageboxes,networkingequipment,etc.)intobroaderdatacenterinfrastructuremanagement(DCIM)toolstogainanaccurateusagemodelofenergyconsumptionacrosstheentiresite.

• Whereinternalprivatecloudsarebeingdeveloped,startmeasuringvirtualmachine(VM)energyusageasakeycomponenttoallocatingenergyconsumption.

2STRATEGIC PLANNING ASSUMPTIONBy2015,80%ofnewlargedatacenterswillreportcontinuousPUEreadings.

ANALYSISThedatacenterpower,coolingandenergysupply,andcostproblemsarelikelytoworsenduringthenextfewyears,becauseorganizationswillgrowtheirtechnologyinfrastructuresastheycomeoutofthisrecessionaryperiod.Energy-relatedcostsaccountforapproximately12%ofoveralldatacentercostsandarethefastest-risingcost.Moreover,withamorethan5%growthofservershipmentsperyearduringthenexttwoyears,organizationsneedto“forcefully”controltheirenergyconsumptionandcosts.Tomanagetheproblems,datacenteroperatorsneedtomeasureenergy-relateddataacrossthewholesite,includingthebuilding,thefacilitiescomponentsandtheITequipmentportfolio.

GartnerhaswrittenaboutemergingDCIMsoftwaretools,whichwillenableuserstostartdoingthis.“DCIM:GoingBeyondIT”explainsthisconcept,and“TheBenefitsofanIntegratedEnergyManagementSoftwareApproach”explainshowsuchanapproachmaywork.

Despitetheavailabilityofthesetoolsandabodyofexpertsandconsultantsworkingonthistopic,Gartnercontinuestogetcallsfromusersaskingthesimplequestion,“Whereisthebestplacetomeasuretheenergyinmydatacenter?”Theproblemiscompoundedbythefactthatthereisatheoreticaloridealsetofmeasurementsthatwillgiveacomprehensivesetofenergyreadingsacrosstheentiresite;however,formanyusers,thiswillbeinformationoverload.What’sneededisabreakdownoftheidealmodelintoapragmaticapproachthatwillgivesufficientinformationformostoperationalandplanningpurposes.Thisresearchfirstshowsanidealizedapproachbyoutliningthesixareaswhereenergycouldbemeasured.Itthenprioritizestheseareastogiveamore-practicalapproachthatusersshouldadopt.

TheIdealApproachtoDataCenterEnergyManagement

Togetacomprehensive,accurateandreal-timerecordoftheenergyusedindatacenters,usersneedtomeasureacrosssixareas:

1. Building

2. Electricalfacilities

3. Buildingfacilities

4. Racks

5. IThardware(server,networking,storagebox,etc.)

6. VM

ThesesixkeyareasareshowninFigure1.

Bymeasuringacrosstheseareas,datacentersoperatorscanobtainahighlydetailed,comprehensiveand,inmostcases,real-timeusagepictureoftheenergythatisbeingconsumedacrossthewholesite.Thiswillmeanthattheenergythat’scomingintoadatacentercanbeallocatedtomajordatacentercomponents,suchasfacilities,buildings,IT,etc.Thiscanthenbeusedforfutureplanning,allocatingscarcecapacity,suchasspaceorcoolingandappropriatefinancial(chargebackorsimplecostallocation)resources.

However,thisisa“perfectworld”scenario,and,formostorganizations,itwillneitherbenecessarynorpossibletomeasureacrossallsixareas.Thismaybebecauseofcostortechnicallimitationsofthesite;furthermore,dependingonthetoolsused,therewillprobablybesomeoverlapwhenmeasuringacrossallsixareas.Hence,werecommendprioritizingtheseareasanddevelopingapragmaticapproach.

The Pragmatic Approach to Data Center Energy ManagementApracticalimplementationoftheaboveidealapproachisshowninFigure2.Thesixareasareprioritizedanddividedintothreemainsegments.

Energymeasurementacross(1)IThardware(server,networkingorstoragebox),(2)racksand(3)electricalfacilitiesshouldbedoneimmediately.Thiswillbesufficientformostdatacenterneeds.Energymeasurementsacross(4)datacenterbuildingfacilitiesand(5)datacenterbuildingswillbenecessarymainlyforhostingprovidersthatwanttochargecustomersspecificallyforenergyusage.Measurementacross(6)VMswillhappenduringthenextfourormoreyears.TheywillbecomerelevantwhenuserswanttoexaminetheenergyassociatedwithaworkloadandwhenthatworkloadisrunninginaseparateVM.Eachareaisdiscussedbelow.

©2010Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.oritsaffiliates.ThispublicationmaynotbereproducedordistributedinanyformwithoutGartner’spriorwrittenpermission.Theinformationcontainedinthispublicationhasbeenobtainedfromsourcesbelievedtobereliable.Gartnerdisclaimsallwarrantiesastotheaccuracy,completenessoradequacyofsuchinformationandshallhavenoliabilityforerrors,omissionsorinadequaciesinsuchinformation.ThispublicationconsistsoftheopinionsofGartner’sresearchorganizationandshouldnotbeconstruedasstatementsoffact.Theopinionsexpressedhereinaresubjecttochangewithoutnotice.AlthoughGartnerresearchmayincludeadiscussionofrelatedlegalissues,Gartnerdoesnotprovidelegaladviceorservicesanditsresearchshouldnotbeconstruedorusedassuch.Gartnerisapubliccompany,anditsshareholdersmayincludefirmsandfundsthathavefinancialinterestsinentitiescoveredinGartnerresearch.Gartner’sBoardofDirectorsmayincludeseniormanagersofthesefirmsorfunds.Gartnerresearchisproducedindependentlybyitsresearchorganizationwithoutinputorinfluencefromthesefirms,fundsortheirmanagers.ForfurtherinformationontheindependenceandintegrityofGartnerresearch,see“GuidingPrinciplesonIndependenceandObjectivity”onitswebsite,http://www.gartner.com/technology/about/ombudsman/omb_guide2.jsp

3Figure 1. The Ideal Approach to Data Center Energy Management

Source:Gartner(September2010)

EnergyInto Data Center

One Building

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Two Electrical Facilities

Three Building Facilities

Electromechanical Efficiencies

Processor, System andLoad-Based Inefficiencies

Four Racks

Five ITHardware

Six VirtualMachines

EnergyInto Data Center

One Building

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Two Electrical Facilities

Three Building Facilities

Electromechanical Efficiencies

Processor, System andLoad-Based Inefficiencies

Four Racks

Five ITHardware

Six VirtualMachines

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Two Electrical Facilities

Three Building Facilities

Electromechanical Efficiencies

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Two Electrical Facilities

Three Building Facilities

Electromechanical Efficiencies

Three Building FacilitiesProcessor, System and

Load-Based Inefficiencies

Four Racks

Five ITHardware

Six VirtualMachines

Processor, System andLoad-Based Inefficiencies-

Four Racks

Five ITHardware

Six VirtualMachines

Figure 2. The Pragmatic Approach to Data Center Energy Management

Source:Gartner(September2010)

Needed for real-time PUE and for hosting sites

(4) D/C Building Facilities

(5) D/C Building

Next 48 months for linking energy to workload

(6) Virtual Machines

Within the next 18 monthsWill provide 80% of needs

(1) IT Hardware

(2) Racks

(3) Electrical Facilities

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Electromechanical Efficiencies

Processor, System andLoad-Based Inefficiencies

Needed for real-time PUE and for hosting sites

(4) D/C Building Facilities

(5) D/C Building

Needed for real-time PUE and for hosting sites

(4) D/C Building Facilities

(5) D/C Building

(4) D/C Building Facilities

(5) D/C Building

Next 48 months for linking energy to workload

(6) Virtual Machines

Next 48 months for linking energy to workload

(6) Virtual Machines

Within the next 18 monthsWill provide 80% of needs

(1) IT Hardware

(2) Racks

(3) Electrical Facilities

Within the next 18 monthsWill provide 80% of needs

(1) IT Hardware

(2) Racks

(3) Electrical Facilities

(1) IT Hardware

(2) Racks

(3) Electrical Facilities

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Electromechanical Efficiencies

Processor, System andLoad-Based Inefficiencies

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Electromechanical Efficiencies

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Electromechanical Efficiencies

Processor, System andLoad-Based InefficienciesProcessor, System and

Load-Based Inefficiencies-

4Readings That Should Be Taken During the Next 18 Months

IT Hardware (Server, Networking or Storage Component)UsersneedtomeasuretheenergyattheindividualITphysicaldeviceorhardwarelevelstogetanaccuratemeasurementoftheenergythatisbeingusedbythatdevice.Thisreferstoservers,networkingdevices,storagesystemsandsubcomponents,suchasrouters,switches,etc.Mostoftheenergyatthisstagegoesintotheservers,andthesecomponentshavethemost-sophisticatedenergymanagementtoolsbuiltin.Theserversandtheworkloadsrunningonthemwillbecomethemostimportantenergymanagementcomponentsduringthenextfewyears.

Hence,it’sreasonableforuserstofocusontheserversonlyatthisstage(theenergyusedbytheotherhardwarecomponentscouldbepickedupattheracklevel).Acommon-senseapproachneedstobedeployedwithreferencetothesubcomponents,becauseitmaynotmakesensetomeasuretheenergyacrossallthehundredsofdevices.Ontheotherhand,manyofthenewerdeviceshavebuilt-inmonitoringtechnology,andthisshouldbeusedwhereappropriate.

Inbladesystems,thehardwarevendorshavetypicallydevelopedsoftwaretoolsthatprovideenergyreadingsattheindividualbladelevel.Forexample,theIBMPowerExecutivesolutionisavailableforIBMBladeCenterandSystemxservers.ItenablesdirectpowermonitoringthroughIBMDirector.HP’sInsightSoftwaresuiteofproductsdoesthesameforHPhardware.Alternatively,vendorssuchasnLyte,Emersonand1Eprovidevendor-agnosticsoftwaretools.

Measuringenergyatthecomponentlevelprovidessomekeybenefits.ItenablesuserstocomparetheenergyefficiencyofdifferentIThardwarecomponents,astheyareactuallyusedintheorganization’sowndatacenters.Thismovesawayfromrelyingpurelyonvendordataforcomponentenergyusageinformationandprovidesanopportunitytocompareausage-basedvendorwithavendorbenchmark(wheresimilardevicesfromdifferentvendorsareused).Measuringwhichdevicesusetheleastamountsofenergyisalsopivotaltoenergy-basedchargeback,and,althoughmostorganizationswillnotoperateinthiswayforanumberofyears,it’susefultostartcollectingthedatanow.OneotheradvantageofmeasuringenergydataattheITcomponentlevelisthatisprovidesanaccuratecost-of-energyreadingforahardwaredeviceoveritsusefullifetime.Thisisvaluablewhenusersupgradehardwareandwanttounderstandthetotalcostofownership(TCO)ofthedevices.

RackAstheenergyinthedatacentercomesintotherack,areadingoftheenergyshouldbetaken.ThisisgenerallyconsideredtobethefirstpointatwhichenergyinadatacentercanbeassociatedspecificallywiththeITequipmenthousedintherack.Mostoftheracksuppliers,suchasRittal,Schneider,Dataracks,Orion,etc.,nowhavesomeformonintelligentpowerstripontherackthatwillprovideanenergyreading.Moreover,supplierssuchasRaritanhaveintelligentpowerdistributionunitsthatprovideintelligent,rack-levelpowermanagementsolutions.Thehardwarevendors,suchasDell,IBMandHP,havesoftwaretoolsbuiltintotheirspecificracksthatprovidearack-levelunderstandingoftheenergybeingused.Also,asusersstarttodeployverticallyintegrated,fabric-basedsolutions,measuringenergyattheracklevelwillgiveusefulenergyreadingsforthewholestack.

Inessence,pickinguptheenergyreadingsattheracklevelisnottoodifficultandisbeingmadeeasierbyvariousbuilt-intools.UsersshouldbeactivelytakingthesereadingsattheinletoftheirITsystems.

Electrical FacilitiesWhenenergycomesintoadatacenter,itisgenerallysplitintotheelectricalfacilitiescomponentsandthebuildingfacilitiescomponents.Inthissection,theenergythatgoesintothecoreelectricalfacilitiescomponentsisexamined.Thesecomponentsincludetransformers,powerdistributionunits(PDUs),uninterruptiblepowersupplies(UPSs),cables,switches,etc.MeasuringtheenergyatthislevelisimportanttogettheoveralldatacenterusagemodelandisrequiredtogetaPUEreading.Manyhavethesecomponentshavebuilt-inmeters,sothereadingscanbeexportedintoacentralrepository.

Although,measuringenergyatthislevelwilltypicallyrequirethehelpandsupportofthefacilitiesteam,ithastwodistinctadvantages.First,theoverallelectricaldistributionatthepointatwhichenergycomesintothedatacenterwilltypicallybedesignedwithadegreeofredundancy—forexample,anN+1ora2Napproach.Thismeansthatreadingswillhavetobetakenateachoftheredundantcomponentlevelstoachieveahighdegreeofaccuracy.Thesecondadvantagedealswiththefactthateachofthecomponentswillhavebuilt-ininefficiencies,whicharetypicallyworkload-based.

Bymeasuringtheenergyconsumedonacontinuousbasisacrossthesedevices,userswillbeabletogetaclearpictureofsomesignificantpiecesofinformation.Forexample,theywillbeabletoseehowthedeviceinefficiencieschangewithworkload.Hence,ifaPDUisrunningatarangeofbetween40%and70%,continuousreadingsofenergymappedontotheworkloadofthedatacenterwillshowwhatlevelofworkloadgeneratesthehighestlevelsofelectricalefficiencyinthePDU.Althoughthiswillprobablynotleadtoanychangesinthewaythedatacenterisrun,itwillprovidevaluableinformationforPDUupgradesordesignsofanewdatacenterforthecompany.

5Readings More Appropriate for Data Center Operators and Hosting Companies

Data Center Building FacilitiesThebuildingfacilitiescomponentsincludelighting,firesuppression,generatorand,ofcourse,thecoolingcomponents.Measuringtheenergyusedherewillprovideusefuldata,becausecoolingissuchalargeconsumerofdatacenterenergy(asmuchas40%ofthetotalenergyused).Inabuildingwherethedatacenterissharedwithofficespace,specificandaccuratemeasurementiskey.Forexample,ifcoolingissharedacrossthewholebuilding,andaproportiongoesintothedatacenter/machineroom,thendifficultiesariseinaccuratemeasurement.

Dependingonthecoolingsystem,itmaybepossibletomeasuretheenergyasthecomputerroomairconditioning(CRAC)level.Ifthisisnotpossible,thensomeapportionmentofenergywillhavetobemade.However,ifthisdone,itwillbedifficulttodeterminehowenergyusageandcostvarywithworkloadandwithusersinasharedenvironment.Ingeneral,wesuggestmeteringtopicklivedata,ratherthanapportionment.Withlightingandfiresuppressionsystems,apportioningusagemaybetheappropriatebecause,basedonclientconversations,theytypicallyformlessthan5%ofthetotalenergyused.Innewdatacenters,allofthesecomponentsshouldbemeteredfromtheoutset,andthedatashouldbefedintoadashboard.

Data Center BuildingForsomedatacenteroperators,itmaybeimportanttomeasurethetotalenergycomingintothedatacenter.Thisisparticularlytrueforlargehostingcompaniesthatwanttoapportionalltheirenergycoststotheirtenants.Thisenergymeasurementmaysoundlikeastraightforwardexercise:simplystickapowermeterattheinletpointandthemeasurementisdone.However,itcanbealittlemorecomplexthanthat.Ifthedatacenterisaseparate,purpose-builtsiteandalltheenergycominginisusedsolelyforthedatacenter,thenasimplemeasurementattheinletpoint(s)isgenerallysufficient.However,ifthedatacenterispartofabuildingthathousesotherpartsofthebusiness(offices,canteens,gyms,etc.),theelectricalnetworkmayhavebeendesignedtosharecomponents,suchaslightingandcooling.

Ageneralfacilitiessectionofthebuildingmayprovidecoolingforthewholebuilding,andaportionofthatcoolingmaybeusedbythedatacenter(machineroom).Inthiscase,anumberofreadingsmayhavetobetakentogetanaccuratefigure.Thepowercomingintothedatacenterstillneedstobemeasuredandrecorded;however,areadingwillalsoneedtobetakenorcalculatedfortheproportionofenergyusedspecificallyforcooling(andpossiblylighting,althoughthismaynotbeamaterialnumber)inthedatacenter.Thiswillbeaddedtotheinletelectricityreading.Together,thesefigureswillgiveapictureofthetotalamountofenergycomingintothedatacenter.

Theproblemisthat,togetreal-timePUEreadings,theenergyforcoolinghastobemeasuredonareal-timebasis,ratherthanaone-offproportionalallocation.Thismayrequireinstallingsomemetersinolderbuildings,andnewbuildingsshouldbedesignedtogetaccurateandcontinuousreadings.VendorssuchasPanduitandSchneider(TAC)havesolutionsthatcanbeused.

Readings More Appropriate to a Workload-Based Approach

VMMostoftheenergyreadingstakenatthefivepointsoutlinedaboverelatetoIThardwareandfacilitiesdevices.Whatwillultimatelybeusefulisassociatingenergyinadatacentertotheworkloadforwhichtheenergyisused.MeasuringenergyattheVMlevelisthefirststeptowardachievingthis.Althoughuserscanestimatethisvalueoruseaproxy,suchasdividingthetotalenergyconsumedbyaserverbythenumberofVMs,actualreadingswillbecomecriticalduringthenextfewyears.Asaresult,toolsarebecomingavailablefromvendorssuchasVMwareand1E.However,mostuserprocessesremaintooimmaturetotakeadvantageofatthistime..

Duringthenextfiveyears,theincreaseduseofvirtualization,theproblemsofVMsprawlandthegrowthofcloudserviceswillbegintomoveuserstothisapproachtodatacenterenergymeasurement.Assoftwareasaservice(SaaS)becomesavailableacrossarangeofhighlyvirtualizedhardwaredevices,measuringacrossVMswillbecomeincreasinglyimportantinunderstandinghowenergyisdistributedacrossdifferentworkloads.

Measuringenergyacrossthedatacenterisacomplex,butincreasinglynecessaryactivity.AnumberofvendorsarecompetinginthisDCIMspace,andusersarebeginningtoadopttheirtechnologies.Here,thesixidealareasformeasuringthatenergyhavebeenprioritizedintoamore-practicalscenario.Weadviseuserstoadoptthispragmaticapproachandstartmeasuringenergyusageacrosstheirsites.