1 Chapter 12: Green Data Centers Yan Zhang and Nirwan Ansari Advanced Networking Laboratory New Jersey Institute of Technology Newark, NJ 07102 HANDBOOK

1

Chapter 12: Green Data Centers

Yan Zhang and Nirwan AnsariAdvanced Networking LaboratoryNew Jersey Institute of Technology

Newark, NJ 07102

HANDBOOK ON GREEN INFORMATION AND COMMUNICATION SYSTEMS

2

Power Consumption of Data Centers

In 2007, Environmental Protection Agency (EPA) Report to Congress on Server and Data Center Energy Efficiency assessed trends in the energy usage and energy costs of data centers and servers in U.S.

Based on the power consumption of data centers in U.S. from year 2000 to 2006, the power consumption of data centers is predicted under five different scenarios: Two baseline prediction scenarios: historical trend scenario,

current efficiency trend scenario. Three energy-efficiency scenarios: improved operation

scenario, best practice scenario, and state-of-the-art scenario.

3

Power Consumption of Data Centers This prediction was performed with the total power consumption of

the installed base of servers, external disk drivers, and network ports in data centers multiplied by a power overhead factor caused by the power usage of power distribution and cooling infrastructure in data centers.

Historical trend: simply estimates the power consumption trends based on the observed power usage from year 2000 to 2006.

Current efficiency trend: estimates the power usage trajectory of U.S. servers and data centers by considering the observed efficiency trends for IT equipment and site infrastructure systems.

Data centers and servers in U.S. consumed about 61 billion kWh in 2006 for a total electricity cost of about $4.5 billion.

The energy use of data centers and servers in 2006 was more than doubled the electricity that was consumed by data centers in 2000.

It is estimated the energy usage of data centers could nearly double again in 2011 to more than 100 billion kWh with historical and current efficiency trends.

Improved operation trend: utilizes any essentially operational technologies requiring little or no capital investment to improve energy efficiency beyond “current efficiency trends”.

Best practice trend: adopts more widespread technologies and practices in the most energy-efficient facilities in operation.

State-of-the-art trend: maximizes the energy efficiency of data centers using the most energy-efficient technologies and best management practices available today.

4

Energy Efficiency of Data Centers

APC White Paper #6 (Ref [2]) investigated the total cost of ownership (TCO) of physical data center infrastructure, and found that the cost of electrical power consumption contributed to about 20% of the total cost.

Numerous studies have shown (Ref [3][4]): Data center servers rarely operate at full utilization. The average server utilization is often below 30 percent of the

maximum utilization in data centers.

At low levels of workload, servers are highly energy-inefficient.

5

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100

Utilization (percent)

Ser

ver

Pow

er U

sage

(pe

rcen

t of

pea

k)

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100

Ser

ver

Ene

rgy

Eff

icie

ncy

(per

cent

age)

Power (typical)

Energy efficiency (typical)Power (proportional)

Energy efficiency (proportional)


Server power usage and energy efficiency at varying utilization levels, from idle to peak performance (L. A. Barroso and U. Hölzle, “The Case for Energy-Proportional Computing,” Computer, 40(12):33–37, Dec. 2007).

Typical servers: consume about half of its full power at the idle state.

Power proportional servers: consume about half of its full power at the idle state.

6


A typical data center power usage (adapted from M. Ton, B. Fortenbery, and W. Tschudi, “DC Power for Improved Data Center Efficiency,” http://hightech.lbl.gov/documents/DATA_CENTERS/DCDemoFinalReport.pdf, Mar. 2008).

7

Energy Efficiency Metrics for Data Centers

In order to quantify the energy efficiency of data centers, several energy efficiency metrics have been proposed to help data center operators to improve the energy efficiency and reduce operation costs of data centers: Power usage effectiveness (PUE) and data center infrastructure

efficiency (DCiE)

Data Center energy Productivity (DCeP)

Datacenter Performance Per Energy (DPPE)

Green Grid Productivity Indicator

8

Power Usage Effectiveness (PUE) The most commonly used metric to indicate the energy

efficiency of a data: PUE and its reciprocal DCiE. PUE definition:

PUE=1.0 implies: there is no power overhead and all power consumption of the data center goes to the IT equipment.

PUE measures the total power consumption overhead caused by the data center facility support equipment, including the cooling systems, power delivery, and other facility infrastructure like lighting.

Total Power Consumption of a Data CenterPUE =

Total Power Consumption of IT Equipment

1 Total Power Consumption of IT EquipmentDCiE = =

PUE Total Power Consumption of a Data Center

9

Power Usage Effectiveness (PUE)

The average data center PUE in the US in 2006 is 2.0, implying that one Watt of overhead power is used to cool and deliver every Watt to IT equipment (Ref[1]).

It also predicts that “state-of-the-art" data center energy efficiency could reach a PUE of 1.2 (Ref [66]).

Google publishes quarterly the PUE results from data centers with an IT load of at least 5MW and time-in-operation of at least 6 months (Ref [67]). The twelve-month, energy-weighted average PUE result

obtained in the first quarter of 2011 is 1.16, which exceeds the EPA's goal for state-of-the-art data center efficiency.

10

Data Center energy Productivity (DCeP)

Energy efficiency and energy productivity are closely related to each other. Energy efficiency focuses on reducing unnecessary power

consumption to produce a work output. Energy productivity of a data center measures the quantity of

useful work done relative to the amount of power consumption of a data center in producing this work.

DCeP allows the continuous monitoring of the productivity of a data center as a function of power consumed by a data center.

DCeP metric tracks the overall work product of a data center per unit of power consumption expended to produce this work.

Useful Work ProducedDCeP =

Total Data Center Power Consumed Producing this Work

11


DPPE evaluates the energy efficiency of data centers as a whole. The DPPE metric indicates data center productivity per unit energy.

DPPE defines four sub-metrics: IT Equipment Utilization (ITEU) ITEE (IT Equipment Energy Efficiency) PUE GEC (Green Energy Coefficient)

These four sub-metrics reflect four kinds of independent energy-saving efforts, and are designed to prevent one kind of energy-saving effort from affecting others.

12


ITEU It measures the degree of energy saving by efficient operation

of IT equipment through virtual techniques and other operational techniques.

ITEE It is defined as the ratio of the total capacity of IT equipment to

the total rated power of IT equipment. This metric aims to encourage the installation of equipment

with high processing capacity per unit electric power in data centers to promote energy savings.

Total Measured Power of IT EquipmentITEU =

Total Rated Power of IT Equipment

Total IT Equipment CapacityITEE =

Rated Power of IT Equipment

13


PUE: It indicates the power saving for data center facilities. The less power consumption of facility infrastructure, the

smaller the value of PUE.

GEC: It is defined as the ratio of the Green Energy produced and used

in a data center to its total power consumption. The value of GEC becomes larger if the production of non-CO2

energy is increased in a data center.

DPPE Considering the definitions of the above four sub-metrics, DPPE

incorporates these four sub-metrics and can be expressed as a function of them as follows:

Green EnergyGEC =

Total Power Consumption of a Data Center

1 1DPPE = ITEU ITEE

PUE 1-GEC

Total Power Consumption of a Data CenterPUE =

Total Power Consumption of IT Equipment

14


Green Grid Productivity Indicator is a multi-parameter framework to evaluate overall data center efficiency.

Through the use of a radial graph, relevant indicators such as DCiE, data center utilization, server utilization, storage utilization, and network utilization can be quickly, concisely and flexibly emerged to provide organizational awareness.

How it works: Set up the target value for each indicator. Plotting the peak and average values of each indicator during

the period of monitoring, together with their target and theoretical maximum values on a radial graph.

• Assess how well the data center resources are utilized, • Check if the business targets are achieved visually and quickly, • Figure out how to spend their efforts to maximize the benefits.

15


Examples of using the Green Grid indicator tool. (adopted from The Green Grid. “The Green Grid Productivity Indicator,” http://www.thegreengrid.org/~/media/WhitePapers/White_Paper_15_-TGG_Productivity_Indicator_063008.pdf?lang=en).

16

Techniques to Improve Energy Efficiency of Data Centers

IT Infrastructure Improvements Servers and Storages Network Equipment

Power DistributionSmart Cooling and Thermal Management Power Management Techniques

Provisioning Consolidation Virtualization

Others

17

IT Infrastructure Improvements

Approximately 40% - 60% power consumption of a data center is devoted to IT infrastructure, which consists of servers, storage, and network equipment

Servers and Storages CPU

Dynamic Voltage/Frequency Scale (DVFS): Dynamic Voltage/Frequency Scale (DVFS): 23% - 36% energy savings.

memory and disk power shifting (Ref [14]): re-budget the available power between processor power shifting (Ref [14]): re-budget the available power between processor

and memory. and memory. Power shifting is a threshold-based throttling scheme to limit the number of operations performed by each subsystem during an interval of time, but power budget violations and unnecessary performance degradation may be caused by improper interval length.

Mini-rank (Ref [16]): an adaptive DRAM architecture to limit power consumption of DRAM by breaking a conventional DRAM rank into multiple smaller mini-ranks with a small bridge chip.

Dynamic Rotations per Minute (DRPM) (Ref [17]): a low-level hardware-based technique to dynamically modulate disk speed to save power in disk drives since the slower the disk drive spins the less power it consumes.

18


Energy proportional systems PowerNap (Ref [19]):

• Attune the server power consumptions to server utilization patterns.

• Transit rapidly between a high-performance active state and a minimal-power nap state in response to instantaneous load.

• PowerNap can be modeled as an M/G/1 queuing system.

19


Network Equipment: switches, routers, wireless access points. Sleeping mode

• Transit into the low-power sleep mode when no transmission is needed, and return back to the active mode when transmission is requested.

• The transition time overhead of putting a device into and out of the sleep mode may reduce energy efficiency significantly.

Rate-adapting • The lower the line-speed is, the less power the devices

consume. • Adapt the transmission rate of network operation to the offered

workload.• Speed negotiation is required in the rate-adaption scheme for

both of the transmission ends.

20

Power Distribution

Current typical power delivery systems for data centers still use alternating current (AC) power: Distributed from utility to the facility, and is then stepped

down via transformers and delivered to uninterruptible power supplies (UPS).

Several levels of power conversion exist in both data center facilities and within IT equipment that results in significant electrical power losses, including power losses in UPS, transformers, and power line losses.

DC power distribution system has been demonstrated and evaluated for data centers (Ref [32]).

21

Smart Cooling and Thermal Management

Most data centers use liquid cooling for computer room air conditioning (CRAC).

Rack-level liquid-cooling solutions bring chilled water or liquid refrigerant closer to the servers. Rear-door liquid cooling Sealed rack liquid cooling In-row liquid-coolers Overhead liquid-coolers

Data center liquid cooling techniques tend to use naturally-cooled water, like lake or sea water Electrical savings by eliminating or reducing the need

for water chillers in data centers.

22

Smart Cooling and Thermal Management

The predominant air cooling scheme for current data centers is to use the CRAC units and an under-floor cool air distribution system.

23

Power Management - Provisioning

Provisioning is an effective solution to reduce the power consumption by turning off the idle servers, storages, and network equipment, or by putting them into a lower power mode. An adaptive dynamic server provisioning technique (Ref [49])

• Effective to dynamically turn on a minimum number of servers required to satisfy application specific quality of service and load dispatching.

• Tailored for long-lived connection-intensive Internet services..

A power-proportional cluster (Ref [50]) • Consists of a power-aware cluster manager and a set of

heterogeneous machines.• Uses currently available energy-efficient hardware, mechanisms for

transiting in and out of low-power sleep states, and dynamic provisioning and scheduling to minimize power consumption.

• Especially tailored for short lived request-response type of workloads.

24


Sierra (Ref [53]) A power-proportional, distributed storage system. Turning off a fraction of storage servers during trough traffic

period. Utilizing a set of techniques: power-aware layout, predictive

gear scheduling, and a replicated short-term store, to maintain data consistency and fault-tolerance as well as system performance.

Rabbit (Ref [54]) A power-proportional distributed file system Provides ideal power-proportionality for large-scale cluster-

based storage and data-intensive computing systems by using a new cluster-based storage data layout.

Rabbit can maintain near ideal power proportionality even with node failures.

25


ElasticTree (Ref [55]) Dynamically adjust the set of active network elements, links

and switches, to satisfy changing data center traffic loads. Given the data center network topology, the traffic demand

matrix, and the power consumption of each link and node, ElasticTree minimizes the total power consumption of a data center by solving a capacitated multi-commodity cost flow (CMCF) optimization problem.

Urja (Ref [56]) A network wide energy monitoring tool Integrated with network management operations to collect

configuration and traffic information from live network switches and to accurately predict their power consumption.

26

Power Management - Consolidation

Power savings with application consolidation Application consolidation in cloud computing (Ref [57]):

• The energy performance trade-offs for consolidation.• The application consolidation problem can be modeled as a

modified bin-packing problem, and the optimal points exist.

Generic application-layer energy optimization (Ref [58]):• Guides the design choices by using energy profiles of various resource

components of an application.

Intelligent data placement and/or data migration can be used to save energy in storage systems. Hibernator (Ref [59]):

• A disk array energy management system.• Several techniques to reduce power consumption while maintaining

performance goals, including disk drives that rotate at different speeds and migration of data to an appropriate-speed disk drive.

27

Power Management - Virtualization

Effective to enhance server utilization, consolidate servers and reduce the total number of physical servers. Power consumption caused by servers is reduced. Cooling requirement should also be reduced

comparably. Effective to build energy proportional storage systems.

Sample-Replicate-Consolidate Mapping (SRCMap) (Ref [61])• A storage virtualization solution for energy-proportional storage.• Consolidating the cumulative workload on a minimal subset of

physical volumes proportional to the I/O workload intensity. GreenCloud (Ref [62])

• Enabling comprehensive online monitoring, live virtual machine migration, and virtual machine placement optimization to reduce data center power consumption while guaranteeing the performance goals..

28

Others

Energy-aware routing (Ref[63]) An energy-aware routing optimization model. The objective is to find a route for a given traffic

matrix that minimizes the total number of switches. The proposed energy-aware routing model is NP-

hard, and a heuristic algorithm was required to solve the energy-aware routing problem.

Energy proportional datacenter network architecture (Ref[64]) A flattened butterfly data center topology is

inherently more power efficient than the other commonly proposed topology for high-performance data centers.

29

Conclusions Power consumption is a central critical issue for data

centers. As reported in 2005, the electricity usage of data centers has

been almost doubled from 2000 to 2005. The electricity cost accounts for about 20% of the total cost of

data centers.

Numerous studies have shown that the average server utilization is often below 30% of the maximum utilization in data centers.

To quantify the energy efficiency of data centers, several energy efficiency metrics have been proposed: PUE, DCiE, DCeP, DPPE, and Green Grid Productivity Indicator.

Techniques to improve energy efficiency of data centers.

30

Thanks for your attention!

Documents

1 Chapter 12: Green Data Centers Yan Zhang and Nirwan Ansari Advanced Networking Laboratory New Jersey Institute of Technology Newark, NJ 07102 HANDBOOK