Sustainable Data Center · 4 Sustainable Ecosystem Research Group –Sustainable Data Center •Integrated management of IT, power and cooling towards a Net-Zero energy data center

©2009 HP Confidential template rev. 12.10.09 1

Yuan Chen

Senior Research Scientist

Sustainable Ecosystems Research Group

Hewlett Packard Laboratories

Sustainable Data Center

2 © Copyright 2010 Hewlett-Packard Development Company, L.P.

HP LABS AROUND THE WORLD Global talent, local innovation

PALO ALTO

ST. PETERSBURG

HAIFA

BRISTOL

BEIJING

BANGALORE

SINGAPORE


HP LABS RESEARCH AREAS Innovation at every touch point of information

Sustainability

Information Analytics

Mobile & Immersive Experience

Printing & Content Delivery

Services

Networking

Intelligent Infrastructure

Cloud & Security

4

Sustainable Ecosystem Research Group

– Sustainable Data Center

• Integrated management of IT, power and cooling towards a Net-Zero energy data

center

– Resource Management as a Service

• Improve sustainability of urban infrastructure

• All the resources needed to run a building or a campus or a city – resources like light,

water, power, waste management -- are all orchestrated and optimized

5

Sustainability: A Holistic Perspective reducing overall available energy consumption

Available Energy consumed by IT

Available Energy saved by application of IT

Available Energy


Number of Data Centers

1

10

100

1,000

10,000

100,000

1,000,000

10,000,000

Single Rack Computer

Room

Midsize DC Enterprise DC Large DC

Data Center Size

Nu

mb

er

of

insta

llati

on

s

Source: Gartner Group, March 2011

98%

The rest of the global economy

Total carbon emissions

2%IT industry

Data Centers at the Core Emerging data centers


Supply Side:

− Lifecycle perspective

−available energy (exergy) required in extraction, manufacturing, operation and reclamation

−Utilize local sources of available energy

−Examine available energy in waste streams

Demand Side:

−Provision resources based on the needs

− Integrated management of IT, Power and Cooling

Supply and Demand Integration

Integrated Supply-Demand Management based on Service Level Agreements

Technical Approach


extraction operation manufacturing End of Life

IT

Power

Cooling

Flexible, Configurable and Efficient Micro-grids

Pervasive Cross-layer Sensing

Integrated Control

Advanced Analytics & Visualization

Data Center Scale Lifecycle Design

Sustainable Data Center Key Components and Key Elements

9

Sustainable Data Center Research Elements Lifecycle Design


IT

Power

Cooling

Flexible and Efficient Resource Microgrids


Integrated Control

Advanced Analytics and Visualization



Lifecycle Perspective Minimizing product lifecycle exergy consumption at design time

Extraction

exergy to produce materials

exergy to fabricate and assemble components

exergy for

transportation

operation manufacturing

exergy for installation

exergy to power electronics and thermal managment

exergy for waste

mitigation

End of LIfe

exergy to de-install and decommision

exergy to disassemble and/or recondition for safe disposal or recycle

time

exergy for Waste mitigation

“sustainable development is

development that meets the

needs of the present without

compromising the ability of

future generations to meet

their own needs”

the Brundtland Commission of the United Nations,

1987

http://en.wikipedia.org/wiki/Brundtland_Commission



http://en.wikipedia.org/wiki/United_Nations


Kinetic Energy + Heat+ Waste Streams

Energy refers to quantity of energy, available energy quantifies the useful portion (or quality) of energy

Lifecycle Perspective Energy or Exergy as a measure

• The quantity of energy was conserved • The usefulness of energy was destroyed


Server Lifetime Exergy Consumption

Extraction

5%

Manufacturing

13%

Transportation

1%

Operation (Power)

41%

Recycling

1%

Operation (Cooling,

incl. Infrastructure)

40%

Lifecycle Perspective Lifetime Exergy Advisor

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Handheld

Note

book

Desk

top

Bla

de

Serv

er

Data

Cente

r

Fra

ctio

n o

f Li

fecy

cle E

nerg

y

Operational

Embedded

Hannemann, C., et. al., “Lifetime Exergy Consumption as a Sustainability Metric for Enterprise Servers”, Proceedings of ASME Energy Sustainability, August 2008

13

Sustainable Data Center Research Elements Flexible Actuators


IT

Power

Cooling



Integrated Control




Data Center Infrastructure

15

Examples of Flexible Infrastructure Components

Power

• Power microgrids • Dynamic power control Cooling

• Cooling microgrids • Variable speed blowers • Adaptive vent titles

IT

• Virtualization • Dynamic resource allocation

Outside Air

Economizer

PV Power

Cooling Supply

Power Supply Data Center Demand

Water

Economizer

Grid PowerP

ow

er In

frastru

ctu

re

CRACs

IT

16


IT

Power

Cooling



Integrated Control



Sustainable Data Center Research Elements Cross-Layer Sensing

17

Pervasive Cross-Layer Sensing

IT Hardware (Servers, etc.) Facilities Power and Cooling

Sea of Sensors Dozens of sensors monitor power and temperature within IT equipment

iPDU and Remote Sensing Hundreds of sensors monitor power and cooling resources at facility-level

IT Software (Applications and Services)

Service Health Reporter and Insight Control Software sensors monitor the health of applications and services

18


IT

Power

Cooling



Integrated Control



Sustainable Data Center Research Elements Advanced Analytics


Advanced Visualization


EXTRACTION OPERATION MANUFACTURING END OF LIFE

Annual Savings: 359 MWh (~10%); 179,580 G direct water; 287, 328 Kg CO2

Motif Mining applied to water and air cooled chiller ensemble

Data Mining Pattern mining of chiller ensemble

Ref: Deb Patnaik, Manish Marwah, Ratnesh Sharma, and Naren Ramakrishnan, Temporal

Data Mining Approaches for Sustainable Chiller Management in Data Centers, ACM

Transactions on Intelligent Systems and Technology, 2011,

Cluster Analysis

Multivariate Time Series Data

Event Encoding

Frequent Motif Mining

Symbolic representation

Transition-event sequence

Frequent motifs

Sustainability characterizationof frequent motifs

Other discrete data sources can be integrated

21


IT

Power

Cooling



Integrated Control



Sustainable Data Center Research Elements Integrated Control Systems

Autonomous Control Dynamic Provisioning of IT and Cooling Resources based on the needs Three key goals

– Avoid doing nothing

– Do nothing well

– Meet SLAs

23 © Copyright 2010, Hewlett-Packard Development Company,

L. P.

Data Center Cooling Delivery and Distribution Cold and Hot

Ref: Chandrakant D. Patel, Cullen E. Bash, Ratnesh Sharma, Smart Cooling of Data Centers, in

ASME 2003 International Electronic Packaging Technical Conference and Exhibition

(InterPACK2003) , July 6–11, 2003 , Maui, Hawaii, USA


L. P.

Data Center Cooling Delivery and Distribution

Conventional control • a carry over from room level comfort cooling, • regulates the return air temperature to a given setpoint • Fans are typically fixed speed running continuously at 100% • Individual Air Handlers are self controlled and not coordinated • results in very cold air being supplied to the room ~ 12°C / 55°F on average due to mixing in the room

Room Level Temperature Sense Points

Room Chilled

Water Supply

AC Unit

Air Movers

Chilled Valve

T T

CW supply and control

TBlower &

Motor

Conventional Room Control


L. P.

Data Center Cooling Delivery and Distribution Conventional Room Control


L. P.

Data Center Cooling Delivery and Distribution Dynamic Zonal Cooling Control

Dynamic Zonal Cooling • Regulate cooling on the supply side of the air handlers. • Supply air cooling is regulated by sensing temperature at the intake of equipment racks • Fans use variable speed control • A much higher supply air temperature ~ 68-72°F, a much higher return air temperature ~ 85-95°F

Room Chilled

Water Supply

AC Unit

Air Movers

Chilled Valve

CW supply and control

TT

VFDVFD

Rack Level Temperature Sense Points


L. P.

Cooling Control in Data Center Level Zonal Effects of the CRAC units

CRAC 1 CRAC 3CRAC 2

CRA

C 6

CRA

C 5

CRA

C 4

Section A

Section B

Section C

Regions of Influence

Data Center Room

CRAC 1 CRAC 3CRAC 2

CRA

C 6

CRA

C 5

CRA

C 4

Section A

Section B

Section C

Regions of Influence

Data Center Room

TCI: Thermal Correlation Index: i

j

jiCRAC

TempTempCRACTCI

),(

Ref: C.E. Bash, C.D. Patel, R.K. Sharma, “Dynamic Thermal Management

of Air Cooled Data Centers,” the Tenth Intersociety Conference on

Thermal and Thermomechanical Phenomena in Electronic Systems

(ITherm ’06), 2006


L. P.

Cooling Control in Data Center Level

Thermal Correlation Index (TCI)

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

A1

.T1

A3

.T1

A5

.T1

A7

.T1

Aext

2.T

1

Aext

4.T

1

Aext

6.T

1

Aext

8.T

1

Aext

10

.T1

B2.T

1

B4.T

1

B6.T

1

Bext

1.T

1

Bext

3.T

1

Bext

5.T

1

Bext

7.T

1

Bext

9.T

1

C2

.T1

C4

.T1

C6

.T1

Cext

2.T

1

Cext

4.T

1

Cext

6.T

1

Cext

8.T

1

Cext

10

.T1

D2

.T1

D4

.T1

D6

.T1

E2

.T1

E4

.T1

E6

.T1

F1

.T1

F3

.T1

F5

.T1

F7

.T1

G1

.T1

G3

.T1

G5

.T1

G7

.T1

G9

.T1

crac1

crac2

crac3

crac4

crac5

crac6

crac7

crac8


L. P.

Cooling Control in Data Center Level Coordinated CRAC Control

R

A

C

K

s

Inlet Temps

PID SAT_SP

VFD_SP

Tmaster CRAC

PID SAT_SP

VFD_SP

Tmaster CRAC

PID SAT_SP

VFD_SP

Tmaster CRAC

Identification of Master Sensors

TCIs

Ref: C.E. Bash, C.D. Patel, R.K. Sharma, “Dynamic Thermal Management of Air Cooled Data Centers,” the Tenth

Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm ’06),

2006.


L. P.

Cooling Control in Data Center Level Coordinated CRAC Control Results

Conventional Mode

Dynamic Zonal Cooling Mode

HP Labs Data Centre

Palo Alto, CA

35% Energy savings Improved reliability Ref: C.E. Bash, C.D. Patel, R.K. Sharma, “Dynamic Thermal

Management of Air Cooled Data Centers,” the Tenth

Intersociety Conference on Thermal and Thermomechanical

Phenomena in Electronic Systems (ITherm ’06), 2006.


L. P.

Local Cooling Control Adaptive Vent Tile

AC Power

Microcontroller

DATA ACCESS LEVEL

1-Wire

AVT Base station

Actuator

Vent Tiles

12-24 VDC Power

Network

AVT Manager

Actuator Damper


L. P.

Local Cooling Control Vent Tile

0 20 40 60 80 100 12019

20

21

22

23

Time(min)

Deg

ree(

C)

0 20 40 60 80 100 1200

50

100

Time(min)

Ope

ning

(%)

TF1

TF2

TF3

VF1

VF2

VF3

•Inter-correlation •Nonlinearity

Ref: Zhikui Wang, Alan McReynolds, Carlos Felix, Cullen Bash, Christopher Hoover, Kratos: Automated Management of

Cooling Capacity in Data Centers with Adaptive Vent Tiles, in Proceedings of the 2009 ASME International

Mechanical Engineering Congress & Exposition (IMECE 2009), Nov 13-19, 2009, Lake Buena Vista, Florida.


L. P.

Local Cooling Control Adaptive MPC AVT control driven by Inlet temperatures

Temp reference

(Tref1, … TrefN)

AVT

Actuators

(1, …, M)

IT

Equipment

Temp

Sensors

(1, …, N)

Cooling

air flows MISO

Controller

Online Model

Estimator

Vent tile openings

(V1, … VM)

Temperatures

(T1, … TN)

Adaptive

Controller

Target

System

Temperatures

(T1, … TN)

1,...,1,0,ˆ)(

1,..,1,0,)(

,...,2,1,)(

)()1()(

)1(

)()1()()(

1

0

2

1

0

2

1

2

2

1

huiVikV

huiVikV

hpiTikT

ikBVikATikT

ikV

ikVikVTikTVJMinref

hu

iR

hu

iR

hp

iQref

V

Ref: Zhikui Wang etc., KRATOS: automated management of cooling capacity in data centers

with adaptive vent tiles, IMECE2009, November 13-19, Lake Buena Vista, Florida, USA


L. P.

Before Control After Control0

50

100T

ile O

peni

ngs

(%)

Before Control After Control

20

25

30

Tem

pera

ture

s(C

)

F G D E 44% Less Cooling Air Flow Needed

Local cooling on-demand delivery through Adaptive Vent Tile

Ref: Zhikui Wang etc., KRATOS: automated management of cooling capacity in data centers with adaptive

vent tiles, IMECE2009, November 13-19, Lake Buena Vista, Florida, USA

Local Cooling Control

35

Dynamic Resource Allocation in Virtualized Server Environments – Determine IT resources needed to maintain application performance

– Dynamically adjust IT resources based on needs

© Copyright 2010 Hewlett-Packard Development Company, L.P.

Management

Domain

Application

Controller 1

Node Controller 1

Application

Controller Q...

Node Controller 2 Node Controller S

Utilization Targets

Physical Server 1

...VM

App 1

VM

App Q

Physical Server 2

...VM

App 1

VM

App 2

Physical Server S

...VM

App 2

VM

App Q

Resource Allocations

Application

Controller 2

...

...

Performance

Target1

Performance

Target2

Performance

Target S

Ref: Zhikui Wang, Yuan Chen, Daniel Gmach, Sharad Singhal, Brian Watson, Wilson Rivera, Xiaoyuan Zhu, and Chris Hyser. AppRAISE: Application-level

Performance Management in Virtualized

Server Environment. IEEE Transaction on Network and Service Management, Vol. 6, No. 4, pp. 240-254, December 2009.

http://www.hpl.hp.com/personal/Yuan_Chen/appraise.pdf






36

Dynamic Resource Allocation in Virtualized Server Environments Application controller

Application Controller

Feed-Forward

Controller

Feedback

Controller

rref

∆r

Client

r(Mean Response Time)

Node Controller

of VM V1

Node Controller

of VM V2

Node Controller

of VM VM

+

Workload

-

+ ...

Offline Perf.

Models

U

mdlU

refU

refMu

Client

Sensor

refu1

refu2

Ref: Zhikui Wang, Yuan Chen, Daniel Gmach, Sharad Singhal, Brian Watson, Wilson Rivera, Xiaoyuan Zhu, and Chris Hyser. AppRAISE: Application-level Performance Management in Virtualized Server Environment. IEEE Transaction on Network and Service Management, Vol. 6, No. 4, pp. 240-254, December 2009.





37

RUBiS Response Time

Dynamic CPU shares allocated to each component of the application

Dynamic Resource Allocation in Virtualized Server Environments

38

additional resource

number of resources

forecasted base workload

actual workload

workload not exceeding base workload

excess workload

Base Workload Predictor

CoordinatorPredictive Controller

Reactive Controller

Sever Pool 1 Server Pool 2

historical workload demand

Dynamic Capacity Provisioning of Server Farm

Combine predictive and reactive control to allocate resources at multi-time scales

• Identify long-term sustained patterns

• Proactively provision for base workload (every a few hours)

• Reactively handle any excess workload (every a few minutes)

Ref: Minimizing Data Center SLA

Violations and Power Consumption via

Hybrid Resource Provisioning. Anshul

Gandhi, Yuan Chen, Daniel Gmach, Martin

Arlitt, and Manish Marwah. Proceedings of

the Second International Green Computing

Conference (IGCC 2011), July 2011

39

Dynamic Capacity Provisioning of Server Farm


40

Integrated Management of IT and Cooling

– A data center is a dynamic thermal environment

• There are significant non-uniformities

• Some locations in a data center are much more efficiently cooled than others

– A significant opportunity for energy savings lies in the integrated

control and placement of IT workloads placement according to

cooling efficiency


41

Integrated Management of IT and Cooling LWPI: Local Workload Placement Index

CRAC

ventinletMINCRACCRACcracnodeinletrefnode TTSATSATTCITTLWPI )()( ,,

Thermal

Management

Margin

Air Conditioning

Margin

Hot Air

Recirculation

Ref: C. E. Bash and G. Forman, “Data center workload

placement for energy efficiency,” in Proc. of ASME

InterPACK Conf., Vancouver, Canada, 2007.

42

Integrated Management of IT and Cooling High-level architecture

Ref: Yuan Chen, Daniel Gmach, Chris Hyser, Zhikui Wang, Cullen Bash, Christopher Hoover, Sharad Singhal,

Integrated Management of Application Performance, Power and Cooling in Data Centers, in Proceedings of the

12th IEEE/IFIP Network Operations and Management Symposium (NOMS 2010), Apr 19-23, 2010, Osaka, Japan.

Facility Management

CRACSCRACS

IT Management

Pod

Controller

Application

Controllers

VMs Resource

Assignment

Nodes Resource

Consumption

App. Resource

Requests

Application

Measurements

Performance

ModelsDaffy

Floor Plan

Information

DSC

CRACS

Sensors Actuators

Blower Speed

Reference Air

Temperatures

Supply Air

Temperature

Thermal

Metrics

IT Power

State

VMs Resource

Consumption

Temperature

SensorsTemperature

SensorsTemperature

SensorsPhysical Node

VM

App.VM

App.VM

App. ...

Node

Controllers

VMs Resource

Allocation

Power

Control

• Dynamically adjust IT

resources based on needs

• Dynamically re-arrange

workloads

• Opportunistically turn on

and turn off idle servers

• Use LWPI to guide

workload placement

• Adjust cooling resources

in response to IT needs

43


HPL Palo Alto data center

• 20 physical servers

• 9 in Rack 10

• 11 in Rack 34

• 35 Virtual Machines

• 2 interactive 3-tier apps

• 29 computational workloads

• 10 hour experiment

experimental Testbed

44


Number of running servers Workload demand

Server power CRAC blower power

results

45

CDF of response times

Integrated Management of IT and Cooling results

Anticipated Savings Per Year (2MW data center)

Energy Costs CO2 Water

4.9GWh $492,000 2915 tons 4,700,000

Gallons

• 35% IT power reduction • 16% cooling power reduction • No performance degradation

Ref: Yuan Chen, Daniel Gmach, Chris Hyser, Zhikui Wang, Cullen

Bash, Christopher Hoover, Sharad Singhal, Integrated

Management of Application Performance, Power and Cooling

in Data Centers, in Proceedings of the 12th IEEE/IFIP Network

Operations and Management Symposium (NOMS 2010), Apr 19-

23, 2010, Osaka, Japan.

Data center

Outside air

Integrated Management of Supply and Demand HPL Palo Alto Data Center

PV micro grid

Cooling

infrastructure

power demand

Data center supply side Data center energy

demand side

Goals:

• Avoid doing nothing

• “Do nothing” well

• Keep the energy budget balanced

Chill

er

CO

P

Outside Air Temperature (°C) Load (kW)

47

Integrated Management of Supply and Demand

Execution

Supply-side Prediction

•Renewable power prediction •Cooling capacity prediction

IT Demand Prediction

IT Workload Planning • Integrate Supply and Demand Side • IT Demand Shaping • Power capping

Measurement

Verification

DC Operation Constraints: • Net-zero energy operation

• Net-zero costs operation

• Maximize use of renewable energy

• Minimize dependability on Grid

Dynamic IT Provisioning

Dynamic Cooling Provisioning

Prediction Planning

Verification and Reporting

6 12 18 240

0.2

0.4

0.6

0.8

1

1.2

Po

we

r(kW

)

Time(hour)

PV Supply

6 12 18 240

0.5

1

po

we

r (K

W)

time (hour)

critical workload

non-critical workload

cooling power

0 5 10 15 200.1

0.15

0.2

0.25

0.3

0.35

Time (hour)

Po

we

r (k

W)

Outside Air Cooling Available Capacity

6 12 18 240

0.2

0.4

0.6

0.8

1

1.2

po

we

r (K

W)

time (hour)

critical workload

non-critical workload

cooling power

renewable supply

1 2

3 4

48

0

200

400

600

800

1000

Supply-aware Supply-oblivious

En

erg

y (

KW

h)

Renewable Grid

Integrated Management of Supply and Demand

Pla

nnin

g

Imp

act

6 12 18 240

50

100

150

Time (hour)

Pow

er

(KW

)

critical WL

non-critical WL

cooling

renewable

6 12 18 240

50

100

150

Time (hour)

Pow

er

(KW

)

Supply-oblivious Supply-aware

Energy Consumption Cost

0%

20%

40%

60%

80%

100%

Recurring Power Energy Storage

Norm

ailz

ed

Cost

Supply-aware Supply-obivilous

-87% -86%

Work

load S

chedulin

g

Renewable and Cooling Aware Workload Management for Sustainable Data Centers. Zhenhua Liu, Yuan Chen, Cullen Bash, Adam Wierman, Daniel Gmach, Zhikui Wang, Manish Marwah, and Chris Hyser. Proceedings of the 12th International Conference on Measurements and Modeling of Computer Systems (SIGMETRICS 2012), June 2012. (to appear)

Towards The Design And Operation Of Net-Zero Energy Data Centers. Martin Arlitt, Cullen Bash, Sergey Blagodurov, Yuan Chen, Tom Christian, Daniel Gmach, Chris Hyser, Niru Kumari, Zhenhua Liu, Manish Marwah, Alan McReynolds, Chandrakant Patel, Amip Shah, Zhikui Wang, and Rongliang Zhou. Proceedings of the 13th Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm 2012), May 2012.


Digester and Biogas Handling

Power Generation

Cooling Infrastructure

Manure

Biogas

Hot Water

Steam

Power

Data Center

Manure Handling

Dairy Farm

Waste Heat

Effluent Handling and Storage

Sustainable IT Ecosystems Server Farm at the Dairy Farm

Ref: Ratnesh K. Sharma etc, " Design of Farm Waste-driven Supply Side Infrastructure for Data

Centers," Proceedings of the ASME 2010 4th International Conference on Energy Sustainability, May

17-22, 2010.

How could a hypothetical farm of 10,000 dairy cows fulfill the

power requirements of a 1MW data center ?


Sustainable Data Center

Ecosystem of Clients

Data Centers at the Core for management of resources at community scale

Water Transport

Power

Education Healthcare


City 2.0 Enabled by a Sustainable IT Ecosystem

Traditional Ecological Engineering

Life-Cycle Design

…

Power

Transport

Water

Waste

Policy-Based Control & Operation

Knowledge Discovery, Data Mining, Visualization

Scalable & Configurable Resource Microgrids

Pervasive Sensing Infrastructure, Aggregation, Dashboard

Design

Mgmt


L. P.

Research Opportunities and Challenges

End-to-end Optimization Analytics for Sustainability Multi-scale Adaptive Control Integrated Control of Power, Cooling and IT Integrated Supply and Demand Management Beyond IT

53

Acknowledgement


The work presented in this talk has been contributed by the team of Sustainable Ecosystems Research Group at HP Labs, students and professors at CMU and Caltech:

Chandrakant Patel, Cullen Bash, Martin Arlitt, Yuan Chen, Tom Christian, Kiara Corrigan, Daniel Gmach, Chris Hyser, Niru Kumari, Geoff Lyon, Manish Marwah, Alan McReynolds, Amip Shah, Zhikui Wang, Rongliang Zhou, Sharad Singhal, Anshul Gandhi, Zhenhua Liu, Adam Wierman

Documents

Sustainable Data Center · 4 Sustainable Ecosystem Research Group –Sustainable Data Center •Integrated management of IT, power and cooling towards a Net-Zero energy data center