How to Make Profit:Exploiting Fluctuating Electricity Prices with Albatross,A Runtime System for Heterogeneous HPC Clusters
Timo Hönig, Christopher Eibel, Adam Wagenhäuser, Maximilian Wagner,
and Wolfgang Schröder-Preikschat
8th International Workshop on Runtime and Operating Systems forSupercomputers (ROSS 2018), Tempe, AZ, USA.
June 12, 2018
Gode Wind Farm: 582 MW North Sea, Europe
Photo: DONG Energy
Solana Generating Station: 280 MW
Arizona, USA
Photo: Abengoa Energy
Solana Generating Station: 280 MW Arizona, USA
Photo: Abengoa Energy, Google
Arizona State University: 24 MW Arizona, USA
Photo: Arizona State University
Building a Runtime System for Heterogeneous HPC Clusters
☼ Reenewable electricity sources havea great impact on the grid.
Availability of renewables leads tofluctuating electricity prices.
How can we design and build an HPC runtimesystem to exploit dynamic electricity pricing?
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 5
Building a Runtime System for Heterogeneous HPC Clusters
☼ Reenewable electricity sources havea great impact on the grid.
Availability of renewables leads tofluctuating electricity prices.
How can we design and build an HPC runtimesystem to exploit dynamic electricity pricing?
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 5
Building a Runtime System for Heterogeneous HPC Clusters
☼ Reenewable electricity sources havea great impact on the grid.
Availability of renewables leads tofluctuating electricity prices.
How can we design and build an HPC runtimesystem to exploit dynamic electricity pricing?
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 5
Energy Mix (Germany, 2017)
Others1 %
Renewables38 %
Natural Gas9 %
Non-renewables62 %
Nuclear Power13 %
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 6
■ non-renewables: 339.93 TWh, renewables: 209.97 TWh
■ dynamic availability of renewables fluctuation electricity prices
Energy Mix (Germany, 2017)
Others1 %
Renewables38 %
Natural Gas9 %
Non-renewables62 %
Nuclear Power13 %
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 6
■ non-renewables: 339.93 TWh, renewables: 209.97 TWh■ dynamic availability of renewables fluctuation electricity prices
Fluctuation of Electricity Prices
Mo Tu We Th Fr Sa Su
20
40
60
80
Time [day of week]
Powe
r[GW
]
-100-80-60-40-20020406080100
Price[EUR/M
Wh]
0
non-renewable electricity sourcesnon-renewable plus renewable electricity sourceselectricity price
[week 43/2017]
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 7
Others1 %
Renewables38 %
Natural Gas9 %
Non-Renewables
Nuclear Power13 %
■ non-renewables can be planned…
■ …but renewables are often unpredictable■ consequence: strong fluctuation of the electricity price■ extreme imbalance of supply & demand negative prices
Fluctuation of Electricity Prices
Mo Tu We Th Fr Sa Su
20
40
60
80
Time [day of week]
Powe
r[GW
]
-100-80-60-40-20020406080100
Price[EUR/M
Wh]
0
non-renewable electricity sourcesnon-renewable plus renewable electricity sourceselectricity price
[week 43/2017]
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 7
Others1 %
Renewables
Natural Gas9 %
Non-Renewables
Nuclear Power13 %
■ non-renewables can be planned…■ …but renewables are often unpredictable
■ consequence: strong fluctuation of the electricity price■ extreme imbalance of supply & demand negative prices
Fluctuation of Electricity Prices
Mo Tu We Th Fr Sa Su
20
40
60
80
Time [day of week]
Powe
r[GW
]
-100-80-60-40-20020406080100
Price[EUR/M
Wh]
negative price
0
non-renewable electricity sourcesnon-renewable plus renewable electricity sourceselectricity price
[week 43/2017]
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 7
■ non-renewables can be planned…■ …but renewables are often unpredictable
■ consequence: strong fluctuation of the electricity price■ extreme imbalance of supply & demand negative prices
Building a System that Benefits from Dynamic Electricity Prices
10:00 11:00 12:00 13:00 14:00 15:00 16:00
0
2
4
6
8
Time [hours]
Powe
rD
eman
d[kW
]
HPC ClusterPower Demand
Excess Demand / Under SupplyPenalty zone I
Target zone
Under Demand / Excess Supply Penalty zone II
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 8
■ challenge: take advantage of low and negative prices■ consider contracts (i.e., penalties enforced by grid operators)■ integrate with existing HPC cluster infrastructure
Building a System that Benefits from Dynamic Electricity Prices
10:00 11:00 12:00 13:00 14:00 15:00 16:00
0
2
4
6
8
Time [hours]
Powe
rD
eman
d[kW
]
HPC ClusterPower Demand
Excess Demand / Under SupplyPenalty zone I
Target zone
Under Demand / Excess Supply Penalty zone II
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 8
■ challenge: take advantage of low and negative prices■ consider contracts (i.e., penalties enforced by grid operators)■ integrate with existing HPC cluster infrastructure
build a flexible runtime system that integrates data onelectricity pricing into its operating decisions
reduce energy demand when prices are high increase energy demand when prices are low (or negative)
Building a System that Benefits from Dynamic Electricity Prices
10:00 11:00 12:00 13:00 14:00 15:00 16:00
0
2
4
6
8
Time [hours]
Powe
rD
eman
d[kW
]
With Albatross
Without Albatross
Excess Demand / Under SupplyPenalty zone I
Target zone
Under Demand / Excess Supply Penalty zone II
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 8
Our proposed runtime system Albatross:
■ implements low- and high-power operation modes■ considers heterogeneity aspects of the cluster hardware■ respects QoS and non-functional requirements of workloads
Albatross: Design and Implementation
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9
■ foobar
Job1
Job2
Jobm
QoSconstraints ...
Electricity & priceconstraints
Master node
Clustermanager
Jobscheduler
Resource group1
Resource groupl
...
Exec1 Execn1 Probe1 Computenode1…
Exec1 Execn2 Probe2 Computenode2…
...
Exec1 Execnk Probek Computenodek…
Powermeters
■ generic system model of an HPC cluster without Albatross
Job1
Job2
Jobm
...
Master node
Clustermanager
Jobscheduler
Execution unit1Compute
node1
Execution unit2Compute
node2...
Execution unitnCompute
noden
Albatross: Design and Implementation
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9
■ extended system model of an HPC cluster with Albatross
Job1
Job2
Jobm
QoSconstraints ...
Electricity & priceconstraints
Master node
Clustermanager
Jobscheduler
Resource group1
Resource groupl
...
Exec1 Execn1 Probe1 Computenode1…
Exec1 Execn2 Probe2 Computenode2…
...
Exec1 Execnk Probek Computenodek…
Powermeters
■ generic system model of an HPC cluster without Albatross
Job1
Job2
Jobm
...
Master node
Clustermanager
Jobscheduler
Execution unit1Compute
node1
Execution unit2Compute
node2...
Execution unitnCompute
noden
Albatross: Design and Implementation
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9
■ extended system model of an HPC cluster with Albatross
Job1
Job2
Jobm
QoSconstraints ...
Electricity & priceconstraints
Master node
Resourcegovernor
Clustermanager
Jobscheduler
Resource group1
Resource groupl
...
Exec1 Execn1 Probe1 Computenode1…
Exec1 Execn2 Probe2 Computenode2…
...
Exec1 Execnk Probek Computenodek…
Powermeters
■ input interfaces for job submission and control■ submission interface for electricity-pricing data■ real-time data as provisioned by grid operator■ detailed job description, added QoS constraints
Albatross: Design and Implementation
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9
■ extended system model of an HPC cluster with Albatross
Job1
Job2
Jobm
QoSconstraints ...
Electricity & priceconstraints
Master node
Resourcegovernor
Clustermanager
Jobscheduler
Resource group1
Resource groupl
...
Exec1 Execn1 Probe1 Computenode1…
Exec1 Execn2 Probe2 Computenode2…
...
Exec1 Execnk Probek Computenodek…
Powermeters
■ power monitoring and control interfaces■ probes operate power-management features (e.g., power capping)■ hardware-specific metering (e.g., RAPL, measuring devices)■ delivery of base data for feedback loop to the resource governor
Albatross: Design and Implementation
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9
■ extended system model of an HPC cluster with Albatross
Job1
Job2
Jobm
QoSconstraints ...
Electricity & priceconstraints
Master node
Resourcegovernor
Clustermanager
Jobscheduler
Resource group1
Resource groupl
...
Exec1 Execn1 Probe1 Computenode1…
Exec1 Execn2 Probe2 Computenode2…
...
Exec1 Execnk Probek Computenodek…
Powermeters
■ runtime control of the HPC system by the resource governor■ job submission based on QoS data and current electricity prices■ control and measurement of power demand for individual jobs■ consider heterogeneity aspects (system, component, and configuration)
Albatross: Design and Implementation
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9
■ extended system model of an HPC cluster with Albatross
Job1
Job2
Jobm
QoSconstraints ...
Electricity & priceconstraints
Master node
Resourcegovernor
Clustermanager
Jobscheduler
Resource group1
Resource groupl
...
Exec1 Execn1 Probe1 Computenode1…
Exec1 Execn2 Probe2 Computenode2…
...
Exec1 Execnk Probek Computenodek…
Powermeters
■ runtime control of the HPC system by the resource governor■ job submission based on QoS data and current electricity prices■ control and measurement of power demand for individual jobs■ consider heterogeneity aspects (system, component, and configuration)
Het
erog
eneit
ySc
ope
System Component
Albatross: Prototype Implementation
Feature Moab Tivoli Univa Slurm
Job pinning ○␣ ○␣ CPU allocation ○␣ Quality of service ○␣ Generic resources ○␣ Cluster status Heterogeneity aware ○␣ ○␣ Power/price aware ○␣ ○␣ ○␣ ○␣
Table: Feature comparison of Albatross with other workload managers.
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 10
■ core of Albatross implementation is based on Slurm■ constraint-aware job-to-node–assignment strategy
Evaluation Overview■ evaluation scenario NAS Parallel Benchmarks (NPB) suite,
Albatross runtime system
heterogeneous cluster with Intel & ARMCPUs, Nvidia & Intel GPUs, and variouspower meters
■ evaluation experiments and goals1. impact of system configuration on
energy demand and execution times2. influence of heterogeneity on the
energy–delay product and power demand3. combination of power and price awareness
to exploit dynamic electricity pricing
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 11
1 MCP39F511
1
3 Current probe
3
2 PCIe riser card
2
4 Red Pitaya
4
C Intel Xeon
C
A / B ARM ODROID-C1+/C2
A B
D Intel HD GPU
D
E Nvidia GPU
E
Evaluation Overview
■ evaluation scenario
NAS Parallel Benchmarks (NPB) suite,Albatross runtime system
heterogeneous cluster with Intel & ARMCPUs, Nvidia & Intel GPUs, and variouspower meters
■ evaluation experiments and goals
1. impact of system configuration onenergy demand and execution times
2. influence of heterogeneity on theenergy–delay product and power demand
3. combination of power and price awarenessto exploit dynamic electricity pricing
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 11
1 MCP39F511
1
3 Current probe
3
2 PCIe riser card
2
4 Red Pitaya
4
C Intel Xeon
C
A / B ARM ODROID-C1+/C2
A B
D Intel HD GPU
D
E Nvidia GPU
E
Experiment 1: Energy Demand and Execution Times
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 12
■ impact of system-level and configuration-level heterogeneity
CG EP FT IS MG0
100
200
Ener
gyD
eman
d[J
]
ODROID-C1+ ODROID-C2 Xeon Xeoncap
ExecutionTime 9.1
s8.0
s0.6
s0.6
s27
.9s
12.6
s1.9
s2.6
s55
.1s
22.8
s1.2
s1.4
s3.4
s1.5
s0.3
s0.3
s14
.8s
7.8s
1.5s
1.5s
■ Xeon and Xeoncap dominate with performance■ energy demand in some cases lower for ARM platforms■ occasional relationship between process energy demand
and execution time
Experiment 1: Energy Demand and Execution Times
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 12
■ impact of system-level and configuration-level heterogeneity
CG EP FT IS MG0
100
200
Ener
gyD
eman
d[J
]
ODROID-C1+ ODROID-C2 Xeon Xeoncap
ExecutionTime 9.1
s8.0
s0.6
s0.6
s27
.9s
12.6
s1.9
s2.6
s55
.1s
22.8
s1.2
s1.4
s3.4
s1.5
s0.3
s0.3
s14
.8s
7.8s
1.5s
1.5s
■ Xeon and Xeoncap dominate with performance■ energy demand in some cases lower for ARM platforms■ occasional relationship between process energy demand
and execution time
Key take-away
race to sleep does not always reduce energy demand knowledge on systems’ impact is absolutely necessary for
operation, requires measurements
Experiment 2: Energy–Delay Product and Power Demand
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 13
■ impact of component-level heterogeneity
0
0.2
0.4
0.6
0.8
1
Norm
alized
EDP
CG EP FT IS MG0
20
40
60
Power
Dem
and
[W]
Xeon Intel HD Nvidia
■ EDP is best for Nvidia for all but one benchmark■ Intel HD has the worst EDP results, but lowest average power
Experiment 2: Energy–Delay Product and Power Demand
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 13
■ impact of component-level heterogeneity
0
0.2
0.4
0.6
0.8
1
Norm
alized
EDP
CG EP FT IS MG0
20
40
60
Power
Dem
and
[W]
Xeon Intel HD Nvidia
■ EDP is best for Nvidia for all but one benchmark■ Intel HD has the worst EDP results, but lowest average power
Key take-away
both, EDP and average power must be considered to balance the cluster’s power drain Albatross trades
average power for EDP (and vice versa)
Experiment 3: Combining Power and Price Awareness
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 14
0 10 20 30 40 50 600
40
80
120
160
Time [min]
Power
Drain
[W]
AlbatrossSLURM
upper limitlower limit
-0.1
0
0.1
Time [day of week]
Cost
[EUR]
-100
-50
0
50
100
Price
[EUR/MW
h]
Th Fr Sa Su
negative price
Albatross
SLURM
price
[week 43/2017]
Conclusion
Albatross, an HPC runtime system thatsuccessfully exploits dynamic electricity pricing.
Albatross…■ considers QoS of jobs and electricity prices for
deploying workloads
■ exploits heterogeneity at system-level,component-level, and configuration-level
■ prepares HPC clusters for future settings whereenergy is just another operating resource
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 15
Discussion
HPDC Poster Session(Wednesday Evening)
Others1 %
Renewables
Natural Gas9 %
Non-Renewables
Nuclear Power13 %
Albatross: Design and Implementation
interface Albatross_Cluster_Control {/* Get cluster configuration */NodeList get_nodes();/* Get all running jobs and their allocation */JobList get_jobs();/* Submit job with attached job−resource info */JobResult submit_job(Job j);
}
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 17
Job1
Job2
Jobm
QoSconstraints ...
Electricity & priceconstraints
Master node
Resourcegovernor
Clustermanager
Jobscheduler
Resource group1
Resource groupl
...
Exec1 Execn1 Probe1 Computenode1…
Exec1 Execn2 Probe2 Computenode2…
...
Exec1 Execnk Probek Computenodek…
Powermeters
Albatross: Design and Implementation
interface Albatross_Power_Control {/* Triggers power measuring for an exec. unit */void start_measurement(Node n, ExecUnit u);PowerValues stop_measurement(Node n, ExecUnit u);
/* Sets the value of a power−management feature */void set_pm(Node n, ExecUnit u, PowerConfig c);
}
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 18
Job1
Job2
Jobm
QoSconstraints ...
Electricity & priceconstraints
Master node
Resourcegovernor
Clustermanager
Jobscheduler
Resource group1
Resource groupl
...
Exec1 Execn1 Probe1 Computenode1…
Exec1 Execn2 Probe2 Computenode2…
...
Exec1 Execnk Probek Computenodek…
Powermeters
Timo Hönig, Christopher Eibel, Adam Wagenhäuser, Maximilian Wagner, andWolfgang Schröder-Preikschat:How to Make Profit: Exploiting Fluctuating Electricity Prices with Albatross,A Runtime System for Heterogeneous HPC ClustersProceedings of the 8th International Workshop on Runtime and OperatingSystems for Supercomputers (ROSS’18), 12 June 2018, Tempe, AZ, USAACM, New York, NY, USA, ISBN: 978-1-4503-5864-4DOI: 10.1145/3217189.3217193
Timo Hönig, Christopher Eibel, Adam Wagenhäuser, Maximilian Wagner, andWolfgang Schröder-Preikschat:Making Profit with ALBATROSS: A Runtime System for HeterogeneousHigh-Performance–Computing ClustersProceedings of the 27th International Symposium on High-Performance Paralleland Distributed Computing (HPDC’18), 11 - 15 June 2018, Tempe, AZ, USAACM, New York, NY, USA, ISBN: 978-1-4503-5899-6DOI: 10.1145/3220192.3220457
https://dl.acm.org/citation.cfm?doid=3217189.3217193https://dl.acm.org/citation.cfm?doid=3217189.3217193http://www.mcs.anl.gov/events/workshops/ross/2018/https://doi.org/10.1145/3217189.3217193https://dl.acm.org/citation.cfm?id=3220457https://dl.acm.org/citation.cfm?id=3220457http://www.hpdc.org/2018/https://doi.org/10.1145/3220192.3220457