Towards energy efficient HPC
HP Apollo 8000 at Cyfronet
Part I
Patryk Lasoń, Marek Magryś
ACC Cyfronet AGH-USTACC Cyfronet AGH-UST
• established in 1973• part of AGH University of Science and
Technology in Krakow, PL• provides free computing resources for scientific
institutions• centre of competence in HPC and Grid Computing• IT service management expertise (ITIL, ISO 20k)• member of PIONIER• operator of Krakow MAN• home for Zeus
International projectsInternational projects
PL-Grid Consortium
• Consortium creation – January 2007• a response to requirements from Polish
scientists
• due to ongoing Grid activities in Europe (EGEE, EGI_DS)
• Aim: significant extension of amount of computing resources provided to the scientific community (start of the PL-Grid Programme)
• Development based on: • projects funded by the European Regional
Development Fund as part of the Innovative Economy Program
• close international collaboration (EGI, ….)
• previous projects (5FP, 6FP, 7FP, EDA…)
• National Network Infrastructure available: Pionier National Project
• computing resources: Top500 list
• Polish scientific communities: ~75% highly rated Polish publications in 5 Communities
•
PL-Grid Consortium members: 5 High Performance Computing Polish Centres, representing Communities, coordinated by ACC
Cyfronet AGH
PL-Grid infrastructurePL-Grid infrastructure
• Polish national IT infrastructure supporting e-Science• based upon resources of most powerful academic resource centres• compatible and interoperable with European Grid• offering grid and cloud computing paradigms• coordinated by Cyfronet
• Benefits for users• one infrastructure instead of 5 separate compute centres• unified access to software, compute and storage resources• non-trivial quality of service
• Challenges• unified monitoring, accounting, security• create environment of cooperation rather than competition
• Federation – the key to success
PLGrid Core projectCompetence Centre in the Field of
Distributed Computing Grid Infrastructures
• Budget: total 104 949 901,16 PLN, including funding from the EC : 89 207 415,99 PLN
• Duration: 01.01.2014 – 31.11.2015
• Project Coordinator: Academic Computer Centre CYFRONET AGH
The main objective of the project is to support the development of ACC Cyfronet AGH as a specialized competence centre in the field of distributed computing infrastructures, with particular emphasis on grid technologies, cloud computing and infrastructures supporting computations on big data.
PLGrid Core project – services
• Basic infrastructure services
• Uniform access to distributed data
• PaaS Cloud for scientists
• Applications maintenance environment of MapReduce type
• End-user services
• Technologies and environments implementing the Open Science paradigm
• Computing environment for interactive processing of scientific data
• Platform for development and execution of large-scale applications organized in a workflow
• Automatic selection of scientific literature
• Environment supporting data farming mass computations
2007 2008 2009 2010 2011 2012
Baribal
Panda
Zeus vSMP
Mars
Zeus
FPGA
Zeus GPU
Zeus
2013
Platon U3
HPC at CyfronetHPC at Cyfronet
ZeusZeus
• over 1300 servers• HP BL2x220c blades• HP BL685c fat nodes (64 cores, 256 GB)• HP BL490c vSMP nodes (up to 768 cores, 6
TB)• HP SL390s GPGPU (2x,8x) nodes• Infiniband QDR (Mellanox+Qlogic)• >3 PB of disk storage (Lustre+GPFS)• Scientific Linux 6, Torque/Moab
Zeus - statisticsZeus - statistics
• 2400 registered users
• >2000 jobs running simultaneously
• >22000 jobs per day
• 96 000 000 computing hours in 2013
• jobs lasting from minutes to weeks
• jobs from 1 core to 4000 cores
CoolingCooling
Rack Rack20°C
40°C 40°C
Hot aisle Hot aisleCold aisle
Why upgrade?Why upgrade?
• Jobs growing
• Users hate queuing
• New users, new requirements
• Technology moving forward
• Power bill staying the same
New buildingNew building
RequirementsRequirements
• Petascale system• Lowest TCO• Energy efficient• Dense• Good MTBF• Hardware:
• core count• memory size• network topology• storage
CoolingCooling
Direct Liquid Cooling!Direct Liquid Cooling!
• Up to 1000x more efficient heat exchange than air
• Less energy needed to move the coolant• Hardware can handle
• CPUs ~70C• memory ~80C
• Hard to cool 100% of HW with liquid• network switches• PSUs
MTBFMTBF
• The less movement the better
• less pumps
• less fans
• less HDDs
• Example
• pump MTBF: 50 000 hrs
• fan MTBF: 50 000 hrs
• 1800 node system MTBF: 7 hrs
The topologyThe topology
576 computing nodes
576 computing nodes
services nodes
Service isle
storage nodes
Computing isle
576 computing nodes
576 computing nodes
Computing isle
576 computing nodes
576 computing nodes
Computing isle
Core IB switches
It should countIt should count
• Max jobsize ~10k cores• Fastest CPUs, but compatible with old codes
• Two sockets are enough• CPUs, not accelerators
• Newest memory• and more than before
• Fast interconnect• still Infiniband• but no need for full CBB fat tree
The hard partThe hard part
• Public institution, public tender
• Strict requirements
• 1.65 PFLOPS, max. 1728 servers
• 128 GB DDR4 per node
• warm water cooling, no pumps inside nodes
• infiniband topology
• compute+cooling, dry-cooler only
• Criteria: price, power, space
And the winner is…And the winner is…
• HP Apollo 8000• Most energy efficient• The only solution with 100% warm water
cooling• Least floor space needed• Lowest TCO
Even more ApolloEven more Apollo
• Focuses also on ‘1’ in PUE!• Power distribution• Less fans• Detailed monitoring
• ‘energy to solution’
• Safer maintenance• Less cables• Prefabricated piping• Simplified management
System configurationSystem configuration
• 1.65 PFLOPS (first 30. of the current Top500)
• 1728 nodes, Intel Haswell E5-2680v3
• 41472 cores, 13824 per island
• 216 TB DDR4 RAM
• PUE ~1.05, 680 kW total power
• 15 racks, 12.99 m2
• System ready for undisruptive upgrade
• Scientific Linux 6 or 7
PrometheusPrometheus
• Created human
• Gave fire to the people
• Accelerated innovation
• Defeated Zeus
Deployment planDeployment plan
• Contract signed on 20.10.2014
• Installation of the primary loop started on 12.11.2014
• First delivery (service island) expected on 24.11.2014
• Apollo piping should arrive before Christmas
• Main delivery in January
• Installation and acceptance in February
• Production work since Q2 2015
Future plansFuture plans
• Benchmarking and Top500 submission
• Evaluation of Scientific Linux 7
• Moving users from the previous system
• Tuning of applications
• Energy-aware scheduling
• First experience presented at HP-CAST 24
[email protected]@cyfronet.pl
More information
•www.cyfronet.krakow.pl/en•www.plgrid.pl/en