The Center for High Performance Computing

Dell Breakfast Events20th June 2016Happy Sithole

Background: The CHPC in SA

CHPC

User Community: South Africa

CHPC

Existing Users

Future Users

Introduction of CHPC

• CHPC is a national facility within the NICIS providing HPC resources and services to academia and industry.

• A broad range of Science and Engineering domains use HPC.

• The Center is funded by the Department of Science and Technology.

• Building capacity within the country and continent as part of the SKA project.

•Vision

• An accomplished and preferred partner for HPC

solutions in the country and the region

•Mission

• Provide world-class HPC that enable cutting-

edge research with high impact on the South

African economy

CHPC New InfrastructureSystem Configuration Phase 1

Dell PowerEdge C6320 Servers:

Standard Compute nodes 128GB memory/ node 1 008

2 x Intel Xeon E5-2690 v3, 2.6 GHz (Haswell) processors(12 Cores Each 24 cores / node)

24 192

Dell PowerEdge R930 servers:

Large Memory Compute Nodes 1024GB / node (FAT nodes) 5

4 x Intel Xeon E7-4850 v3 processors, 2.2 GHz(14 Cores Each 56 cores / node)

280

Infiniband FDR 2:1 non-Blocking (56 Gbps)

Parallel Storage (Useable) PB 4

Total Number of Racks (including Compute, Login, Management and Storage Nodes)

19

Centos 7.1

Measured Linpack Compute (Tflop/s) 782.9

LENGAU Launch

Lustre Parallel File System@CHPC

Intel Lustre 2.5.4

- Intel Management for Lustre (IML)

- 4PB useable

IOR Results

Future Upgrades• Phase 2 upgrade to be completed by end of October 2016.

• Initial power restriction removed, now 2 MWatts of power available to the datacenter.

• Cooling configuration for the additional servers could be considered (for example onDLC or immersion)

• Processor Roadmap (including accelerators) coupled with Dell Server Roadmap willassist the technology decision for the upgrade.

• Interconnect fabric scalable to 1PFLOP from initial design.

• Storage expansion to be considered in conjunction with archiving options.

• Optimisation of fan speed on the servers is necessary.

Fluent F1 racecar

WRF-3.7, Intel vs gcc

TsessebeInitialisation time = 6.33 sCalculation time = 1033114.91 sFinalisation time = 72.00 sTotal time = 1033193.24 sPeak Memory Use = 710728 kBit took approximately 286 hours ~ 11 days.Overall parallel efficiency rating: Terrible (16%)

LengauInitialisation time = 2.58 sCalculation time = 64277.31 sFinalisation time = 9.00 sTotal time = 64288.88 sPeak Memory Use = 986236 kBIt took approximately 18 hours =3/4 day.Overall parallel efficiency rating: Very good (82%)

Before optimization After optimization after optimization

Email: pngoepe@ul.ac.za

CASTEP LengauTsessebe

1 km grid spacing

60 km grid spacing 8 km grid spacing

• Computational resources determine grid

spacing

• Subgrid-processes parameterised

• Parameterisations – source of uncertainty

Ultra high-resolution climate modelling on

the CHPC clusters

Extending CSIR climate modelling to Africa on the CHPC clusters

Tailor-made projections for Mozambique, Ethiopia, Egypt, Nigeria and Namibia currently running on new CHPC cluster

CCAM applied in stretched-grid mode

8 km simulations nudged in lower resolution CORDEX simulations

1 km resolution simulations over of 200 x 200 km^2 nudged within 8 km res simulations

C160 stretched-grid with 8 km horizontal resolution over a domain of about 1500 x 1500 km^2, centered

over Port Said

Simulation period:

2009-2012

Multiple nudging: ERA

reanalyses

75km to

8km

(1300kmx1300km

domain size)

1km (150kmx150km)

Industry Initiatives

Highly scalable methods for modelling and simulation exploit massive parallelism and data visualisation.

Heterogeneous computing environments require new programming models..

Emerging real-time, data-intensive applications require robust HPC capability.

Energy requirements increase industry overheads.

Tax incentive models from government.

Offerings Applications are made to scale.

HPC is accessible to businesses and applications of all types.

HPC is effectively utilised.

Easier to migrate applications to other HPC systems.

Virtual Prototypes

What are the Drivers?• SASOL• De Beers• Johnson

Matthey• ESKOM• XTRATA• ECJIV• NNR• TOTAL• MINTEK• HATCH

African Data Intensive Research Cloud

Human Capital Development Initiatives

• Annual Winter School targeting new post-graduate researchers and focus on basicsof HPC and induction into HPC facilities, hosted in conjunction with Universitiesaround the country.

• Annual Introductory Scientific Programming School focusing on parallelprogramming and python packaged for post-graduate students.

• Domain focused workshops arranged in collaboration with research communitiesand tailored to meet the needs of the researchers.

• New architecture, such as the heterogeneous computing environments (GP-GPU andXeon-Phi).

• System Administrator training with hands-on configuration on HPC system.Providing administrators with smaller HPC systems that they manage at their owninstitutions.

Student Cluster Challenge• The Student Cluster Challenge is aimed at training

students in HPC.

• The program is focused at undergraduate students, fromall S&E disciplines.

• National selection process starts in June (Winter School)and provides teams that compete in December. Normally10 teams of 4 students compete.

• The winning team in the national competition + 2 bestfrom runner-ups form a team that represent South Africain the International competition.

• The team spends a week at Dell Research Laboratory inAustin in January for further training.

• Dell is the major sponsor with hardware and logistics forboth local competition and international team. Mellanoxprovide the interconnect.

• The students are recruited to different HPC entities in thecountry (For example, CHPC, SKA etc..)

The CHPC Team Wins!!(Added post ISC)

Noteworthy Projects• Dell ICT Academy announced in 2015 through equity equivalence.

Building already completed and the first intake of students (10 highschool graduates and 5 graduates).

• Some of the graduates from the academy will form Dell Channelpartners with strong focus on HPC system integration.

• Focus is on enterprise system and high performance computing.Discussions to include a team from Academy in SC program.

• CHPC Solution Center to focus on testing of new technologies withDell in conjunction with other solutions centers such as CambridgeUniversity and PISA.