The CHPC is an initiative funded by the Department of Science and Technology of South Africa managed by the Council on Scientific and Industrial Research

The Contribution of High Performance

Computing to the Success of SKA.

• University Researchers • Petrochemical Industries

(SASOL, PetroSA)• Nuclear Industry (NECSA,

NNR, ESKOM)• Mining industry (gold,

platinum, diamond)• Auto manufacturing

(Optimal Energy)• Animation industry• Science Councils• Government departments• Financial sector• Sports• Large Scale Science Projects

(CERN, SKA, Climate Change)

Drivers of HPC demand in S.A.

Computation(Analysis)

Computation(Simulation)

Theory Experiment

3rd and 4th Modes of Research

© CSIR 2010 Slide 4

Definition of Cyberinfrastructure

© CSIR 2010 Slide 5

Cyberinfrastructure Components

Cyberinfrastructure = Hardware + Software + Bandwidth + People

High speed networks and

Software

Data, Data Analysis and Visualisation

Distributed Communities and

Virtual Organisations

Human Capital Development

High Performance Computing

© CSIR 2010 Slide 6

The National Backbone Network

Introducing the CHPC

Centre for High Performance Computing

•The first node is situated at the Rosebank, Cape Town.

•The centre aims to enhance the competitiveness of South Africa through

- High performance Computing

- High-end data curatorship

•It is funded from cyberinfrastructure project, a line-item of the Department of Treasury,

through the Department of Science and Technology. (i.e the centre is of national interest)

Structure

Meraka Centre Director

Meraka Management and Support Teams

Competency Area

Competency Area

Competency Area

Competency Area

Cyberinfrastructure

CHPC/VLDB

and

SANReN

HPC Users in South Africa

2007 2008 2009 20100

50

100

150

200

250

300

350

Over 500 active users

2000: Started as a collaborative HPC initiative among computational scientists.

2001: The DST recognised the potential and importance of this initiative and invited CSIR as a partner.

2004: DST appointed scientific advisory committee and develop business plan.

2005: The CHPC incorporated into CSIR's Meraka Institute.

2006: Refurbishment of Rosebank Campus.

Time-line of Progression

2007: Official launch of the 1st Phase of computational resource and Flagship Projects

2008: Introduced Human Capital Development schemes and invite general users. Implemented Innovative Flagship Projects.

Established a partnership with IBM and host a Blue Gene/p computer to promote the development of HPC in Africa.

Introduced its own research programme – Advanced Computer Engineering Laboratory

Timeline of Progression

2009: Implemented the Phase 2 of computational resource. The SUN hybrid system was ranked on the TOP500 list and Largest in Africa.

2010: Introduced new computer fabrics – a small cluster consists of the General Purpose Graphical Processing Units (A test-bed made of 20 GPGPUs)

Increasing new users, implement new Flagship Project and introduce commercial partners (such as SASOL, a computational engineering consultancy firm and two animation companies)

2011…. Foster more collaboration…….

Timeline of Progression

Each node is equipped with two dual-core AMD Opteron 2.6GHz Rev. F processors (640

CPUs in total at approximately 2.5 Teraflops/s peak performance) and 16GB of DDR2

667MHz random access memory.

The nodes are interconnected with the shared-file system of the SAN, accessed over

the Infiniband 4X SDR 10 GB cluster via HTX from Voltaire and PathScale.

Eight of the cluster nodes are equipped with ClearSpeed accelerator cards

In addition to local hard disks, all nodes have access to a shared storage system with a

capacity of 94TB via a General Parallel File System (GPFS)

Phase 1 Computational Resource

As part of IBM's Global Innovation

Outlook and the IBM-CHPC partnership,

the CHPC hosts a rack of Blue Gene

(BG4A) donated by IBM. The Blue

Gene®/P system is capable of 14 trillion

individual calculations per second, and

is five times more powerful than the

second-fastest research computer on

the African continent (in Egypt). The

Blue Gene®/P provides 1024 compute

nodes, each with four fully cache-

coherent cores and 2GB RAM. The cores

run at 850MHz.

Blue Gene for Africa

Phase 2 Computational Resource

System Name Sun Hapertown Sun Nehalem SMP (M9000-64)CPU Intel Xeon Intel Nehalem Sparc

CPU Clock 3.0 GHz 2.93 GHz 1.9 GHzCPU Cores 384 2304 256 (512 Threads)

RAM 768 GB 3456 GB 2048 GBPeak Performance 3 TFlops 24 TFlops 2 TFlopsInterconnect Jupiter Connector Infiniband Network Infiniband Network

Sun Shared Storage 480 TB (Multi-cluster)

Introducing new technologiesThe CHPC constant watches the trend of development within the areas of HPC and data management. The centre aims to introduce cutting-edge technology and work along side with the communities to facilitate the deployment of these technologies.

For example:General Purpose Graphical Processing Units

Promoting Research and HCD

© CSIR 2010 Slide 19

Stakeholder Engagement

Research and HCD

Technical Advisory Committee

HPC and Technical Support

CHPC Management

Operations and Corporate Affairs

Steering Committee

Scientific Advisory Committee

•Advanced Computer Engineering

•Astronomy/Cosmology

•Bioinformatics

•Epidemiology

•Chemistry

•Computer Science/GRID

•Computational Finance

•Earth Sciences

•Physics

•Materials Science

•Visualisation

•High Education Institutions

•Government Departments

•Science Councils

•Research Agencies

•Industry

•International Collaborators

•Policy Makers

SIGsCHPC Partners

Training Programs (Workshops, Short-Courses ) – > 50 Course to dateAnnual HPC Conference – 4 successful meetingsInternational visitorsIntegration with Main Stream Conferences

SANReN

PartnersCHPC

Partners

Partners

Portals / middleware

Strong Emphasis on Creating Partnership

- Infrastructure and services

- Enhance collaboration (sharing common knowledge/expertise)

- Training initiatives

- Foster Innovation (own research lab and flagship projects)

- Promote commercial usage

Partners

Engagement Model

Data

HPC

Network

Data

HPC

Network

Data

HPC

Network

CSIR

R&D Councils

TEIs

CoEs

Frontier Projects

International Organisations

SET and Education Institutions

Governmental Agencies

Commercial partners

SANReN

Commercial Network

CPT

Site 2

Site 3

NGOs

TEIs

Research FacilitationFlagship Projects relevant to MeerKAT/SKA

- Prof. Marius Potgieter (North-West University)

- Prof. David Davidson (Stellenbosch University)

- Dr. Kavilan Moodley, (UKZN), Prof. Catherine Cress (UWC) and Prof. Bruce Bassett (UCT)

- 26 students and postdoctoral fellows + 19 peer reviewed publications

(8 other Flagship Projects – approx. 50 students and postdoctoral fellows)

© CSIR 2010 Slide 23

Other Flagship Projects

Climate Change Energy SecurityMineral BeneficiationDrug discovery

Fluid Dynamics

Astronomy and Space Physics

HIV MutationBrain ImagingCardio-Vascular

3D Animation

Prof. M.S. Potgieter (North-West University)A series of computational investigation on the cosmic Ray Propagation in the galaxy, cosmic rays in the local interstellar medium, heliospheric properties, space Climate, Space Weather to geospace and neural networks

Professor David Davidson (Stellenbosch University)This project addresses electromagnetic computer simulation of key elements of the proposed Square Kilometre Array.

Dr. Kavilan Moodley, (UKZN), Prof, Catherine Cress (UWC) and Prof Bruce Bassetts (UCT)The project is complemented by four focus areas which collectively aims to gain insight on the evolution of cold gas in galaxies and radio sources in galaxy clusters and estimate key cosmological parameters estimation simulations. Simulations to analyse microwave background data from the Atacama Cosmology Telescope and optimize future supernovae surveys for dark energy constraints were also conducted.

Short Description of these Flagship Projects

Research Collaboration•Apart from facilitating a number of general users to make use of the CHPC infrastructure, the CHPC aims to collaborate with universities and noteworthy organisations such as MeerKAT/SKA, SAAO, SALT, SAC, HMO, Space Agency, and the likes.• Agreements with other Centers around the world

• MSU –RCC, NCSA, NCHC, EPCC, HPCx, Cambridge, Stanford, ESC etc..

The CHPC is developing several own research programmes. For the ACE Lab

Programme, past and current relevant projects include:

Hardware design

Mike Aitken: HTX co-processor card

Jason Salkander: Low cost PCI-express data acquisition board- Astrogig

Software acceleration and design

Andrew Woods: FPGA accelerated Software Radio Astronomy Correlator

Arjun Radhakrishnan: GPU accelerated Pulsar D-dispersion

A number students are currently engage in technologies of interest to the

astronomy communities but applying on other research areas such as

bioinformatics, finance and remote sensing.

Own Research (ACE Lab)

Test-Bed Team

OEM, VendorsTechnology development

Local Industry-Integration

Academia•Research in Novel architecture

Test Bed Teams•HCD

Production System-System Administration and User Support

Provision of HEC (Tier1/0 systems)

USERS

•Academia•Industry•Science councils

Tier2 Facilities

Knowledge TransferThe CHPC hosts a number of joint workshops with our flagship project groups, South African research chairs and leaders within different communities.

For example in the following financial years:2008 – 2009 = 20 workshops

2009 – 2010 = 22 workshops

2010 – 2011 (to date) = 16 workshops

Annual HPC SchoolEach year, the CHPC hosts a HPC school (typically participated by about 50 – 60 students and young researchers) aiming to

- introduce researchers and students to the core skills and advanced topics relevant to HPC.

- enhance the skill transfer between international experts and local communities.

- promote the development of students, particularly those originated from the previously disadvantaged background.

- provide full sponsorship for South African students and partial sponsorship towards the African students.

Towards an integrated Cyberinfrastructure Training School

Apart from the HPC school, the CHPC aims to roll out the following large training schools which cover:

- research domain specific topics

- data curatorship

- Cyberinfrastructure IT courses (HPC, network, data management)

Proposed collaboration

Joint studentship and postdoctoral fellowship on hardware and software development.

Joint training initiatives on HPC and data curation technologies.