View
214
Download
1
Category
Preview:
Citation preview
www.cineca.it
CINECA: the Italian HPC infrastructure and his evolution in the European scenario
Giovanni Erbacci, Supercomputing, Applications and Innovation Department, CINECA, Italy
g.erbacci@cineca.it
www.cineca.it
Agenda
• CINECA: the Italian HPC Infrastructure• CINECA and the Euroepan HPC Infrastructure • Evolution: Parallel Programming Trends in Extremely Scalable Architectures
www.cineca.it
Agenda
• CINECA: the Italian HPC Infrastructure• CINECA and the Euroepan HPC Infrastructure • Evolution: Parallel Programming Trends in Extremely Scalable Architectures
www.cineca.it
CINECA
CINECA non profit Consortium, made up of 50 Italian universities*, The National Institute of Oceanography and Experimental Geophysics - OGS, the CNR (National Research Council), and the Ministry of Education, University and Research (MIUR).
CINECA is the largest Italian computing centre, one of the most important worldwide.The HPC department - manages the HPC infrastructure, - provide support end HPC resources to Italian and European researchers, - promote technology transfer initiatives for industry.
www.cineca.it
1969: CDC 6600 1st system for scientific computing1975: CDC 7600 1st supercomputer1985: Cray X-MP / 4 8 1st vector supercomputer 1989: Cray Y-MP / 4 64 1993: Cray C-90 / 2 128 1994: Cray T3D 64 1st parallel supercomputer1995: Cray T3D 1281998: Cray T3E 256 1st MPP supercomputer2002: IBM SP4 512 1 Teraflops2005: IBM SP5 512 2006: IBM BCX 10 Teraflops2009: IBM SP6 100 Teraflops2012: IBM BG/Q > 1 Petaflops
The StoryThe Story
www.cineca.it
CINECA and Top 500CINECA and Top 500
www.cineca.it
Trend di sviluppoTrend di sviluppo
10 PF
1 PF
100 TF
10 TF
1 TF
100 GF
10 GF
1 GF
>2 PF
www.cineca.it
HPC Infrastructure for Scientific computingHPC Infrastructure for Scientific computing
Logical Name SP6 (Sep 2009) BGP (jan 2010) PLX (2011)
Model IBM P575 IBM BG / P IBM IDATAPLEX
Architecture SMP MPP Linux Cluster
Processor IBM Power 6 4.7 Ghz IBM PowerPC 0,85 GHz Intel Westmere Ec 2.4 Ghz
# of core 5376 40963288 + 548 GPGPU Nvidia Fermi
M2070
# of node 168 32 274
# of rack 12 1 10
Total RAM 20 Tera Byte 2 Tera Byte ~ 13 Tera Byte
Interconnection Qlogic Infiniband DDR 4x IBM 3D Torus Qlogiq QDR 4x
Operating System AIX Suse RedHat
Total Power ~ 800 Kwatts ~ 80 Kwatts ~ 200 Kwatts
Peak Performance > 101 Tera Flops ~ 14 Tera Flops ~ 300 Tera Flops
www.cineca.it
Visualisation system Visualisation system
Visualisation and computer graphycs Virtual Theater6 video-projectors BARCO SIM5 Audio surround systemCylindric screen 9.4x2.7 m, angle 120°Ws + Nvidia cardsRVN nodes on PLX system
www.cineca.it
Storage Infrastructure Storage Infrastructure
System
Available bandwidth
(GB/s) Space (TB)Connection Tecnology
Disk Tecnology
2 x S2A9500 3,2 140 FCP 4Gb/s FC
4 x S2A9500 3,2 140 FCP 4Gb/s FC
6 x DCS9900 5,0 540 FCP 8Gb/s SATA
4 x DCS9900 5,0 720 FCP 4Gb/s SATA
3 x DCS9900 5,0 1500 FCP 4Gb/s SATA
Hitachi Ds 3,2 360 FCP 4Gb/s SATA
3 x SFA1000 10,0 2200 QDR SATA
1 x IBM5100 3,2 66 FCP 8Gb/s FC
> 5,6 PB
www.cineca.it
SP Power6 @ CINECASP Power6 @ CINECA
- 168 compute nodes IBM p575 Power6 - 168 compute nodes IBM p575 Power6 (4.7GHz)(4.7GHz)- 5376 compute cores (32 core / node)- 5376 compute cores (32 core / node)- 128 Gbyte RAM / node (21Tbyte RAM)- 128 Gbyte RAM / node (21Tbyte RAM)- IB x 4 DDR (double data rate) - IB x 4 DDR (double data rate) Peak performance 101 TFlops Peak performance 101 TFlops Rmax 76.41 Tflop/sRmax 76.41 Tflop/s Efficiency (workload) 75.83 % Efficiency (workload) 75.83 % N. 116 Top500 (June 11)N. 116 Top500 (June 11)
- 2 login nodes IBM p560- 2 login nodes IBM p560- 21 I/O + service nodes IBM p520- 21 I/O + service nodes IBM p520- 1.2 PByte Storage row: - 1.2 PByte Storage row:
500 Tbyte working area High 500 Tbyte working area High PerformancePerformance 700 Tbyte data repository700 Tbyte data repository
www.cineca.it
BGP @ CINECABGP @ CINECA
Model: IBM BlueGene / P Architecture: MPPProcessor Type: IBM PowerPC 0,85 GHz Compute Nodes: 1024 (quad core, 4096 total)RAM: 4 GB/compute node (4096 GB total)Internal Network: IBM 3D TorusOS: Linux (login nodes) CNK (compute nodes)
Peak Performance: 14.0 TFlop/s
www.cineca.it
PLX @ CINECAPLX @ CINECAIBM Server dx360M3 – Compute nodeIBM Server dx360M3 – Compute node2 x processori Intel Westmere 6c X5645 2.40GHz 2 x processori Intel Westmere 6c X5645 2.40GHz 12MB Cache, DDR3 1333MHz 80W12MB Cache, DDR3 1333MHz 80W48GB RAM su 12 DIMM 4GB DDR3 1333MHz48GB RAM su 12 DIMM 4GB DDR3 1333MHz1 x HDD 250GB SATA1 x HDD 250GB SATA1 x QDR Infiniband Card 40Gbs1 x QDR Infiniband Card 40Gbs2 x NVIDIA m2070 (m2070q su 10 nodi)2 x NVIDIA m2070 (m2070q su 10 nodi)
Peak performance 32 TFlops Peak performance 32 TFlops (3288 cores a 2.40GHz)(3288 cores a 2.40GHz) Peak performance Peak performance 565 TFlops Single Precision o 565 TFlops Single Precision o 283 TFlops Double Precision (548 Nvidia M2070) 283 TFlops Double Precision (548 Nvidia M2070)
N. 54 Top500 (June 11)N. 54 Top500 (June 11)
www.cineca.it
Science @ CINECAScience @ CINECAScientific AreaChemistry Physics Life Science Engineering AstronomyGeophysicsClimateCultural Heritage
National InstitutionsINFM-CNRSISSA INAF INSTMOGSINGVICTPAcademic Institutions
Main ActivitiesMolecular DynamicsMaterial Science SimulationsCosmology SimulationsGenomic AnalysisGeophysics SimulationsFluid dynamics SimulationsEngineering Applications
Application Code development/ parallelization/optimizationHelp desk and advanced User support Consultancy for scientific software Consultancy and research activities supportScientific visualization support
www.cineca.it
The HPC Model at CINECA From agreements with National Institutions to National HPC Agency in an European context
- Big Science – complex problems - Support Advanced computational science projects - HPC support for computational sciences at National and European level - CINECA calls for advanced National Computational Projects
ISCRA Italian SuperComputing Resource Allocation
http://iscra.cineca.itObjective: support large-scale, computationally intensive projects not possible or productive without terascale, and in future petascale, computing.
Class A: Large Projects (> 300.000 CPUh x project): two calls per Year Class B: Standard Projects. two calls per Year Class C: Test and Development Projects (< 40.000 CPU h x project):
continuous submission scheme; proposals reviewed 4 times per year,
www.cineca.it
ISCRA: Italian SuperComputing Resource Allocation
iscra.cineca.it
Sp6, 80TFlops (5376 core)
N. 116 Top500, June 2011
BGP, 17 TFlops (4096 core) PLX, 142TFlops (3288 core + 548 nVidia M2070)N. 54 Top500 June 2011
- National scientific committee- Blind National Peer review system- Allocation procedure
www.cineca.it
CINECA and Industry CINECA provides HPC service to Industry:
– ENI (geophysics)– BMW-Oracle (American cup, CFD structure)– Arpa (weather forecast, Meteoclimatology)– Dompé (pharmaceutical)
CINECA hosts the ENI HPC system: HP ProLiant SL390s G7 Xeon 6C X5650, Infiniband,
Cluster Linux HP, 15360 cores N. 60 Top 500 (June 2011) 163.43 Tflop/s Peak, 131.2
Linpack
www.cineca.it
CINECA Summer schools
www.cineca.it
Agenda
• CINECA: the Italian HPC Infrastructure• CINECA and the Euroepan HPC Infrastructure • Evolution: Parallel Programming Trends in Extremely Scalable Architectures
www.cineca.it
PRACE
PRACE Research Infrastructure (www.prace-ri.eu): the top level of the European HPC ecosystem
CINECA: - represents Italy in PRACE- hosting member in PRACE - Tier-1 system
> 5 % PLX + SP6 - Tier-0 system in 2012
BG/Q 2 Pflop/s- involved in PRACE 1IP, 2IP- PRACE 2IP prototype EoI
European (PRACE)
Local
Tier 0
Tier 1
Tier 2
National
The European HPC-Ecosystem
Creation of a European HPC ecosystem involving all stakeholders HPC service providers on all tiers Scientific and industrial user communities The European HPC hw and sw industry
www.cineca.it
HPC-Europa 2: Providing access to HPC resources
HPC-Europa 2 - consortium of seven European HPC infrastructures - integrated provision of advanced computational services to the European research community - Provision of transnational access to some of the most powerful HPC facilities in Europe - Opportunities to collaborate with scientists working in related fields at a relevant local research institute
http://www.hpc-europa.eu/
HP-Europa 2: 2009 – 2012
(FP7-INFRASTRUCTURES-2008-1)
www.cineca.it
DEISA PRACE
EUDAT
EMI
HPC-EuropaVerce
MMM@HPC
Plug-it
Europlanet
Vph-op
EESI
HPCworld
VMUST
Deep Montblanc
www.cineca.it
Agenda
• CINECA: the Italian HPC Infrastructure• CINECA and the Euroepan HPC Infrastructure • Evolution: Parallel Programming Trends in Extremely Scalable Architectures
www.cineca.it
BG/Q in CINECAThe Power A2 core has a 64-bit instruction set (unlike the prior 32-bit PowerPC chips used in BG/L and BG/P The A2 core have four threads and has in-order dispatch, execution, and completion instead of out-of-order execution common in many RISC processor designs. The A2 core has 16KB of L1 data cache and another 16KB of L1 instruction cache. Each core also includes a quad-pumped double-precision floating point unit: Each FPU on each core has four pipelines, which can be used to execute scalar floating point instructions, four-wide SIMD instructions, or two-wide complex arithmetic SIMD instructions.
This Power A2 core has a 64-bit instruction set, like other commercial Power-based processors sold by IBM since 1995 but unlike the prior 32-bit PowerPC chips used in prior BlueGene/L and BlueGene/P supercomputers. The A2 core have four threads and has in-order dispatch, execution, and completion instead of out-of-order execution common in many RISC processor designs. The A2 core has 16KB of L1 data cache and another 16KB of L1 instruction cache. Each core also includes a quad-pumped double-precision floating point unit, which is blocked out thus:
16 core chip @ 1.6 GHz
- a crossbar switch links the cores and L2 cache memory together.
- 5D torus interconnect
www.cineca.it
HPC EvolutionMoore’s law is holding, in the number of transistors– Transistors on an ASIC still doubling every 18 months at constant cost– 15 years of exponential clock rate growth has ended
Moore’s Law reinterpreted– Performance improvements are now coming from the increase in the number of cores on a processor (ASIC)– #cores per chip doubles every 18 months instead of clock– 64-512 threads per node will become visible soon
From Herb Sutter<hsutter@microsoft.com>
www.cineca.it
Top500
….
Number of cores of no 1 system from Top500
0
100000
200000
300000
400000
500000
600000
Jun-93
Jun-94
Jun-95
Jun-96
Jun-97
Jun-98
Jun-99
Jun-00
Jun-01
Jun-02
Jun-03
Jun-04
Jun-05
Jun-06
Jun-07
Jun-08
Jun-09
Jun-10
Jun-11
Num
ber
of c
ores Paradigm Change in HPC
What about applications?
Next HPC systems with more than e in the order of 500.000 cores
www.cineca.it
Real HPC Crisis is with SoftwareA supercomputer application and software are usually much more long-lived than a hardware
- Hardware life typically four-five years at most.- Fortran and C are still the main programming models
Programming is stuck- Arguably hasn’t changed so much since the 70’s
Software is a major cost component of modern technologies.- The tradition in HPC system procurement is to assume that the software is free.
It’s time for a change - Complexity is rising dramatically - Challenges for the applications on Petaflop systems - Improvement of existing codes will become complex and partly impossible - The use of O(100K) cores implies dramatic optimization effort - New paradigm as the support of a hundred threads in one node implies new parallelization strategies - Implementation of new parallel programming methods in existing large
applications has not always a promising perspective There is the need for new community codes
www.cineca.it
Roadmap to Exascale(architectural trends)
www.cineca.it
What about parallel App?
In a massively parallel context, an upper limit for the scalability of parallel applications is determined by the fraction of the overall execution time spent in non-scalable operations (Amdahl's law).
maximum speedup tends to 1 / ( 1 − P )
P= parallel fraction
1000000 core
P = 0.999999
serial fraction= 0.000001
www.cineca.it
Programming Models
• Message Passing (MPI)• Shared Memory (OpenMP)• Partitioned Global Address Space Programming (PGAS)
Languages UPC, Coarray Fortran, Titanium
• Next Generation Programming Languages and Models Chapel, X10, Fortress
• Languages and Paradigm for Hardware Accelerators CUDA, OpenCL
• Hybrid: MPI + OpenMP + CUDA/OpenCL
www.cineca.it
trends
Vector
Distributed memory
Shared Memory
Hybrid codes
MPP System, Message Passing: MPI
Multi core nodes: OpenMP
Accelerator (GPGPU, FPGA): Cuda, OpenCL
Scalar Application
www.cineca.it
Message Passingdomain decomposition
memory
CPU
node
memory
CPU
node
memory
CPU
node
memory
CPU
node
memory
CPU
node
memory
CPU
node
Internal High Performance Network
www.cineca.it
Ghost Cells - Data exchange
i,j i+1,ji-1,j
i,j+1
i,j-1
sub-domain boundaries
Ghost Cells
i,j i+1,ji-1,j
i,j+1
i,j-1
Processor 1
Processor 2
i,j i+1,ji-1,j
i,j+1
Ghost Cells exchangedbetween processors at every update
i,j i+1,ji-1,j
i,j+1
i,j-1
Processor 1
Processor 2
i,j i+1,ji-1,j
i,j+1
www.cineca.it
Message Passing: MPI
Main Characteristic• Library• Coarse grain• Inter node parallelization
(few real alternative)• Domain partition• Distributed Memory• Almost all HPC parallel
App
Open Issue• Latency• OS jitter• Scalability
www.cineca.it
Shared memory
mem
ory
CPU
node
CPU
CPU
CPU
Thread 0
Thread 1
Thread 2
Thread 3x
y
www.cineca.it
Shared Memory: OpenMP
Main Characteristic• Compiler directives• Medium grain• Intra node parallelization (pthreads)• Loop or iteration partition• Shared memory• Many HPC App
Open Issue• Thread creation overhead• Memory/core affinity• Interface with MPI
www.cineca.it
OpenMP !$omp parallel dodo i = 1 , nsl call 1DFFT along z ( f [ offset( threadid ) ] )end do!$omp end parallel docall fw_scatter ( . . . )!$omp paralleldo i = 1 , nzl !$omp parallel do do j = 1 , Nx call 1DFFT along y ( f [ offset( threadid ) ] ) end do!$omp parallel do do j = 1, Ny call 1DFFT along x ( f [ offset( threadid ) ] ) end doend do!$omp end parallel
www.cineca.it
Accelerator/GPGPU
Sum of 1D array
+
www.cineca.it
CUDA samplevoid CPUCode( int* input1, int* input2, int* output, int length) { for ( int i = 0; i < length; ++i ) { output[ i ] = input1[ i ] + input2[ i ]; }}
__global__void GPUCode( int* input1, int*input2, int* output, int length) { int idx = blockDim.x * blockIdx.x + threadIdx.x; if ( idx < length ) { output[ idx ] = input1[ idx ] + input2[ idx ]; }}
Each thread execute one loop iteration
www.cineca.it
CUDAOpenCL
Main Characteristic• Ad-hoc compiler• Fine grain• offload parallelization (GPU)• Single iteration parallelization• Ad-hoc memory• Few HPC App
Open Issue• Memory copy• Standard• Tools• Integration with other
languages
www.cineca.it
Hybrid (MPI+OpenMP+CUDA+…
Take the positive off all modelsExploit memory hierarchyMany HPC applications are adopting this modelMainly due to developer inertiaHard to rewrite million of source lines
…+python)
www.cineca.it
Hybrid parallel programming
MPI: Domain partition
OpenMP: External loop partition
CUDA: assign inner loopsIteration to GPU threads
http://www.qe-forge.org/ Quantum ESPRESSO
Python: Ensemble simulations
www.cineca.it
Storage I/O
• The I/O subsystem is not keeping the pace with CPU
• Checkpointing will not be possible
• Reduce I/O• On the fly analysis and
statistics• Disk only for archiving• Scratch on non volatile
memory (“close to RAM”)
www.cineca.it
Conclusion
• Exploit millions of ALU• Hybrid Hardware• Hybrid codes• Memory Hierarchy• Flops/Watt (more that Flops/Sec)• I/O subsystem• Non volatile memory• Fault Tolerance!
Parallel programming trends in extremely scalable architectures
Recommended