1Storage Consortium 2007
1 High Performance Computing Center StuttgartPeter W. Haas
Höchstleistungsrechnenin Forschung und Entwicklung
Peter W. Haas, Michael M. Resch
Anwenderkonferenz des Storage Consortiums14. Februar 2007, Garching
Storage Consortium 20072 High Performance Computing Center Stuttgart
Peter W. Haas
Table of Contents
1. Höchstleistungsrechenzentrum Stuttgart (HLRS)– The HLRS Framework– Key Application Scenarios
2. The Teraflop-Workbench Concept– Concepts and Workflow Examples– NEC SX-8 Configuration
3. Teraflop-Workbench Projects– Methods and Application Areas– Individual Application Projects
4. Terascale Storage and Communication– The BelWü 40-Gbit/s Research Network
3Storage Consortium 2007
3 High Performance Computing Center StuttgartPeter W. Haas
Höchstleistungsrechenzentrum Stuttgart
synergy@work
4Storage Consortium 2007
4 High Performance Computing Center StuttgartPeter W. Haas
The HLRS Framework
hwwhww
Production
Teaching &Research
Computational Science & EngineeringSchool for Mechanical Engineering
Computational Science & EngineeringSchool for Mechanical Engineering
Undergraduate,Graduate & PhDStudents
Chair for Chair for High High Performance Performance ComputingComputingHLRS
Center for Center for Scientific ComputingScientific ComputingUniversity of StuttgartUniversity of Stuttgart
hkzhkz--bwbwStuttgart Stuttgart –– Karlsruhe Karlsruhe -- HeidelbergHeidelberg
Services
Storage Consortium 20075 High Performance Computing Center Stuttgart
Peter W. Haas
Porschenetworks
Networks
networksT-Systemsnetworks
Untertürkheim
4 Gbit/sd ~20 km
Vaihingen
DaimlerChryslernetworks
Universität Stuttgart
n x 10 Gbit/s
100 Mbit/s
1 Gbit/s
Universität Karlsruhe
BelWü40 Gbit/s
Storage Consortium 20076 High Performance Computing Center Stuttgart
Peter W. Haas
Supercomputer
Fileserver
Stuttgart
NECAzusa (16)
HP zx6000 (16)
Appro(40)
NEC SX-6(48)
BullNovaScale (16)
NEC(416)
Cray(256)
IA64 (48) IA32 (446) Opteron (256)
NEC SX-8 (576)
Karlsruhe
HP XC 6000(328)
Ulm
SNU Fire 6800(96)
Storage Consortium 20077 High Performance Computing Center Stuttgart
Peter W. Haas
Medical Application Example
GRID
Storage Consortium 20078 High Performance Computing Center Stuttgart
Peter W. Haas
Ulrich Ray, Director EP/QI, Mercedes Car Group, EDM 2003, July 16-17, 2003
9Storage Consortium 2007
9 High Performance Computing Center StuttgartPeter W. Haas
The Teraflop-Workbench Concept
HLRS - synergy@work
Storage Consortium 200710 High Performance Computing Center Stuttgart
Peter W. Haas
Workbench Concept
File-System
Super-Computer
PP-Server
PP-Server
PP-Server
Vis-ServerAccess
Storage Consortium 200711 High Performance Computing Center Stuttgart
Peter W. Haas
Execute Simulation
If Success
Notify user
Postprocessing
Filtering
Export Data
Create Input Data
Import Data
Preprocessing JobCluster
Client
Compute Task
Workflow Example
Storage Consortium 200712 High Performance Computing Center Stuttgart
Peter W. Haas
Parallel Post-Processing
• Interactive visualization
• Simulation steering
Sim-Cluster Vis-Cluster Compositor
Sim-Cluster Vis-Cluster Compositor
Par.-FS.
Post-Processing + Rendering
Simulation + Post-Processing Post-Processing + Rendering
Storage Consortium 200713 High Performance Computing Center Stuttgart
Peter W. Haas
Requirements
• Coordinated with industry and user community– 50% memory bandwidth dominated
applications -> vector systems– 50% scalability dominated applications -> big
clusters
• 3 TFLOP/s sustained (expected peak of 8-15 TFLOP/s)
• Highly integrated and balanced system• New technology approaches welcome• Close co-operation between vendor and HLRS• Common planning for a 5 year project
Storage Consortium 200714 High Performance Computing Center Stuttgart
Peter W. Haas
Final NEC Configuration at HLRS
SX-8 72 nodes
FC Switch
FC Disk
IXS
114ports
#1 #2
FC Disk
FC Disk
FC Disk
FC Disk
FC Disk
FC Disk
FC Switch114ports
FC Switch114ports
GbEther SW
FC Switch114ports
GbEther SW
SX-8#3
SX-8#1
SX-8#2
SX-8#4
SX-8#1
SX-8#6
SX-8 SX-8#71
SX-8#72#70
GFS Server TX-7
NoconaNoconaNoconaNoconaNocona
NoconaNoconaNocona
Infiniband216 ports
Storage Consortium 200715 High Performance Computing Center Stuttgart
Peter W. Haas
Teraflop WorkbenchVector SMP ClusterIA64 SMP Cluster PC Cluster
SAN• Integrated workbench for preprocessing, simulation, postprocessing
and vizualisation• Choose the best platform for each task• Demonstrate feasibility of workbench approach
Storage Consortium 200716 High Performance Computing Center Stuttgart
Peter W. Haas
NEC SX-8
~ 400 m²~ 750 kW
Overall SystemNumber of Nodes 72IXS Switch 128x128 16 GB/sProcessorsNumber of Processor 576Peak Performance 12,67 TF/sMemorySize of Memory 9,2 TBSpeed of Memory 36,8 TB/s
Storage Consortium 200717 High Performance Computing Center Stuttgart
Peter W. Haas
Current Cisco MDS Configuration
MDS 9509112port
Disk#1
Disk#36
Disk#37
Disk#72
MDS 9509112port
MDS 9509112port
MDS 9509112port
MDS 950664port
MDS 950664port
~ ~
SX#1
SX#72
TX7#1
TX7#1-2・・・
x36 x36 x36 x36
x72 x72 x72 x72x4 x4 x4 x4
x12
x8x8
x12
MDS9506/ 64port x2MDS9509/112port x4
TX7#1
PCCluster
x N x N
Storage Consortium 200718 High Performance Computing Center Stuttgart
Peter W. Haas
Fabric Topology
Disk#1
MDS 9509domain 2
Disk
MDS 9509domain 3
MDS 9509domain 2
MDS 9509domain 3
MDS 9506domain 1
MDS 9506domain 1
SX TX7#1TX7 TX7
#1
PCCluster
Disk#1
Disk
Fabric 1 Fabric 2
SX
Domain number must be uniquewithin a Fabric.VSAN number may be same for all MDS (use VSAN 1 only).
Storage Consortium 200719 High Performance Computing Center Stuttgart
Peter W. Haas
Zoning Configuration (SX-8 host)
MDS 9509Domain 2
MDS 9509Domain 3
SX#1
SX#2
SX#72
Disk#1
Disk#2
Disk#37
Disk#38
Disk#72
Disk#36
Same zone Same zone
Different zone
Different zone
Zone policy of recommendation is One Zone per One HBA.
20Storage Consortium 200720 High Performance Computing Center Stuttgart
Peter W. Haas
Teraflop-WorkbenchProject
HLRS – synergy@work
Storage Consortium 200721 High Performance Computing Center Stuttgart
Peter W. Haas
Areas
• Aeroacoustics• Aerodynamics• Astrophysics• Bioinformatics• Combustion• Fluid-Structure
Interaction• Helicopter Aerodynamics• Nanotechnology• Solid State Physics• Turbo Machinery• Turbulence Phenomena
22Storage Consortium 200722 High Performance Computing Center Stuttgart
Peter W. Haas
FENFLOSS
IHS – University of StuttgartHLRS – synergy@work
Storage Consortium 200723 High Performance Computing Center Stuttgart
Peter W. Haas
Projects: Fenfloss
• IHS, University Stuttgart• Scientific Area: Turbo
Machinery• Method: CFD
•Two and three dimensional on unstructured grids
• Laminar and turbulent flows
• Reynolds-averaged Navier-Stokes
• various turbulence models
• 50% efficiency on single CPU SX-6
• estimated 2TF on SX-8
• Integrated work environment
• Example:
•controllable pitch propeller tailored to a specific ship to reduce noise and avoid cavitation.
Storage Consortium 200724 High Performance Computing Center Stuttgart
Peter W. Haas
Optimization of geometry using Virtual Reality
and „online“ flow simulation
DEMONSTRATION in the HLRS-Cave
Storage Consortium 200725 High Performance Computing Center Stuttgart
Peter W. Haas
26Storage Consortium 200726 High Performance Computing Center Stuttgart
Peter W. Haas
CCARAT
LNM, TU-MunichHLRS – synergy@work
Storage Consortium 200727 High Performance Computing Center Stuttgart
Peter W. Haas
Projects: CCarat – what is it?
• IBS, University of Stuttgart and LNM, TU München • Scientific Area: Fluid Structure Interaction• Method: Finite Element
• Transient interaction of incompressible viscousflows and nonlinear flexible structures
• GLS stabilised fluid elements in 2d and 3d on a moving mesh (ALE formulation)
• Sophisticated structural elements (e.g. shell, brickand beam)
• External solvers (e.g. AZTEC, SPOOLES, UMFPACK and TRILINOS)
• Performance limited due to iterative solver
• Example: 2d simulation of the failure of Tacoma Narrows suspension bridge.
Storage Consortium 200728 High Performance Computing Center Stuttgart
Peter W. Haas
CCarat Simulation of Tacoma Bridge
Simulation IBS, University of Stuttgart
Storage Consortium 200729 High Performance Computing Center Stuttgart
Peter W. Haas
Terascale Storage&
Communication
Storage Consortium 200730 High Performance Computing Center Stuttgart
Peter W. Haas
Unified Heterogeneous File Systems
Shannon Filing Cabinet, Schlicht & Field Co., Rochester, NY, 1886http://www.officemuseum.com
Storage Consortium 200731 High Performance Computing Center Stuttgart
Peter W. Haas
File Systems for Clusters
Symmetric ClusteredFile Systeme.g. GPFS
Lock management Is bottleneck
C C C
C C C
ComponentServer
Parallel File System like PFS
AsymmetricMD Server is bottleneck
ComponentServer
C C C
Server
Distributed File Systeme.g. NFS/CIFS
Server is bottleneck
SAN basedFile Systemslike SANergy
Server is bottleneckScale limited
sanServer
C C C
MetadataServer
Storage Consortium 200732 High Performance Computing Center Stuttgart
Peter W. Haas
Lustre Solution
C C C
OST
Asymmetric Cluster File SystemScalable MDS handles object allocation,
OSTs handle block allocation
OST MDS Cluster
Storage Consortium 200733 High Performance Computing Center Stuttgart
Peter W. Haas
HLRS Data Management Strategy
Backup
100 km
hkz-bw
KarlsruheStuttgart
DataManagement
System
VIS
PPVIS
HSM
PP
HP/CFS Lustre
IBM HPSS
Storage Consortium 200734 High Performance Computing Center Stuttgart
Peter W. Haas
HLRS Storage Configuration
100 kmKarlsruhe Stuttgart
Metadata &Storage Servers
Meta data bases &Storage Servers
SSCK HPC SystemHP XC
HLRS HPC SystemsNEC, Cray, et.al.
FC Network FC Network
...RAID Systems
RAID Systems
System Area Network(10GE, 10GFC, 4 x IB)
...
... ...
Parallel Tapes
Storage Consortium 200735 High Performance Computing Center Stuttgart
Peter W. Haas
10-Gbit/s Fibre Channel Remote VSANs
FC
Cisco MDS 9513With 4 port 10G FC card
In Stuttgart
Cisco MDS 9509With 4 port 10G FC cardand Inter-VSAN-Routing
in Hohenheim
Disk VSAN
Server VSAN
Inter-VSANRouting
10 Gig FC 10 Gig FC
DWDM DWDM
10G λ – 12km
Storage
Server
Feb 23, 2006:
Ralf Eberhardt, CiscoJoachim Thül, ADVAJoseph Michl, UoHPeter Haas, HLRS
Storage Consortium 200736 High Performance Computing Center Stuttgart
Peter W. Haas
Generic Set of Client Interfaces
43 Gbit/soptical:B&W orcolored
43 Gbit/sDe/mux,
Transmitter,Receiver,adaptive
Regenerator
with FEC
43 Gbit/sDe/mux,
...
- 10GE LAN PHY, 10.3G
- SDH STM-64c, 9.9G
- OTN G709, OTU2, 10.7G
- 10GFC, 10.5G or via 10GE
- 4xLX IB, 10.0G or via 10GE
40-Gbit/s Multiplexing Transponder
Storage Consortium 200737 High Performance Computing Center Stuttgart
Peter W. Haas
Transmission Impairment Mitigation
FEC FEC
MUX
and
REGENDEMUX
MUX
and
REGENDEMUX
Analog 40Gbit/slink performance
Regenerated error rate
Corrected error rate
10G Ethernet packet performance
a) End-to-End Monitoring of all Layers . . .
Storage Consortium 200738 High Performance Computing Center Stuttgart
Peter W. Haas
Reduction of TCO
Directly modulated 1550-nm DFB laser diode module
DCF1
43 Gbit/sbinary, NRZ
Laser diode module
1686WMEDFA
ChannelFilter
43 Gbit/sReceiver
104 kmstandard fibre
DCF2
106 kmstandard fibre
43 Gbit/sinput
210 km standard fibre bridged at 43 Gbit/s with optical amplifiers and chromatic dispersion compensation out of Alcatel 1686WM product. Courtesy SEL/Alcatel, Stuttgart
Storage Consortium 200739 High Performance Computing Center Stuttgart
Peter W. Haas
Transparent Switching of 10G Channels
40-Gbit/s Add/Drop Node
4:1MUX
1:4DE
MUX
40GOTX
40GORX
4:1MUX
1:4DE
MUX
40GOTX
40GORX
40 Gbit/soptical interface
40 Gbit/soptical interface
1 2 3 410-Gbit/s add/drop
40 Gbit/swest
40 Gbit/seast10-Gbit/s
transparentmatrix
Storage Consortium 200740 High Performance Computing Center Stuttgart
Peter W. Haas
Thank you very muchThank you very muchfor your attention!for your attention!
Special Special thanksthanks areare duedue to:to:
ADVA ADVA OpticalOptical NetworkingNetworkingAlcatel SEL AG, StuttgartAlcatel SEL AG, StuttgartCiscoCisco SystemsSystemsDeutsches Zentrum fDeutsches Zentrum füür Luftr Luft-- und Raumfahrtund RaumfahrtEuropean HPC Technology CentreEuropean HPC Technology CentreInstitute of Institute of AerodynamicsAerodynamics and and GasdynamicsGasdynamicsInstitute Institute forfor HydraulicHydraulic MachineryMachineryInstitute of Institute of ProcessProcess Engineering and Power Plant TechnologyEngineering and Power Plant TechnologyInstitute of Institute of StructuralStructural MechanicsMechanicsHLRS HLRS VisualizationVisualization DepartmentDepartment