Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Simulation Strategies for Massively Parallel
Supercomputer Design
Authored by:Ansoft Corporation
Ansoft 2003 / Global Seminars: Delivering PerformancePresentation #2
Special Thanks to:Cray
Introduction
• Cray: Red Storm Supercomputer– Sandia National Laboratories awarded Cray Inc. a multiyear contract to develop and
deliver a new massively parallel processing (MPP) supercomputer called Red Storm. The computer will use 10,000 Advanced Micro Devices Inc. Opteron™ processors connected via a high-bandwidth, three-dimensional mesh interconnect network.
Introduction
• About Cray– Approximately 850 employees worldwide
– Corporate headquarters: Seattle, WA
– 3 major engineering centers: • Chippewa Falls, WI, • Mendota Heights, MN, • Seattle, WA
– NASDAQ: CRAY
Introduction
• Red Storm: System Overview– Theoretical peak performance: 40 trillion calculations per second– 10,368 Compute Nodes: AMD 64 bit Opteron™ processors
• Connected via a low-latency, high-bandwidth, three-dimensional mesh interconnect network based on HyperTransport™ technology
– Approximately 3000 ft² including disk systems
Introduction
• Red Storm: High Speed Network (HSN)– 3D Mesh that interconnects all of the compute nodes
• 27 x 16 x 24 (x, y, z) mesh• High-Speed Serial Link• Nominal Data Rate: 3.2Gbps
PCI - X
+Y
-Y
+Z-X
+X-Z
+Y
-Y
+Z-X
+X-Z
Compute NodeCompute Node
High Speed Network(HSN)
High Speed Network(HSN)
Introduction
• NEC Earth Simulator– Performance: 40Tflops– Processor: NEC .15um vector CPU– Date: 1997-2002
– Cost: $450M– Development Schedule: >54 months
• Cray Red Storm– Performance: 40Tflops– Processor: AMD Opteron™– Date: 2002-2004
– Cost: $90M– Development Schedule: 26 months
$$º
$$º
Custom HardwareCustom
HardwareSystem
IntegrationSystem
Integration
Introduction• Relative “Cost” of Finding Hardware Design Problems
– “Cost” = “Pain” = $$$, Time to Market, Your Job, etc.
Detailed DesignDetailed Design IntegrationIntegration ValidationValidation OperationOperationPreliminary Design
Preliminary Design
1
2
5
10
20
50
100
Software Test and Measurement
Introduction
• Designing for High-Speed– Difficult Aspects
• As Speed increases, luck decreases– Large number of codependent terms
» They are not always controllable/understood – random variation
• New effects– Large Systems composed of many sub-systems
» Variables that could be ignored in the past must be known to a very high precision» Signal Channel Management – How do we account for and manage information?
• New techniques– At high-speeds: Signal Integrity Engineering = Microwave Engineering
» New Design Flows» New Techniques and Terms: Frequency Domain vs. Time-Domain» New Tools: Harmonic Balance, Quasi-Static, Full-Wave, etc.» New Models: 2D and 3D Physical Device Models» Model Abstraction
“Cost” Increases SPEED
LU
CK
A. Fraser, S. Argyrakis, “Does Signal Integrity Engineering have a Future”, DesignCon 2003,.
Introduction• Designing for High-Speed
– Reverse the trend• Decrease “Cost”: Move more Integration and Validation into early design
stages. Virtual Prototypes!• Stop relying on Luck: Better models, techniques, and tools increase the
probability of first past success.– Microwave Engineers have been using these techniques for over a decade
Introduction• Virtual Prototypes
– Full System & Sub-SystemsFull System
Sub-System - Routing
Sub-System - Transitions
Sub-System - Connectors
Sub-System - Packaging
Sub-System – Board/Stackup
Sub-System – Daughter Card
Introduction• Channel Management
– Challenge: Move Integration and Validation into Virtual Prototype System
ChannelModel
Management
Connectors
Boards
Packaging
Isolation
TransitionsPower Delivery
Vias
Modes
BandwidthCross-Talk
Layout
BERLoss
Skin Effect
Eye Diagram
ISI
Impedance
Load
Source
3D Models SPICE Models
2D Models
Frequency Dependence
TDR
Delay
Introduction• Channel Management
– Common Design Environment/Integrated Database• Solver on Demand
– Circuit: Transient/Linear/Non-Linear Harmonic Balance– System: Mixed Mode Analysis - Baseband-through-RF
– Planar EM: 2.5D Full-Wave Method of Moments
– 3D Full-Wave: HFSS v9 Finite Elements (Solver on Demand, Now in Ansoft Designer 1.1)
– 3D Quasi-Static: Spicelink Boundary Elements (Solver on Demand, Version 6.0 coming soon)
• Solver on Demand - Information Hiding– Prevents higher levels of design from becoming dependent on low-level details such as
3D Physical Device Modeling.
Ansoft DesignEnvironmentChannel Manager
Circuit
System
Planar EM
3DLayoutAnsoftLinks
MechanicalCAD
DXF/GDSII
SPICE
Matlab
C code
Introduction• Why are better models, techniques, and tools needed?
– Speed = Problems• Evolution of a short circuit
– Once interconnects stop behaving as transmission lines, SPICE models and SPICE like tools can not predict performance
SPEED
A. Fraser, S. Argyrakis, “Does Signal Integrity Engineering have a Future”, DesignCon 2003,.
Introduction• Why are better models, techniques, and tools needed?
– Co-dependent terms• Example: As speed increases, the connector performance begins to depend on the board
integration.– Adopting new models, techniques, and tools that can identify these co-dependent
performance factors reduces the probability of discovering hardware problems late in the product development cycle
» Remember: The possibility of uncontrollable or unforeseen variables can still appear
?
Introduction
• What are these uncontrollable or unforeseen variables?– Virtual Prototypes are abstractions
• They only contain the essential details of a complex system– Essential Details = Those that are critical to the electrical performance– Model Abstraction efficiently uses limited computer resources and product
development time– Example: Cavity filter designers routinely use screws to tune the filter and
account for manufacturing variations. When they simulate their filter designs they would not include the threads on the screw. The threads are essential mechanical details, not electrical details
– Manufacturing Process Variations • Example: If the virtual prototype does not account for the substrate
thickness shrinking because of thermal effects in the manufacturing process, you will not predict the performance correctly.
Introduction
• Ansoft and Cray– Ansoft: Provide End-to-End Simulations of HSN Channel
• Five different classes of simulations / analysis1. PCB/Interconnects
– Mezzanine, Module, Backplane, and Red/Black Switch
2. Connectors– NexLev, GbX, and VHDM
3. Cabling– Self-Equalizing Twin-Ax (1.1m - 8m)
4. Packaging– HyperBGA – High Performance Organic Flip-Chip BGA
5. System– Frequency and Time Based Performance Extraction
– Cray: Provide• Electrical Specifications• Electrical Models• Mechanical Models• Board Layouts
• Red Storm: HSN Physical ConfigurationIntroduction
BackplaneBackplane
ComputeBoard
ComputeBoard
VHDM Connector
VHDM Connector
GbXConnector
GbXConnector
SerDesASIC
SerDesASIC
AMD 64 bit Opteron™
AMD 64 bit Opteron™
Introduction• Red Storm: HSN Electrical Configuration
HyperBGA+
Mezzanine Board
ModuleBoard Backplane Red/Black
Switch
Connector Connector ConnectorCable
Introduction• Red Storm: HSN Electrical Configuration
ModuleBoard
TeradyneGbX
Connector
BackplaneBoard
MolexVHDM
Connector
MolexTwin-axCable
Red/BlackSwitch
TeradyneGbX
Connector
MolexVHDM
Connector
MolexTwin-axCable
SerDesHyperBGA
TeradyneNexlev
Connector
MezzanineBoard
Still in Model Development
PCB/Interconnects• Module Board
YP3_FMBP{18,19}32mm
YM0_TOBP{18,19}28mm
GBXNexLev
NexLev GBX
PCB/Interconnects• Module Board
System(Frequency or Time Based Analysis)
Circuit Solver On Demand Planar EM
Speed AccuracyChoose the level of speed and accuracy
Planar EM – Coupled Bend
Port1
Port2 Port3
Port4
Port1
P1
S=9.11milW=5milP=376mil
S=9.11milW=5milP=716mil
W18_EM_SLCBENDS27
S=9.11milW=5mil
P=401mil
W20_EM_SLCBENDS28
W21
_EM_SLCBENDS29
S=9.11milW=5mil
P=645mil
S=9.11milW=5mil
P=3095mil
S=9.11milW=5milP=293mil
Por t 1
Por t 2
Por t 3
Por t 4U3
PlanarEM7
PCB/Interconnects
• Backplane
P1BYP3_FMBP{18,19}217mm
P1BYM0_TOBP{18,19}291mm
PCB/Interconnects
• Backplane – Spicelink 2D– Layer Height (B): 0.27178 mm (10.7 mil)– Trace Width (W): 0.125 mm– Trace Separation (S): 0.25 mm– Trace Thickness: 0.5 Oz Copper (0.7 mil)
BS W
εr = 3.4, tanδ = 0.006
Layer B W S Zse Zd Zcom
S1/ S10 0.272 0.125 0.250 49.15 96.05 25.13All Dimensions are in mm
PCB/Interconnects
• Backplane Routing – Via Stub– In the link the GbX and VHDM
will contain a best and worst case via stub
Route Layer: s10(Via Stub: 10.75mil)
Route Layer: s1(Via Stub: 123.95mil)
VHDM Connector VHDM Connector
Bes
t Cas
e
Wor
st C
ase
Backplane
VHDM
GbX
PCB/Interconnects
• Backplane Routing – Via Stub
These results do not include loss
PCB/Interconnects
• Backplane Routing – Anti-pad
Antipad Radius: 0.5mm
(Layout)
Antipad Radius: 0.7mm
PCB/Interconnects• Backplane Routing – Anti-pad (Layer: S1)
These results do not include loss
PCB/Interconnects
• Backplane Routing – Anti-pad (Layer: S10)
These results do not include loss
• Teradyne’s GbXadvanced performance interconnect provides the highest density optimized differential connector available today.
– Delivering data rates greater than 5 Gb/s.
– High Density: GbXprovides up to 55 pairs per linear inch (4-pair configuration).
– Reliability: Two points of contact at a separable interface.
– Flexibility: Choice of density configurations (3, 4 and 5-pair) for higher application flexibility.
– Vertical and Horizontal Routing make GbX the ideal solution for star or mesh backplane design.
HFSS side view
bottom view
GbX Connector
Connectors
• GbX– All links contain 2 backplane sections
• One channel outbound from SerDes ASIC.• One channel inbound to SerDes ASIC.
– GbX models encapsulate connectors and escape vias/routing• Connector performance is very dependent on board interface.• Interface is critically dependent on board metrics:
– route layer– via stub length– antipad dimensions– board materials
• Escape routing is different on the outbound and inbound channels.
GbX
VHDM
VHDM
To ASIC from backplane
From ASIC to backplane
GbX
Backplane
Module
Module
Connectors• Models are generated separately for the GbX components. Each channel includes models
for:1. Backplane board escape routing, with adjacent pins.2. GbX connector with single wafer.3. Module board escape routing, with adjacent pins.
• Different levels of complexity were retained initially for the escape routing.– “From Backplane” routing will be used to determine what level of complexity is necessary.
To Backplane
From Backplane
+Complexity
-
+Complexity
-
backplane escape routing
GbX connector
module escape routing
Connectors• VHDM
– Very High Density Matrix
Connectors
Connectors• VHDM - Backplane
backplane escape routing
VHDM connector
twin-ax cable feed
Connectors• Red/Black switch allows supercomputer to be physically
divided for secure (classified) processing– Red/Black switch is two VHDM-HSD connectors in a back-to-back configuration– A center-plane circuit board provides support for the back-to-back configuration
HFSS model
Cable
• Gore Twin-Ax– 100 differential– “Self Equalization”
Cable
• Self Equalization– Attenuation increases with sqrt(f) due to conductor skin effects
• Higher frequency components attenuations >> fundamental frequency– Increased jitter and inter-symbol interference– Limits length of cable
• Dielectric loss vary directly with frequency– Low loss dielectric
– Cable Equalization• Produces a near linear attenuation response vs. frequency• Use different skin depth properties of conducting materials
– Base material has low conductivity and/or high permeability» Coat with a good conductor
Cable
Standard Cable
Equalized
Package
• HyperBGA– High Performance Organic Package– Flip Chip
Packaging
Packaging
Conclusions• Cray and Ansoft Corporation are in collaboration to verify for the 3.2Gb/s serial data channel of the Cray
Red Storm Supercomputer high-bandwidth, three-dimensional mesh interconnect network.– Cray recognized the value of electromagnetic-based simulation to ensure reliable supercomputer
performance.
• This presentation showed how a combination of electromagnetic field simulation coupled with circuit and system simulation was used to predict the interconnect performance.
– The successful/accurate characterization of the system was made possible by utilizing:• Electromagnetics based analyses software
– Circuit/System Level» Ansoft Designer
– Passive Physical Device Modeling» Ansoft HFSS » Ansoft Designer» Ansoft SpiceLink» Ansoft Optimetrics
• Modern high-speed designs are requiring engineers to achieve new levels of technological advances. – The methodologies introduced here show how to systematically reduce a complex system to a
solvable problem.– This structured procedure breaks the design-build-redesign loop commonly found in the old
methodology of addressing problems after signal integrity errors are encountered.