Interconnect Your Future with Mellanox InfiniBand

Paving the Road to Exascale

April 2016

Interconnect Your Future

© 2016 Mellanox Technologies 2

Co-Design Architecture to Enable Exascale Performance

CPU-Centric Co-Design

Limited to Main CPU Usage

Results in Performance Limitation

Creating Synergies

Enables Higher Performance and Scale

Software

In-CPU

ComputingIn-Network

Computing

In-Storage

Computing


Breaking the Application Latency Wall

Today: Network device latencies are on the order of 100 nanoseconds

Challenge: Enabling the next order of magnitude improvement in application performance

Solution: Creating synergies between software and hardware – intelligent interconnect

Intelligent Interconnect Paves the Road to Exascale Performance

10 years ago

~10

microsecond

~100

microsecond

NetworkCommunication

Framework

Today

~10

microsecond

Communication

Framework

~0.1

microsecond

Network

~1

microsecond

Communication

Framework

Future

~0.05

microsecond

Co-Design

Network


High-Performance Intelligent 100Gb/s Interconnect Solutions

Transceivers

Active Optical and Copper Cables

(10 / 25 / 40 / 50 / 56 / 100Gb/s)VCSELs, Silicon Photonics and Copper

36 EDR (100Gb/s) Ports, <90ns Latency

Throughput of 7.2Tb/s

7.02 Billion msg/sec (195M msg/sec/port)

100Gb/s Adapter, 0.7us latency

150 million messages per second

(10 / 25 / 40 / 50 / 56 / 100Gb/s)

32 100GbE Ports, 64 25/50GbE Ports

(10 / 25 / 40 / 50 / 100GbE)

Throughput of 6.4Tb/s


InfiniBand Architecture

InfiniBand Router Answers The Needs of Next Generation Scalable Data Centers

Native InfiniBand Connectivity Between

Different InfiniBand Subnets (Each

Subnet can Include 40K nodes)

Isolation Between Different InfiniBand

Networks (Each Network can be

Managed Separately)

Native InfiniBand Connectivity Between

Different Network Topologies (Fat-Tree,

Torus, Dragonfly, etc.)


Introducing: InfiniBand Router Solution

SB7780 Router 1U

Supports up to 6 Different Subnets


NCAR-Wyoming Supercomputing Center (NWSC) – “Cheyenne”

Cheyenne supercomputer system

5.34-petaflop SGI ICE XA Cluster

Intel “Broadwell” processors

More than 4K compute nodes

Mellanox EDR InfiniBand interconnect

Mellanox Unified Fabric Manager

Partial 9D Enhanced Hypercube interconnect topology

DDN SFA14KX systems

20 petabytes of usable file system space

IBM GPFS (General Parallel File System)


The Performance Advantage of EDR 100G InfiniBand (28-80%)

28%


Interconnect Architecture Comparison

Offload versus Onload (Non-Offload)


Offload versus Onload (Non-Offload)

Two interconnect architectures exist – Offload-based and Onload-based

Offload Architecture

• The Interconnect manages and executes all network operations

• The interconnect is capable of including application acceleration engines

• Offloads the CPU and therefore free CPU cycles to be used by the applications

• Development requires large R&D investment

• Higher data center ROI

Onload architecture

• A CPU-centric approach – everything must be executed on and by the CPU

• The CPU is responsible for all network functions, the interconnect only pushes the data into the wire

• Cannot support acceleration engines, no support for RDMA, and network transport is done by the CPU

• Onload the CPU and reduces the CPU cycles available for the applications

• Does not require R&D investments or interconnect expertise


Claims Used to “Market” Onload Architecture – “Too Many Cores”

Claim: There are many CPU cores, and the applications cannot use them all, so one can

dedicate some cores to manage the interconnect operations

Reality: False claim

• CPU vendors increase the CPU core count due to applications requirements!

• In cases where applications require less core, data center owners can buy the needed core count

• CPUs with less core are dramatically cheaper! Why would one spend more $ if not needed?

Intel Haswell CPU

10-Cores

Intel Haswell CPU

12-Cores

Intel Haswell CPU

14-CoresConclusions

CPU cost: $1502 CPU cost: $2170 CPU cost: $3003CPU cores cost more

than the interconnect!

CPU cores are not

free!

12-Core to 10-Core

Difference: $668

14-Core to 12-Core

Difference: $833

Dual Socket Server

Difference: $1336

Dual Socket Server

Difference: $1666


Application Performance Comparison – WEIN2K

WIEN2k is a Quantum

Mechanical Simulation

Higher is better

InfiniBand Delivers 35% Higher Performance With Only 10 Nodes


Application Performance Comparison – WEIN2K

WIEN2k is a Quantum

Mechanical Simulation

Higher is better

FDR and EDR InfiniBand Deliver Higher Performance


Application Performance Comparison – Quantum ESPRESSO

Higher is better

InfiniBand Delivers 42% Higher Performance With Only 10 Nodes

Quantum ESPRESSO is

an electronic structure

and materials modeling

Simulation


Application Performance Comparison – LS-DYNA

InfiniBand Delivers 48-63% Higher Performance With Only 12 Nodes

Omni-Path Does Not Scale Beyond 10 Nodes

A structural and fluid analysis software, used for automotive, aerospace, manufacturing simulations and more

Higher is better Higher is better


Maximizing Data Center Return on Investment (ROI)

No Loss of CPU Efficiency (Offloads)

Compute systems cost: $1M

InfiniBand: $0.15M

Total cost: $1.15M

CPU overhead: 0%

Compute worth (ROI, higher is better): $1M

For compute worth of $0.7M, total cost would be $0.8M!

20% lower cost for the same performance!

30% Loss of CPU Efficiency (no-offloads)

Compute systems cost: $1M

Omni-Path: $0

Total cost: $1M

CPU overhead: 30%

Compute worth (ROI, higher is better): $0.7M

App App App App App App

AppApp

App App

AppApp

CPU CPU CPU CPU

Why do users prefer to pay

for Mellanox interconnect

even if Omni-Path would

have been provided

free of charge?


Technology Roadmap – One-Generation Lead over the Competition

2000 202020102005

“Roadrunner”Mellanox Connected

1st3rd

TOP500 2003Virginia Tech (Apple)

2015

Terascale Petascale Exascale

10G 20G 40G 56G 100G 200G 400G

TrueScale

No-Offload Network

40G (InfiniBand)

Same PathScale Technology

Under Intel Logo

Infinipath

No-Offload Network

20G (InfiniBand)

Omni-Path

No-Offload Network

100G (Proprietary)

Same PathScale Technology

Under QLogic Logo


Omni-Path Performance Claims?

ISC’15 July 2015

160M messages per Second

Linley Group Conference

February 2016

108M messages per second

IDF China April 2016

79-105M messages per Second


InfiniBand Delivers Highest Applications Performance

Performance Advantage of InfiniBand Increases with System Size

InfiniBand the Only Interconnect that Scales with System Size

InfiniBand Enables Higher System Performance with 50% of the Servers

Multiple Application Cases Demonstrate 35-63% Performance Advantage

Thank You