View
128
Download
6
Category
Preview:
Citation preview
© 2019 Mellanox Technologies | Confidential 1
Paz BarkanSep 2019
BlueField Overview
© 2019 Mellanox Technologies | Confidential 2
Safe Harbor StatementThese slides and the accompanying oral presentation contain forward-looking statements and information.
The use of words such as “may”, “might”, “will”, “should”, “expect”, “plan”, “anticipate”, “believe”, “estimate”, “project”, “intend”, “future”, “potential” or “continued”, and other similar expressions are intended to identify forward-looking statements.
All of these forward-looking statements are based on estimates and assumptions by our management that, although we believe to be reasonable, are inherently uncertain. Forward-looking statements involve risks and uncertainties, including, but not limited to, economic, competitive, governmental and technological factors outside of our control, that may cause our business, industry, strategy or actual results to differ materially from the forward-looking statement.
These risks and uncertainties may include those discussed under the heading “Risk Factors” in the Company’s most recent 10K and 10Qs on file with the Securities and Exchange Commission, and other factors which may not be known to us. Any forward-looking statement speaks only as of its date. We undertake no obligation to publicly update or revise any forward-looking statement, whether as a result of new information, future events or otherwise, except as required by law.
© 2019 Mellanox Technologies | Confidential 3
BlueField – Field of Advantages
BlueField – The Premier I/O Processing Unit Solution
© 2019 Mellanox Technologies | Confidential 4
Storage
BlueField Enables Multiple Market Segments
Security
NFV & Cloud
▪ Enabling NVMe-over-Fabrics storage solutions
▪ Enhancing software-defined and hyperconverged storage solutions
▪ Enabling secured and isolated cloud solutions
▪ New data center security models
▪ Enabling accelerated NFV host agnostics solutions through SmartNIC
▪ Offloading OVS control & data planes
Edge Computing
▪ Enabling mini-servers at the network edge
▪ Involve superior network capabilities
HPC, Machine Learning▪ Co-processor for privacy and proprietary advanced algorithms
▪ InfiniBand storage solutions
© 2019 Mellanox Technologies | Confidential 5
BlueField - The Ultimate Co-Processor
Host-agnostic network solution
Integrated Control & Data planes
Encrypt/Decrypt, RegEx, Compress/Decompress
Distributed networking resources
Functional Isolation
NVMe Target, NVMe Emulation
Scalability & Programmability
© 2019 Mellanox Technologies | Confidential 6
BlueField Product Portfolio
▪ Storage card▪ Dual 100Gb/s
▪ PCIe x16
▪ FHHL
Dual 25GbE SmartNIC
Dual Port 100Gb/sController
BF1600
Dual Port 100Gb/sController
BF1700
BlueField™ PlatformBF1100 & BF1200
Dual Port 100Gb/sController
BF1500
▪ Different SKUs▪ GPU and SSD▪ 1U and 2U▪ Up to 16 SSDs
▪ Different SKUs▪ 16 / 8 / 4 cores▪ 8GB / 16GB DDR▪ PCIe x8▪ HHHL
▪ Storage card▪ Dual 100Gb/s
▪ PCIe x32
▪ FHFL
▪ Different SKUs▪ Dual 100Gb/s▪ PCIe x16▪ PCIe x32▪ FH ¾ L
▪ SDK and Development Tools▪ Networking , Security Features▪ Full NVMe storage capability
Full Software Enablement
Dual 100Gbs / EDR SmartNIC VPI
▪ VPI 2-ports 100G▪ 16 Cores▪ 8GB / 16GB DDR ▪ PCIe x16▪ FHHL
© 2019 Mellanox Technologies | Confidential 7
BlueField IC & SW
© 2019 Mellanox Technologies | Confidential 8
BlueField Block Diagram
▪ Tile Architecture - 16 ARM® A72 CPUs subsystem▪ SkyMesh™ fully coherent low-latency interconnect▪ 8MB L2 Cache, 8 Tiles▪ 48KB I-Cache, 32KB D-Cache per core▪ 12MB L3 Last Level Cache▪ ARM Frequency: 0.8GHz - 1.3GHz
▪ Dual Port 100g IO Controller, based on ConnectX-5▪ Dual 100Gb/s Ethernet/InfiniBand, compatible with ConnectX-5▪ NVMe-oF hardware accelerator▪ High-end Networking Offloads: RDMA, Erasure Coding, T10-DIF
▪ Fully Integrated PCIe switch ▪ 32 Bifurcated PCI Gen3/4 lanes (up to 200Gb/s)▪ Root Complex or Endpoint modes▪ 2x16, 4x8, 8x4 or 16x2 configurations
▪ Hardware Accelerators, Crypto Engines▪ Bulk crypto by A72 Neon ISA (AES, SHA)▪ Public Key acceleration, True RNG
▪ Memory Controllers▪ 2x Channels DDR4 Memory Controllers w/ ECC▪ NVDIMM-N Support
Dual VPI PortsEthernet/InfiniBand:
1, 10, 25,40,50,100G
32-lanesPCIe Gen3/4
© 2019 Mellanox Technologies | Confidential 9
BlueField Comparative SKU Line-up B
lue
Fie
ld™
P-S
eri
es
IPU
• 16/8/4 ARM Cores 1.3GHz
• 2-Ports 100Gb/s VPI
• Total Bandwidth 200Gb/s
• PCIe Gen4.0 x32
• Power: 90W (Typ)
• +/-CryptoB
lue
Fie
ld™
E-S
eri
es
IPU
• 16/8/4 ARM Cores 1.1GHz
• 2-Ports 100Gb/s VPI
• Total Bandwidth 160Gb/s
• PCIe Gen4.0 x32
• Power: 62W (Typ)
• +/-Crypto
Blu
eFi
eld
™ G
-Se
rie
s IP
U
• 16/8/4 ARM Cores 0.8GHz
• 2-Ports 100Gb/s VPI
• Total Bandwidth 100Gb/s
• PCIe Gen4.0 x32
• Power: 32W(Typ)
• +/-Crypto
© 2019 Mellanox Technologies | Confidential 10
BlueField™ Software Components
BlueField SmartNIC \ BlueField Storage Controller
Arm
Bootloader
(BFB)
ARM Trusted Firmware
UEFI PXE Boot
Secure Boot
Or
Applications &
Drivers
BlueOS (Yocto Linux Build)
BMC (Aspeed 2500)
ConnectX-5 NIC FirmwarePCIe Switch
BlueField IPU
ACPI
BootROM
Linux Distro (CentOS, Ubuntu)
IO DriversBlueField
DriversIO Drivers
BlueField
Drivers
OFED MFTBlueField
UtilsDiag. Utils
Storage
App
OpenBMC
IPMI
NC-SI (ConnectX-5)
Security
AppNetworking
App
▪ Bootloader▪ UEFI
▪ ATF – Arm Trusted FW
▪ ACPI (Arm resource management)
▪ Linux OS Distro▪ BlueOS Yocto Poky Kernel▪ CentOS reference drivers▪ Ubuntu commercial OS
▪ Mellanox Drivers & Packages▪ MLNX_OFED driver
▪ ASAP2
▪ NVMe-oF Kernel support & target reference▪ NVMe SNAP™
▪ IPSec Kernel support
▪ Secure Boot and Secure Firmware Upgrade
▪ OpenBMC for BMC Management
▪ Hardware Diagnostic
▪ ARM DS5 suite and CoreSight™ debug & trace
▪ ConnectX-5 Firmware - Binary File
© 2019 Mellanox Technologies | Confidential 11
Linux Operating Systems Options
▪ BlueOS▪ Linux Kernel 4.20▪ Yocto build environment (Bitbake)▪ Full cross build environment (Poky SDK)
▪ Memory EDAC driver & L3 cache profiles▪ RSHIM host net/console (tmfifo virtio driver for
PCIe or USB)▪ PMC performance counters▪ PKA driver & optimized OpenSSL▪ I2C & CPLD driver▪ GPIO driver▪ IPMI & BMC drivers▪ OFED (includes NVMe-oF support
& ASAP2)
BlueOS Community & Commercial Distro
▪CentOS, Ubuntu, Suse, etc.
▪Non-Inbox blocks▪ MLNX_OFED and patches installed on top
▪ Support Model▪ Mellanox tests the distro▪ Mellanox supports BlueField related issues▪ Distro partner & Customer own OS updates
© 2019 Mellanox Technologies | Confidential 12
Online Software Documentation
▪ https://docs.mellanox.com/display/BlueFieldSW
▪ Advantages▪ Easy navigation ▪ Links to other on-line documents▪ Available on-line for everyone▪ Advanced search capabilities▪ Export specific sections or all to PDF
© 2019 Mellanox Technologies | Confidential 13
Open Development Environment
▪ Standard Arm®v8 (64-bit) development tools▪ GNU-based toolchain (gcc, gdb, oProfile, etc.)▪ Cross-platform and native toolchain▪ Arm v8 cryptography extension support (SIMD)▪ AES encryption and decryption instructions▪ Secure Hash Algorithm (SHA) instructions
▪ Linux Operating System▪ Yocto build environment (script based SDK)▪ Full cross build environment (SDK)▪ Enables development on standard x86 machines
▪ Embedded Linux baseline (BlueOS)▪ OFED and all drivers available as patches and pre-installed on rootfs▪ Ubuntu commercial distro with Inbox drivers
▪ Debug Tools▪ OpenOCD for customizable debug▪ Perf for statistical profiling▪ Arm DS-5 includes additional tools
© 2019 Mellanox Technologies | Confidential 14
Software Defined Everything (SDX) Kills Performance
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
ApplicationCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
CoreCoreCoreCore
ApplicationCore
Virtualization & SDX PenaltyCore
SmartNIC
Bare-Metal Virtualized & Software Defined
Software DefinedHardware Accelerated
© 2019 Mellanox Technologies | Confidential 15
SmartNIC
BlueField SmartNIC
SmartNIC VPI
▪ 2-Ports 25GbE in a standard PCIe form factor – HHHL
▪ BlueField G-Series SoC – 16 / 8 Cores
▪ DDR4 memory 8GB / 16GB
▪ PCIe Gen4.0 x8
▪ 2-Ports 100GbE / EDR in a standard PCIe form factor – FHHL
▪ BlueField G-Series SoC – 16 Cores
▪ 16GB / 8GB DDR4 memory
▪ PCIe Gen4.0 x16
© 2019 Mellanox Technologies | Confidential 16
BlueField BF1500 Family Controller Card
▪ 2-Ports 100GbE / EDR in a standard PCIe form factor – FHHL
▪ BlueField G-Series SoC – 16 Cores
▪ Single channel 16GB on board DDR4 memory
▪ PCIe Gen4.0 x16 on Golden Fingers
▪ Single slot thermal solutions
▪ NC-SI connector on card
© 2019 Mellanox Technologies | Confidential 17
BlueField BF1600 Family Controller Card
▪ 2-Ports 100GbE / EDR in a standard PCIe form factor – FH¾L
▪ BlueField E-Series SoC – 16 Cores
▪ DDR4 memory ▪ 2x 16GB SODIMM (Optional)
▪ PCIe▪ PCIe Gen4.0 x16 on Golden Fingers▪ PCIe Gen3.0 x16 on 2x I-PEX Harnesses (Optional, achieve PCIe Gen3 x32)
▪ Single slot / Dual slot thermal solutions
▪ NC-SI connector on card
▪ External PCIe Power Connector
© 2019 Mellanox Technologies | Confidential 18
Building JBOF Solution with BlueField Controllers
▪ Leverage BF1600 controller will help customer accelerate JBOF TTM▪ Standard form factor, fits existing JBOF designs▪ Available in different form factors and connectivity options
▪ Build the most performance and dense JBOF▪ Best performance in the market on a single controller▪ Up to 50% OPEX reduction by providing double Storage utilization▪ Double Bandwidth - Dual port 100Gb/s▪ Double Performance▪ Higher SSD volume per JBOF
▪ Supports HA Systems
Same # of SSD
x2 BWx2 IOPs
x2 BWx2 IOPs
© 2019 Mellanox Technologies | Confidential 19
Balanced Storage System for Higher Performance
▪ JBOF system based on x86 + NIC ▪ JBOF system based on BlueField Controller
PCIe x32SSD
2-Port 100Gb/s
250Gb/s
200Gb/s(Active/Active)
Network and Storage Bandwidth is Balanced
PCIe x32SSD
2-Port 100Gb/s
250Gb/s
100Gb/s(Active/Standby)
Storage Bandwidth is Trapped
x86CPU
PC
Ie x
16
100Gb/s
© 2019 Mellanox Technologies | Confidential 20
BlueField SmartNIC
ProgrammabilityDifferent Product Flavors
IsolationPerformance
© 2019 Mellanox Technologies | Confidential 21
L2/3 Cache
CPU
Hardware-based accelerators
Memory
A fully functioning Operating System
Network Adapter
BlueField SmartNIC is a Computer
© 2019 Mellanox Technologies | Confidential 22
BlueField SmartNIC – The Next Generation NIC
▪ Accelerates wide range of security, networking and storage workloads▪ Offloading Control and Data Planes▪ Functional Isolation ▪ Security
▪ Range of Flavors available▪ 8/16 cores▪ 25Gbs and 100Gb/s dual port cards▪ Standard PCIe form factors▪ Ethernet and InfiniBand (100Gb/s)
▪ Combines best-in-class hardware network offloads with ARM processing power▪ Reduces TCO by offloading main CPU ▪ Main CPU is left for compute and applications rather than security or networking functions
▪ Standard embedded Linux software stack
© 2019 Mellanox Technologies | Confidential 23
BlueField SmartNIC: Performance and Architecture Uniformity▪ Performance ▪ Zero host CPU utilization for Control or Data▪ More I/O capacity per server ▪ Increased infrastructure scalability
▪ Handles various types of work load ▪ Bare Metal▪ Virtualized cloud▪ Containers
▪ Simplifies management, more flexibility in server assignment▪ Same arch applies to Virtualized and Bare-Metal▪ Increases revenue for Cloud providers▪ Reduces cost for Enterprise
▪ Agnostic Solution▪ Host independent implementation▪ A single solution for different systems - independent of OEM Server vendor
© 2019 Mellanox Technologies | Confidential 24
BlueField SmartNIC – For Security
ProgrammabilityDifferent Product Flavors
IsolationPerformance
© 2019 Mellanox Technologies | Confidential 25
Security Challenges in the Data Center
▪ Traditional perimeter based security model is broken
▪ Need for privacy drives adoption of cryptography to protect data
▪ New regulation requirements (e.g. GDPR)
▪ Zero Trust in the data center!
▪ Software based security and cryptography is very CPU intensive
Secure the data In-transit and at-rest
OS
UEFI
Firmware
Hardware
Securethe Chip
ZeroTrust
New Security Regulations
© 2019 Mellanox Technologies | Confidential 26
Zero Utilization: Transparent encryption accelerations
Innovative Security Approach with BlueField
Zero Trust: Security controls are built around applications
Zero Touch: Adaptive, automated Security at the edge of the network
▪ Protection of Data-in-Motion and Data-at-Rest▪ Host’s CPU is fully offloaded from encryption functions▪ BlueField SmartNIC enables fully Isolated control plane and key management
▪ Bare-metal, VM and Container complete visibility and line-rate mitigation▪ Security infrastructure runs on BlueField SmartNIC in an isolated environment▪ Resilient SDN micro-segmentation solutions
▪ BlueField SmartNIC Implements security applications independent of the host▪ Agnostic Solution for different operational environments (bare-metal, legacy, on-prem)▪ Enables massive infrastructure scalability
© 2019 Mellanox Technologies | Confidential 27
Secured Control Plane with BlueField SmartNIC
▪ Control Plane isolation and offload▪ Networking, Security and
Storage Infrastructure functions fully implemented in SmartNIC
▪ Host access to SmartNIC can be blocked by HW
▪ Security functionality runs in separate trust domain from host▪ Security agents run in the Arm
protecting the host and network▪ SmartNIC enforces policies even
when the host is attacked
▪ Security acceleration▪ IPSec and SHA acceleration▪ HW Connection Tracking▪ Public Key Infrastructure
accelerators
VMContainerBare
MetalSR-IOV
VM
OVS OVS-DPDK
OvsdbServer
Controller
Neutron
eSwitch and Hardware table
Network Interfaces
Isolation
Controlplane
ArmHost
© 2019 Mellanox Technologies | Confidential 28
BlueField Cybersecurity Cornerstones
▪ Secure Firmware Upgrade ▪ Secure Boot ▪ ARM Trust Zone▪ Key Management and attestation
▪ Accelerated transparent IPSec ▪ Accelerated SHA operations▪ Hardware public key acceleration
▪ Accelerated Connection Tracking ▪ Network Based Application Recognition▪ Host Introspection
Crypto Acceleration (Regulation and Privacy)Secured NIC (Root-of-Trust)
Programmability with IsolationAdvanced Security (L4 –L7)▪ Secured control plane with Functional Isolation▪ Isolated bare metal provisioning and control ▪ Security Ecosystem with partners▪ Bring your own security secret sauce
© 2019 Mellanox Technologies | Confidential 29
BlueField SmartNIC – for Cloud Network Solutions
ProgrammabilityDifferent Product Flavors
IsolationPerformance
© 2019 Mellanox Technologies | Confidential 30
Separated Hosts Mode (default configuration)
▪Both the x86 and the Arm hosts are symmetric▪Each host can send and
receive traffic, without dependency on the other host▪No OVS offload (ASAP2) in
this mode
BlueField
Ethernet
Port 1
ConnectX-5+
PC
Ie S
wit
ch
Ethernet
Port 0
eS
wit
ch
Arm Subsystem
DDR4
Memory
DDR4
MemoryPacket-
La
yer
RD
MA
Host Server
VM [0] VM [1]
QPI
PCIe
Driver
VM [2] VM [3]
vSwitch (OVS)
Dri
ver
Network Protocols
Security Functions
Socket 0 Socket 1
Flow
Tables
vSwitch (OVS)
Network Protocols
tmfifo
Rshim
USB
© 2019 Mellanox Technologies | Confidential 31
SmartNIC mode (Arm Switch Ownership)
▪OVS (with ASAP2) runs on the Arm cores▪All host traffic is controlled
by the switch▪ Secure mode option▪ RSHIM interface is blocked▪ Port Configuration commands are
only allowed from the ARM side(the x86 host is treated as a VF)
▪ All device related resources that require host memory are allocated on ARM memory
▪ Host PXE boot goes through Arm cores as well
BlueField
Ethernet
Port 1
ConnectX-5+
PC
Ie S
wit
ch
Ethernet
Port 0
eS
wit
ch
ARM Subsystem
DDR4
Memory
DDR4
MemoryPacket-
La
yer
RD
MA
Host Server
VM [0] VM [1]
QPI
PCIe
Driver
VM [2] VM [3]
vSwitch (OVS)
Dri
ver
Network Protocols
Security Functions
Socket 0 Socket 1
Flow
Tables
vSwitch (OVS)
Network Protocols
tmfifo
Rshim
USB
© 2019 Mellanox Technologies | Confidential 32
▪ Use Mellanox intelligent and smart adapters hardware to accelerate and offload Data-Plane
▪ Maintain control plane in Software▪ Minimize Open-Stack or SDN changes
▪ Supporting different customer configuration▪ SR-IOV or VirtIO▪ Control plane running in Kernel or in User Space (DPDK)▪ Accelerate customers’ custom Virtual Switches/Routers or known open source solution (OVS, Tungstenfabric, etc.)
▪ One architecture for ConnectX family and BlueField family
▪ Upstream and Inbox solution▪ Host OS▪ Openstack
© 2019 Mellanox Technologies | Confidential 33
BlueField ASAP2
▪Data Plane offload▪Control Plane offload to Arm▪ Isolation of Control plane from main host (Zero trust)▪Add your Software Secret Sauce on Arm (Security apps etc.)
© 2019 Mellanox Technologies | Confidential 34
BlueField SmartNIC for Bare Metal Servers
▪ Cloud customer leases an entire chassis▪ Cloud provider keeps control over the Network▪ Separated control plane management in Arm with Mellanox ASAP2 offloads▪ Scalable and transparent security policies▪ Bare Metal provisioning with OpenStack
▪ Maximal server performance (vCPU==CPU)▪ Integrity of the network kept by the SmartNIC▪ Deploy security functions in Arm▪ Micro-segmentation▪ Anti-DDoS▪ Transparent encryption▪ Key management▪ Secured NIC and Network
OVS OVS-DPDK
OvsdbServer
Controller
Ironic
eSwitch and Hardware table
Network Interface
Isolation
Controlplane
ArmBare Metal Server
Customer Applications
VirtIO-net NVMe SNAP™
SmartNIC
© 2019 Mellanox Technologies | Confidential 35
BlueField SmartNIC – for Storage
ProgrammabilityDifferent Product Flavors
IsolationPerformance
© 2019 Mellanox Technologies | Confidential 36
Software-defined SmartNIC for Accelerated ProcessingMellanox BlueField NVMe SNAP
Host Server
OS/Hypervisor
Physical Local Storage
Framework to Emulate NVMe Local Storage
Connected to Remote CloudStorage
BlueField for Virtualized Cloud
BlueField for Bare Metal Cloud
OS Agnostic with RDMA inside
Supports Different Network Transport Protocols
PCIeBUS
OS/Hypervisor
Emulated NVMeSSD Storage
NVMe SNAP
Remote Storage
vStorage
Host Server
PCIeBUS
© 2019 Mellanox Technologies | Confidential 37
Use Case 1: Bare Metal Cloud
▪ Local storage is easiest to insure application software compatibility, performance and security
▪ Limited by local storage capacity
▪ Difficult to backup and manage local storage remotely
▪ High Availability (HA) limited to local RAID
Local Physical Drive in Bare Metal
NetworkAdapter
Bare Metal Any OS
NVMe
Host Server
NIC
Physical Local Storage
PCIe BUS
▪ Performs like local storage
▪ OS and application agnostic
▪ Backup and HA using network storage
▪ Any wire protocol & storage management
▪ Same adapter for storage and networking
▪ Security through isolation
NVMe SNAP Emulation For Bare Metal
Bare Metal Any OS
Host Server
NVMe SNAP
NVMe
Remote Storage
PCIe BUS
NIC
Network
SmartNIC
© 2019 Mellanox Technologies | Confidential 38
Use Case 2: Efficient Cloud Scaling –“Rack Scale Design”
▪ OS agnostic
▪ Applications believe they still have high performance local storage
▪ Allows mixing of different storage protocols
▪ Provides platform for managing orchestration of resources
TOR Switch
Compute +
Compute +
Compute +
Compute +
Compute +
Compute +
Compute + Co
mp
ute
+ L
oca
l Sto
rage
NETWORKINGSTORAGE
COMPUTE
NVMe SNAP enables easy migration to Rack Scale
STORAGE POOL
COMPUTE POOL
NETWORKING POOL
TOR Switch
Switch
Storage Target1
Storage Target2
NVMe SNAP
NVMe SNAP
NVMe SNAP
NVMe SNAP
NVMe SNAP
NVMe SNAP
NVMe SNAP
Compute
Compute
Compute
Compute
Compute
Compute
Compute
ComputePool
StoragePool
Switch Network Pool
© 2019 Mellanox Technologies | Confidential 39
BlueField BF1100 and BF1200 Platforms
▪ Powerful reference platforms in a standard 2U/1U 19” enclosure
▪ 2U Storage Platform for advanced NVMe storage use cases▪ Includes NVMe-ready modular mid-plane and connectivity of up to 16 SSDs ▪ Supports connectivity of up to 2 GPUs via PCIe 3.0/4.0 x16 interface
▪ 1U Network Reference Platform for high performance cloud and networking
Product Family OPN Product Description
BF1200
MBE1201A-BN1/BC12U BlueField™ Reference Platform, BlueField™ P-Series. A storage controller platform with option for up to 16 SSDs. (SSDs are not included.)
MBE1200A-BN1/BC12U BlueField™ Reference Platform, BlueField™ E-Series. A storage controller platform with option for up to 16 SSDs. (SSDs are not included.)
BF1100 MBE1100A-BN1/BC11U BlueField™ Reference Platform, BlueField™ E-Series. A network appliance platform.
© 2019 Mellanox Technologies | Confidential 40
1U Reference Platform (BF1100) - Inside View
Power supply Unit
System Fan Trays
PCIe Add-in Card location: Chassis comes with riser to support ONE FHHL, single slot pitch card, installed horizontally.
© 2019 Mellanox Technologies | Confidential 41
2U Storage Platform (BF1200) – Inside View
NVMe Midplanes
© 2019 Mellanox Technologies | Confidential 42
BlueField Reference System External Interfaces (2U)
Front RST & Status LEDs
Front Disks Slots
2x PSU FRU
MLNX test connector
BMC console (top) & BF UART1
BF USB 2.0
BMC ETH (top) & BMC USB (bottom)
BF 2x QSFP ports
© 2019 Mellanox Technologies | Confidential 43
BlueWhale Development Environment: Connections
“BlueField”
“BMC”Serial
UART
BMC Console
UART
BlueField Console
Host USB
Bluefield Boot
© 2019 Mellanox Technologies | Confidential 44
BlueWhale Development Environment: BlueField Console
“BlueField”
“BMC”Serial
UART
BlueField Console
#minicom –s
(
/dev/ttyUSB0
115200baud
)
© 2019 Mellanox Technologies | Confidential 45
BlueWhale Development Environment: USB Boot Channel
“BlueField”
“BMC”Serial
HOST terminal
USB connection
Bluefield Boot
© 2019 Mellanox Technologies | Confidential 46
BlueWhale Development Environment: BMC Console
“BlueField”
“BMC”Serial
UART
BMC Console
#minicom –s
(
/dev/ttyUSB1
115200baud
)
Booting a BlueField
Booting ConnectX versus Arm
▪ ConnectX and ARM boots somewhat independently (Similar to a x86 host and ConnectX NIC)
▪ ConnectX boots from SPI-ROM (i.e. no different than traditional ConnectX-4/ConnectX-5)▪ Firmware includes ini/mlx image/efirom
image/GUIDs/MACs…
▪ Livefish mode available
▪ Arm is booted by a bfb image over one of the BlueField Bootable Interfaces▪ eMMC
▪ PCI EP
▪ USB
SPI ROM
Boot records, fw image, device info eMMC0
flash
BLUEFIELD SoC
ConnectX-5 Subsystem ARM Subsystem
Flash_preset_ConnectX-5
USB Device
eMMC0 Controller
2x 16 PCIeGen3/4 RC/EP
NC-SI
2 x 100G Interface
SPI Interface
ConfigStrapping
CONFIG_ROM[1:0]
Flash_preset_ARM
Boot records
PCIe EP Bootable Interface
Bootable interface
Bootable interface
What is a bfb
▪ BlueField Bootstream (aka bfb)▪ An image used to boot a BlueField
• BlueField understands its format
▪ May be read by BlueField from boot partition of eMMC
▪ May be pushed into BlueField by external host over USB or PCIe using RSHIM driver
▪ BFB header▪ magic number [32 bits]: BFB_IMGHDR_MAGIC defined in bluefield_boot.h.
▪ major version [4 bits]: Major version number of image.
▪ minor version [4 bits]: Minor version number of image.
▪ reserved [12 bits]: Reserved for future use.
▪ header length [4 bits]: Length of header is 4 byte words.
▪ image ID [8 bits]: Image type or ID.
• 1 = BL2, 3 = BL31, 5 = BL33 (UEFI), 55 = Name of ACPI Table, 58 = Boot image path, 59 = Boot image args
▪ image length [32 bits]: Length of image/file in bytes. Note that the image is padded to a multiple of 8 bytes but padding is not included in this value.
▪ For more information on BFB format, see README-bfb in the BlueOS under src/atf
What is a ATF
▪ ARM Trusted Firmware ▪ Boot Level 1 (BL1), Boot Level 2 (BL2), Boot Level 31 (BL31)
▪ Boot Level 33 (BL33) is UEFI
▪ BL1▪ Burned into ROM inside of BlueField. Can’t be change.
▪ Root of Trust of secured boot
▪ Read bootstream, then validate and boot BL2
▪ BL2▪ Platform specific configuration
• DDR4 configuration, training and test
• I2C, console, PLL misc configurations
▪ Secure Boot - Validate BL31 and HCA secure firmware
▪ Boot BL31
▪ BL31▪ Setup page tables/MMU
▪ Setup timer and interrupts
▪ Secure Monitor
▪ Secure Boot – Validate BL33 (UEFI)
▪ Boot UEFI
BlueField Boot Sequence
BlueField ROMBL1
BL2
BL31
UEFI
OSkernel & Root
Filesystem
▪ Early boot is what happens between reset and when we start
Linux
▪ On x86, this is typically referred to as the BIOS
▪ We use ATF (ARM Trusted Firmware) and UEFI (Unified Extensible
Firmware Interface)
▪ ATF’s first stage (BL1) is burned into the SoC itself as a ROM
▪ It does some basic silicon bookkeeping and figures out how to find the
rest of the boot code
▪ The remainder of ATF and UEFI are the “BlueField boot stream”
▪ ATF BL2 configured DDR memory and does some other bookkeeping
▪ ATF BL31 loads and stays resident to do things that require higher
privilege than Linux kernel
▪ UEFI figures out how to find Linux and the initial root filesystem image
BlueField Boot Sources
BlueField eMMC
User Defined Data Partitions
Boot Part 1 Boot Part 2
BlueField RSHIM
Boot FIFOBL1 (ROM)
SRAM
Pull from eMMC
Push from USB/PCIe
ARM cores
▪ Boot Sources:▪ eMMC0 (*** default setting on BlueWhale)
• Two boot partitions for safe upgrade• Boot partitions contain ATF/UEFI
• User data partitions contain kernel and root filesystem
• Partitions swap available via external pins, software utility on the ARM or watchdog timeout
▪ USB – Pushed from external host via Rshim host driver
▪ PCIe – Pushed from external host via Rshim host drive
BlueField Rshim Host Side Driver
▪ Rshim Host Driver runs on host (typically x86) and enables host interaction with BlueField Rshim interface over USB/PCIe
▪ Features▪ boot - Dedicated Boot channel created into BlueField
▪ console - console channel into BlueField (e.g. screen /dev/rshim0/console 115200)
▪ misc - Communication channel to the drive (e.g. echo “SW_RESET 1” > /dev/rshim0/misc
▪ net - Support IP over USB/PCIe link between BlueField and the host
▪ rshim - Ability to access BF registers
Host or BMC
USB
BlueField
ARM Cores
RShim
BOOT FIFO
TM FIFO
Rshim Registers
CoreSight Access
* Note: this is concept high-level view, not the real HW design block. PCI
eMMC
BlueField Rshim Host Side Driver
▪ Driver source in BlueOS under src/drivers/rshim
▪ How to in BlueOS under Documentation/HOWTO-rshim
▪ To build:▪ From rshim directory:
• make -C /lib/modules/`uname -r`/build M=$PWD
• make -C /lib/modules/`uname -r`/build M=$PWD modules_install
▪ To install▪ modprobe the rshim, rshim_usb, rshim_net or rshim_pci , rshim_pci_net and rshim_pcie_lf drivers
▪ Can’t have both rshim_usb and rshim_pci running at the same time (SmartNIC use case).
▪ Tested on CentOS 7
▪ rshim_usb/rshim_net Available on Bluewhale BMC Beta release▪ Allow BMC to act as host to BlueField
BlueField TMFIFO ARM Side Driver
▪ TMFIFO driver on ARM side used for IP over USB/PCIe communication with host
▪ Network name on ARM side will be tmfifo_net0
▪ Driver is tmfifo
▪ Prebuilt tmfifo driver is included in the initramfs and install-xxx.bfb images provided in BlueOS.
▪ Driver source available in BlueOS under src/driver/tmfifo
▪ Note – Driver not include in the Centos by default. ▪ BlueOS Centos script installs the driver with the BlueField disk driver. MLNX_OFED also will install the tmfifo driver
▪ Make sure tmfifo source is available when upgrading kernels in Centos.
Booting Bluefield ARM – USB Connection and Drivers
❑Use USB cable to connect BlueField and the Host❑BlueField USB shows up as USB Endpoints on Host
# lsusb…Bus 003 Device 003: ID 22dc:0004 // Vendor ID & Product ID
# lsusb –vvBus 003 Device 003: ID 22dc:0004 Device Descriptor:
bEndpointAddress 0x01 EP 1 OUT // Bulk OUT (Boot)bEndpointAddress 0x82 EP 2 IN // Interrupt (TmFIFO)bEndpointAddress 0x03 EP 3 OUT // Bulk OUT (TmFIFO)bEndpointAddress 0x84 EP 4 IN // Bulk IN (TmFIFO)
USBBlueField
ARM
Cores
RShim
Rshim
Registers External Host(or BMC)
TM FIFO
BOOT FIFO
❑BOOT (host only):• rshim.ko• rshim_usb.koOnly driver needed to boot BlueField over BOOT FIFO
❑IP over USB (host side):• rshim_net.koWorks in tandem with Bluefield-side driver to enable network transfers over TM_FIFO
❑IP over USB (ARM side):• tmfifo.koWorks in tandem with host-side driver to enable network transfers over TM_FIFO
Booting Bluefield ARM – Boot over USB/PCIe
▪ Boot the image over USB or PCIe
▪ Cat the image over the rshim boot channel▪ cat “boot_image.bfb” > /dev/rshim0/boot
▪ Now watch for the boot messages over the UART of the Bluefield
▪ Host driver will “reach” into the BF over the rshim to allow the bootstream over USB or PCIe.
▪ Note pushing an image over USB is not a persistent boot, i.e. a reboot will not boot the same image.
Example Booting BlueField ARM – Non-Persistent Boot
BL1Boot ROM
BL2 BL31UEFIBL33
Linux Initramfs
ATF (ARM Trusted Firmware)
load load load
Reset
Single Image (BFB)
⓪
① ② ③
▪ Boot the BlueField where ATF/UEFI, Linux kernel and initramfs are included in a single bfb. Bfb will be pushed over rshim../build-bfb --bfb ../bin/default.bfb --kernel <location of your kernel>/Image --initramfs <location of your initramfs>/initramfs rshim -i rshim --no-gpt my_test_image.bfb
Notes: the “-i” here means use initramfs, the default bfb contains the ATF and UEFI
Example Booting BlueField ARM – Yocto eMMC Boot
BL1Boot ROM
BL2 BL31UEFIBL33
KernelRoot
Filesystem
ATF (ARM Trusted Firmware)
load load load
Reset⓪
① ② ③
▪ Build the equivalent “default.bfb” that will installed in the eMMC boot partitionecho "ttyAMA1 earlycon=pl011,0x01800000 root=/dev/mmcblk0p2 rootwait" > bootarg
echo "/dev/mmcblk0p1/Image" > bootpath
echo “default” > acpi
./mlx-mkbfb --bl2 ../boot/bluewhale/bl2.bin --bl31 ../boot/bluewhale/bl31.bin --bl33 ../boot/BLUEFIELD_EFI.fd --boot-args=bootarg --boot-path=bootpath --boot-acpi=acpi test_default.bfb
EMMC0
eMMC Boot Partition eMMC User Data Partition
Test_default.bfb
Booting Bluefield ARM - PXE
▪ Network booting can be configured via the UEFI menu
BL1Boot ROM
BL2 BL31UEFIBL33
kernelRoot
Filesystem
ATF (ARM Trusted Firmware)
load load load
Reset
Single Image (BFB)
⓪
① ② ③
PXE Network
© 2019 Mellanox Technologies | Confidential 77
Thank You
Recommended