BlueField Overview · BlueOS Yocto Poky Kernel CentOS reference drivers Ubuntu commercial OS Mellanox Drivers & Packages MLNX_OFED driver ASAP2 NVMe-oF Kernel support & target reference

© 2019 Mellanox Technologies | Confidential 1

Paz BarkanSep 2019

BlueField Overview


Safe Harbor StatementThese slides and the accompanying oral presentation contain forward-looking statements and information.

The use of words such as “may”, “might”, “will”, “should”, “expect”, “plan”, “anticipate”, “believe”, “estimate”, “project”, “intend”, “future”, “potential” or “continued”, and other similar expressions are intended to identify forward-looking statements.

All of these forward-looking statements are based on estimates and assumptions by our management that, although we believe to be reasonable, are inherently uncertain. Forward-looking statements involve risks and uncertainties, including, but not limited to, economic, competitive, governmental and technological factors outside of our control, that may cause our business, industry, strategy or actual results to differ materially from the forward-looking statement.

These risks and uncertainties may include those discussed under the heading “Risk Factors” in the Company’s most recent 10K and 10Qs on file with the Securities and Exchange Commission, and other factors which may not be known to us. Any forward-looking statement speaks only as of its date. We undertake no obligation to publicly update or revise any forward-looking statement, whether as a result of new information, future events or otherwise, except as required by law.


BlueField – Field of Advantages

BlueField – The Premier I/O Processing Unit Solution


Storage

BlueField Enables Multiple Market Segments

Security

NFV & Cloud

▪ Enabling NVMe-over-Fabrics storage solutions

▪ Enhancing software-defined and hyperconverged storage solutions

▪ Enabling secured and isolated cloud solutions

▪ New data center security models

▪ Enabling accelerated NFV host agnostics solutions through SmartNIC

▪ Offloading OVS control & data planes

Edge Computing

▪ Enabling mini-servers at the network edge

▪ Involve superior network capabilities

HPC, Machine Learning▪ Co-processor for privacy and proprietary advanced algorithms

▪ InfiniBand storage solutions


BlueField - The Ultimate Co-Processor

Host-agnostic network solution

Integrated Control & Data planes

Encrypt/Decrypt, RegEx, Compress/Decompress

Distributed networking resources

Functional Isolation

NVMe Target, NVMe Emulation

Scalability & Programmability


BlueField Product Portfolio

▪ Storage card▪ Dual 100Gb/s

▪ PCIe x16

▪ FHHL

Dual 25GbE SmartNIC

Dual Port 100Gb/sController

BF1600


BF1700

BlueField™ PlatformBF1100 & BF1200


BF1500

▪ Different SKUs▪ GPU and SSD▪ 1U and 2U▪ Up to 16 SSDs

▪ Different SKUs▪ 16 / 8 / 4 cores▪ 8GB / 16GB DDR▪ PCIe x8▪ HHHL

▪ Storage card▪ Dual 100Gb/s

▪ PCIe x32

▪ FHFL

▪ Different SKUs▪ Dual 100Gb/s▪ PCIe x16▪ PCIe x32▪ FH ¾ L

▪ SDK and Development Tools▪ Networking , Security Features▪ Full NVMe storage capability

Full Software Enablement

Dual 100Gbs / EDR SmartNIC VPI

▪ VPI 2-ports 100G▪ 16 Cores▪ 8GB / 16GB DDR ▪ PCIe x16▪ FHHL


BlueField IC & SW


BlueField Block Diagram

▪ Tile Architecture - 16 ARM® A72 CPUs subsystem▪ SkyMesh™ fully coherent low-latency interconnect▪ 8MB L2 Cache, 8 Tiles▪ 48KB I-Cache, 32KB D-Cache per core▪ 12MB L3 Last Level Cache▪ ARM Frequency: 0.8GHz - 1.3GHz

▪ Dual Port 100g IO Controller, based on ConnectX-5▪ Dual 100Gb/s Ethernet/InfiniBand, compatible with ConnectX-5▪ NVMe-oF hardware accelerator▪ High-end Networking Offloads: RDMA, Erasure Coding, T10-DIF

▪ Fully Integrated PCIe switch ▪ 32 Bifurcated PCI Gen3/4 lanes (up to 200Gb/s)▪ Root Complex or Endpoint modes▪ 2x16, 4x8, 8x4 or 16x2 configurations

▪ Hardware Accelerators, Crypto Engines▪ Bulk crypto by A72 Neon ISA (AES, SHA)▪ Public Key acceleration, True RNG

▪ Memory Controllers▪ 2x Channels DDR4 Memory Controllers w/ ECC▪ NVDIMM-N Support

Dual VPI PortsEthernet/InfiniBand:

1, 10, 25,40,50,100G

32-lanesPCIe Gen3/4


BlueField Comparative SKU Line-up B

lue

Fie

ld™

P-S

eri

es

IPU

• 16/8/4 ARM Cores 1.3GHz

• 2-Ports 100Gb/s VPI

• Total Bandwidth 200Gb/s

• PCIe Gen4.0 x32

• Power: 90W (Typ)

• +/-CryptoB

lue

Fie

ld™

E-S

eri

es

IPU




• PCIe Gen4.0 x32

• Power: 62W (Typ)

• +/-Crypto

Blu

eFi

eld

™ G

-Se

rie

s IP

U




• PCIe Gen4.0 x32

• Power: 32W(Typ)

• +/-Crypto


BlueField™ Software Components

BlueField SmartNIC \ BlueField Storage Controller

Arm

Bootloader

(BFB)

ARM Trusted Firmware

UEFI PXE Boot

Secure Boot

Or

Applications &

Drivers

BlueOS (Yocto Linux Build)

BMC (Aspeed 2500)

ConnectX-5 NIC FirmwarePCIe Switch

BlueField IPU

ACPI

BootROM

Linux Distro (CentOS, Ubuntu)

IO DriversBlueField

DriversIO Drivers

BlueField

Drivers

OFED MFTBlueField

UtilsDiag. Utils

Storage

App

OpenBMC

IPMI

NC-SI (ConnectX-5)

Security

AppNetworking

App

▪ Bootloader▪ UEFI

▪ ATF – Arm Trusted FW

▪ ACPI (Arm resource management)

▪ Linux OS Distro▪ BlueOS Yocto Poky Kernel▪ CentOS reference drivers▪ Ubuntu commercial OS

▪ Mellanox Drivers & Packages▪ MLNX_OFED driver

▪ ASAP2

▪ NVMe-oF Kernel support & target reference▪ NVMe SNAP™

▪ IPSec Kernel support

▪ Secure Boot and Secure Firmware Upgrade

▪ OpenBMC for BMC Management

▪ Hardware Diagnostic

▪ ARM DS5 suite and CoreSight™ debug & trace

▪ ConnectX-5 Firmware - Binary File


Linux Operating Systems Options

▪ BlueOS▪ Linux Kernel 4.20▪ Yocto build environment (Bitbake)▪ Full cross build environment (Poky SDK)

▪ Memory EDAC driver & L3 cache profiles▪ RSHIM host net/console (tmfifo virtio driver for

PCIe or USB)▪ PMC performance counters▪ PKA driver & optimized OpenSSL▪ I2C & CPLD driver▪ GPIO driver▪ IPMI & BMC drivers▪ OFED (includes NVMe-oF support

& ASAP2)

BlueOS Community & Commercial Distro

▪CentOS, Ubuntu, Suse, etc.

▪Non-Inbox blocks▪ MLNX_OFED and patches installed on top

▪ Support Model▪ Mellanox tests the distro▪ Mellanox supports BlueField related issues▪ Distro partner & Customer own OS updates


Online Software Documentation

▪ https://docs.mellanox.com/display/BlueFieldSW

▪ Advantages▪ Easy navigation ▪ Links to other on-line documents▪ Available on-line for everyone▪ Advanced search capabilities▪ Export specific sections or all to PDF

https://docs.mellanox.com/display/BlueFieldSW


Open Development Environment

▪ Standard Arm®v8 (64-bit) development tools▪ GNU-based toolchain (gcc, gdb, oProfile, etc.)▪ Cross-platform and native toolchain▪ Arm v8 cryptography extension support (SIMD)▪ AES encryption and decryption instructions▪ Secure Hash Algorithm (SHA) instructions

▪ Linux Operating System▪ Yocto build environment (script based SDK)▪ Full cross build environment (SDK)▪ Enables development on standard x86 machines

▪ Embedded Linux baseline (BlueOS)▪ OFED and all drivers available as patches and pre-installed on rootfs▪ Ubuntu commercial distro with Inbox drivers

▪ Debug Tools▪ OpenOCD for customizable debug▪ Perf for statistical profiling▪ Arm DS-5 includes additional tools


Software Defined Everything (SDX) Kills Performance

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

ApplicationCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

CoreCoreCoreCore

ApplicationCore

Virtualization & SDX PenaltyCore

SmartNIC

Bare-Metal Virtualized & Software Defined

Software DefinedHardware Accelerated


SmartNIC

BlueField SmartNIC

SmartNIC VPI

▪ 2-Ports 25GbE in a standard PCIe form factor – HHHL

▪ BlueField G-Series SoC – 16 / 8 Cores

▪ DDR4 memory 8GB / 16GB

▪ PCIe Gen4.0 x8

▪ 2-Ports 100GbE / EDR in a standard PCIe form factor – FHHL

▪ BlueField G-Series SoC – 16 Cores

▪ 16GB / 8GB DDR4 memory

▪ PCIe Gen4.0 x16


BlueField BF1500 Family Controller Card

▪ 2-Ports 100GbE / EDR in a standard PCIe form factor – FHHL

▪ BlueField G-Series SoC – 16 Cores

▪ Single channel 16GB on board DDR4 memory

▪ PCIe Gen4.0 x16 on Golden Fingers

▪ Single slot thermal solutions

▪ NC-SI connector on card


BlueField BF1600 Family Controller Card

▪ 2-Ports 100GbE / EDR in a standard PCIe form factor – FH¾L

▪ BlueField E-Series SoC – 16 Cores

▪ DDR4 memory ▪ 2x 16GB SODIMM (Optional)

▪ PCIe▪ PCIe Gen4.0 x16 on Golden Fingers▪ PCIe Gen3.0 x16 on 2x I-PEX Harnesses (Optional, achieve PCIe Gen3 x32)

▪ Single slot / Dual slot thermal solutions

▪ NC-SI connector on card

▪ External PCIe Power Connector


Building JBOF Solution with BlueField Controllers

▪ Leverage BF1600 controller will help customer accelerate JBOF TTM▪ Standard form factor, fits existing JBOF designs▪ Available in different form factors and connectivity options

▪ Build the most performance and dense JBOF▪ Best performance in the market on a single controller▪ Up to 50% OPEX reduction by providing double Storage utilization▪ Double Bandwidth - Dual port 100Gb/s▪ Double Performance▪ Higher SSD volume per JBOF

▪ Supports HA Systems

Same # of SSD

x2 BWx2 IOPs

x2 BWx2 IOPs


Balanced Storage System for Higher Performance

▪ JBOF system based on x86 + NIC ▪ JBOF system based on BlueField Controller

PCIe x32SSD

2-Port 100Gb/s

250Gb/s

200Gb/s(Active/Active)

Network and Storage Bandwidth is Balanced

PCIe x32SSD

2-Port 100Gb/s

250Gb/s

100Gb/s(Active/Standby)

Storage Bandwidth is Trapped

x86CPU

PC

Ie x

16

100Gb/s


BlueField SmartNIC

ProgrammabilityDifferent Product Flavors

IsolationPerformance


L2/3 Cache

CPU

Hardware-based accelerators

Memory

A fully functioning Operating System

Network Adapter

BlueField SmartNIC is a Computer


BlueField SmartNIC – The Next Generation NIC

▪ Accelerates wide range of security, networking and storage workloads▪ Offloading Control and Data Planes▪ Functional Isolation ▪ Security

▪ Range of Flavors available▪ 8/16 cores▪ 25Gbs and 100Gb/s dual port cards▪ Standard PCIe form factors▪ Ethernet and InfiniBand (100Gb/s)

▪ Combines best-in-class hardware network offloads with ARM processing power▪ Reduces TCO by offloading main CPU ▪ Main CPU is left for compute and applications rather than security or networking functions

▪ Standard embedded Linux software stack


BlueField SmartNIC: Performance and Architecture Uniformity▪ Performance ▪ Zero host CPU utilization for Control or Data▪ More I/O capacity per server ▪ Increased infrastructure scalability

▪ Handles various types of work load ▪ Bare Metal▪ Virtualized cloud▪ Containers

▪ Simplifies management, more flexibility in server assignment▪ Same arch applies to Virtualized and Bare-Metal▪ Increases revenue for Cloud providers▪ Reduces cost for Enterprise

▪ Agnostic Solution▪ Host independent implementation▪ A single solution for different systems - independent of OEM Server vendor


BlueField SmartNIC – For Security




Security Challenges in the Data Center

▪ Traditional perimeter based security model is broken

▪ Need for privacy drives adoption of cryptography to protect data

▪ New regulation requirements (e.g. GDPR)

▪ Zero Trust in the data center!

▪ Software based security and cryptography is very CPU intensive

Secure the data In-transit and at-rest

OS

UEFI

Firmware

Hardware

Securethe Chip

ZeroTrust

New Security Regulations


Zero Utilization: Transparent encryption accelerations

Innovative Security Approach with BlueField

Zero Trust: Security controls are built around applications

Zero Touch: Adaptive, automated Security at the edge of the network

▪ Protection of Data-in-Motion and Data-at-Rest▪ Host’s CPU is fully offloaded from encryption functions▪ BlueField SmartNIC enables fully Isolated control plane and key management

▪ Bare-metal, VM and Container complete visibility and line-rate mitigation▪ Security infrastructure runs on BlueField SmartNIC in an isolated environment▪ Resilient SDN micro-segmentation solutions

▪ BlueField SmartNIC Implements security applications independent of the host▪ Agnostic Solution for different operational environments (bare-metal, legacy, on-prem)▪ Enables massive infrastructure scalability


Secured Control Plane with BlueField SmartNIC

▪ Control Plane isolation and offload▪ Networking, Security and

Storage Infrastructure functions fully implemented in SmartNIC

▪ Host access to SmartNIC can be blocked by HW

▪ Security functionality runs in separate trust domain from host▪ Security agents run in the Arm

protecting the host and network▪ SmartNIC enforces policies even

when the host is attacked

▪ Security acceleration▪ IPSec and SHA acceleration▪ HW Connection Tracking▪ Public Key Infrastructure

accelerators

VMContainerBare

MetalSR-IOV

VM

OVS OVS-DPDK

OvsdbServer

Controller

Neutron

eSwitch and Hardware table

Network Interfaces

Isolation

Controlplane

ArmHost


BlueField Cybersecurity Cornerstones

▪ Secure Firmware Upgrade ▪ Secure Boot ▪ ARM Trust Zone▪ Key Management and attestation

▪ Accelerated transparent IPSec ▪ Accelerated SHA operations▪ Hardware public key acceleration

▪ Accelerated Connection Tracking ▪ Network Based Application Recognition▪ Host Introspection

Crypto Acceleration (Regulation and Privacy)Secured NIC (Root-of-Trust)

Programmability with IsolationAdvanced Security (L4 –L7)▪ Secured control plane with Functional Isolation▪ Isolated bare metal provisioning and control ▪ Security Ecosystem with partners▪ Bring your own security secret sauce


BlueField SmartNIC – for Cloud Network Solutions




Separated Hosts Mode (default configuration)

▪Both the x86 and the Arm hosts are symmetric▪Each host can send and

receive traffic, without dependency on the other host▪No OVS offload (ASAP2) in

this mode

BlueField

Ethernet

Port 1

ConnectX-5+

PC

Ie S

wit

ch

Ethernet

Port 0

eS

wit

ch

Arm Subsystem

DDR4

Memory

DDR4

MemoryPacket-

La

yer

RD

MA

Host Server

VM [0] VM [1]

QPI

PCIe

Driver

VM [2] VM [3]

vSwitch (OVS)

Dri

ver

Network Protocols

Security Functions

Socket 0 Socket 1

Flow

Tables

vSwitch (OVS)

Network Protocols

tmfifo

Rshim

USB


SmartNIC mode (Arm Switch Ownership)

▪OVS (with ASAP2) runs on the Arm cores▪All host traffic is controlled

by the switch▪ Secure mode option▪ RSHIM interface is blocked▪ Port Configuration commands are

only allowed from the ARM side(the x86 host is treated as a VF)

▪ All device related resources that require host memory are allocated on ARM memory

▪ Host PXE boot goes through Arm cores as well

BlueField

Ethernet

Port 1

ConnectX-5+

PC

Ie S

wit

ch

Ethernet

Port 0

eS

wit

ch

ARM Subsystem

DDR4

Memory

DDR4

MemoryPacket-

La

yer

RD

MA

Host Server

VM [0] VM [1]

QPI

PCIe

Driver

VM [2] VM [3]

vSwitch (OVS)

Dri

ver

Network Protocols

Security Functions

Socket 0 Socket 1

Flow

Tables

vSwitch (OVS)

Network Protocols

tmfifo

Rshim

USB


▪ Use Mellanox intelligent and smart adapters hardware to accelerate and offload Data-Plane

▪ Maintain control plane in Software▪ Minimize Open-Stack or SDN changes

▪ Supporting different customer configuration▪ SR-IOV or VirtIO▪ Control plane running in Kernel or in User Space (DPDK)▪ Accelerate customers’ custom Virtual Switches/Routers or known open source solution (OVS, Tungstenfabric, etc.)

▪ One architecture for ConnectX family and BlueField family

▪ Upstream and Inbox solution▪ Host OS▪ Openstack


BlueField ASAP2

▪Data Plane offload▪Control Plane offload to Arm▪ Isolation of Control plane from main host (Zero trust)▪Add your Software Secret Sauce on Arm (Security apps etc.)


BlueField SmartNIC for Bare Metal Servers

▪ Cloud customer leases an entire chassis▪ Cloud provider keeps control over the Network▪ Separated control plane management in Arm with Mellanox ASAP2 offloads▪ Scalable and transparent security policies▪ Bare Metal provisioning with OpenStack

▪ Maximal server performance (vCPU==CPU)▪ Integrity of the network kept by the SmartNIC▪ Deploy security functions in Arm▪ Micro-segmentation▪ Anti-DDoS▪ Transparent encryption▪ Key management▪ Secured NIC and Network

OVS OVS-DPDK

OvsdbServer

Controller

Ironic

eSwitch and Hardware table

Network Interface

Isolation

Controlplane

ArmBare Metal Server

Customer Applications

VirtIO-net NVMe SNAP™

SmartNIC


BlueField SmartNIC – for Storage




Software-defined SmartNIC for Accelerated ProcessingMellanox BlueField NVMe SNAP

Host Server

OS/Hypervisor

Physical Local Storage

Framework to Emulate NVMe Local Storage

Connected to Remote CloudStorage

BlueField for Virtualized Cloud

BlueField for Bare Metal Cloud

OS Agnostic with RDMA inside

Supports Different Network Transport Protocols

PCIeBUS

OS/Hypervisor

Emulated NVMeSSD Storage

NVMe SNAP

Remote Storage

vStorage

Host Server

PCIeBUS


Use Case 1: Bare Metal Cloud

▪ Local storage is easiest to insure application software compatibility, performance and security

▪ Limited by local storage capacity

▪ Difficult to backup and manage local storage remotely

▪ High Availability (HA) limited to local RAID

Local Physical Drive in Bare Metal

NetworkAdapter

Bare Metal Any OS

NVMe

Host Server

NIC

Physical Local Storage

PCIe BUS

▪ Performs like local storage

▪ OS and application agnostic

▪ Backup and HA using network storage

▪ Any wire protocol & storage management

▪ Same adapter for storage and networking

▪ Security through isolation

NVMe SNAP Emulation For Bare Metal

Bare Metal Any OS

Host Server

NVMe SNAP

NVMe

Remote Storage

PCIe BUS

NIC

Network

SmartNIC


Use Case 2: Efficient Cloud Scaling –“Rack Scale Design”

▪ OS agnostic

▪ Applications believe they still have high performance local storage

▪ Allows mixing of different storage protocols

▪ Provides platform for managing orchestration of resources

TOR Switch

Compute +

Compute +

Compute +

Compute +

Compute +

Compute +

Compute + Co

mp

ute

+ L

oca

l Sto

rage

NETWORKINGSTORAGE

COMPUTE

NVMe SNAP enables easy migration to Rack Scale

STORAGE POOL

COMPUTE POOL

NETWORKING POOL

TOR Switch

Switch

Storage Target1

Storage Target2

NVMe SNAP

NVMe SNAP

NVMe SNAP

NVMe SNAP

NVMe SNAP

NVMe SNAP

NVMe SNAP

Compute

Compute

Compute

Compute

Compute

Compute

Compute

ComputePool

StoragePool

Switch Network Pool


BlueField BF1100 and BF1200 Platforms

▪ Powerful reference platforms in a standard 2U/1U 19” enclosure

▪ 2U Storage Platform for advanced NVMe storage use cases▪ Includes NVMe-ready modular mid-plane and connectivity of up to 16 SSDs ▪ Supports connectivity of up to 2 GPUs via PCIe 3.0/4.0 x16 interface

▪ 1U Network Reference Platform for high performance cloud and networking

Product Family OPN Product Description

BF1200

MBE1201A-BN1/BC12U BlueField™ Reference Platform, BlueField™ P-Series. A storage controller platform with option for up to 16 SSDs. (SSDs are not included.)

MBE1200A-BN1/BC12U BlueField™ Reference Platform, BlueField™ E-Series. A storage controller platform with option for up to 16 SSDs. (SSDs are not included.)

BF1100 MBE1100A-BN1/BC11U BlueField™ Reference Platform, BlueField™ E-Series. A network appliance platform.


1U Reference Platform (BF1100) - Inside View

Power supply Unit

System Fan Trays

PCIe Add-in Card location: Chassis comes with riser to support ONE FHHL, single slot pitch card, installed horizontally.


2U Storage Platform (BF1200) – Inside View

NVMe Midplanes


BlueField Reference System External Interfaces (2U)

Front RST & Status LEDs

Front Disks Slots

2x PSU FRU

MLNX test connector

BMC console (top) & BF UART1

BF USB 2.0

BMC ETH (top) & BMC USB (bottom)

BF 2x QSFP ports


BlueWhale Development Environment: Connections

“BlueField”

“BMC”Serial

UART

BMC Console

UART

BlueField Console

Host USB

Bluefield Boot


BlueWhale Development Environment: BlueField Console

“BlueField”

“BMC”Serial

UART

BlueField Console

#minicom –s

(

/dev/ttyUSB0

115200baud

)


BlueWhale Development Environment: USB Boot Channel

“BlueField”

“BMC”Serial

HOST terminal

USB connection

Bluefield Boot


BlueWhale Development Environment: BMC Console

“BlueField”

“BMC”Serial

UART

BMC Console

#minicom –s

(

/dev/ttyUSB1

115200baud

)

Booting a BlueField

Booting ConnectX versus Arm

▪ ConnectX and ARM boots somewhat independently (Similar to a x86 host and ConnectX NIC)

▪ ConnectX boots from SPI-ROM (i.e. no different than traditional ConnectX-4/ConnectX-5)▪ Firmware includes ini/mlx image/efirom

image/GUIDs/MACs…

▪ Livefish mode available

▪ Arm is booted by a bfb image over one of the BlueField Bootable Interfaces▪ eMMC

▪ PCI EP

▪ USB

SPI ROM

Boot records, fw image, device info eMMC0

flash

BLUEFIELD SoC

ConnectX-5 Subsystem ARM Subsystem

Flash_preset_ConnectX-5

USB Device

eMMC0 Controller

2x 16 PCIeGen3/4 RC/EP

NC-SI

2 x 100G Interface

SPI Interface

ConfigStrapping

CONFIG_ROM[1:0]

Flash_preset_ARM

Boot records

PCIe EP Bootable Interface

Bootable interface

Bootable interface

What is a bfb

▪ BlueField Bootstream (aka bfb)▪ An image used to boot a BlueField

• BlueField understands its format

▪ May be read by BlueField from boot partition of eMMC

▪ May be pushed into BlueField by external host over USB or PCIe using RSHIM driver

▪ BFB header▪ magic number [32 bits]: BFB_IMGHDR_MAGIC defined in bluefield_boot.h.

▪ major version [4 bits]: Major version number of image.

▪ minor version [4 bits]: Minor version number of image.

▪ reserved [12 bits]: Reserved for future use.

▪ header length [4 bits]: Length of header is 4 byte words.

▪ image ID [8 bits]: Image type or ID.

• 1 = BL2, 3 = BL31, 5 = BL33 (UEFI), 55 = Name of ACPI Table, 58 = Boot image path, 59 = Boot image args

▪ image length [32 bits]: Length of image/file in bytes. Note that the image is padded to a multiple of 8 bytes but padding is not included in this value.

▪ For more information on BFB format, see README-bfb in the BlueOS under src/atf

What is a ATF

▪ ARM Trusted Firmware ▪ Boot Level 1 (BL1), Boot Level 2 (BL2), Boot Level 31 (BL31)

▪ Boot Level 33 (BL33) is UEFI

▪ BL1▪ Burned into ROM inside of BlueField. Can’t be change.

▪ Root of Trust of secured boot

▪ Read bootstream, then validate and boot BL2

▪ BL2▪ Platform specific configuration

• DDR4 configuration, training and test

• I2C, console, PLL misc configurations

▪ Secure Boot - Validate BL31 and HCA secure firmware

▪ Boot BL31

▪ BL31▪ Setup page tables/MMU

▪ Setup timer and interrupts

▪ Secure Monitor

▪ Secure Boot – Validate BL33 (UEFI)

▪ Boot UEFI

BlueField Boot Sequence

BlueField ROMBL1

BL2

BL31

UEFI

OSkernel & Root

Filesystem

▪ Early boot is what happens between reset and when we start

Linux

▪ On x86, this is typically referred to as the BIOS

▪ We use ATF (ARM Trusted Firmware) and UEFI (Unified Extensible

Firmware Interface)

▪ ATF’s first stage (BL1) is burned into the SoC itself as a ROM

▪ It does some basic silicon bookkeeping and figures out how to find the

rest of the boot code

▪ The remainder of ATF and UEFI are the “BlueField boot stream”

▪ ATF BL2 configured DDR memory and does some other bookkeeping

▪ ATF BL31 loads and stays resident to do things that require higher

privilege than Linux kernel

▪ UEFI figures out how to find Linux and the initial root filesystem image

BlueField Boot Sources

BlueField eMMC

User Defined Data Partitions

Boot Part 1 Boot Part 2

BlueField RSHIM

Boot FIFOBL1 (ROM)

SRAM

Pull from eMMC

Push from USB/PCIe

ARM cores

▪ Boot Sources:▪ eMMC0 (*** default setting on BlueWhale)

• Two boot partitions for safe upgrade• Boot partitions contain ATF/UEFI

• User data partitions contain kernel and root filesystem

• Partitions swap available via external pins, software utility on the ARM or watchdog timeout

▪ USB – Pushed from external host via Rshim host driver

▪ PCIe – Pushed from external host via Rshim host drive

BlueField Rshim Host Side Driver

▪ Rshim Host Driver runs on host (typically x86) and enables host interaction with BlueField Rshim interface over USB/PCIe

▪ Features▪ boot - Dedicated Boot channel created into BlueField

▪ console - console channel into BlueField (e.g. screen /dev/rshim0/console 115200)

▪ misc - Communication channel to the drive (e.g. echo “SW_RESET 1” > /dev/rshim0/misc

▪ net - Support IP over USB/PCIe link between BlueField and the host

▪ rshim - Ability to access BF registers

Host or BMC

USB

BlueField

ARM Cores

RShim

BOOT FIFO

TM FIFO

Rshim Registers

CoreSight Access

* Note: this is concept high-level view, not the real HW design block. PCI

eMMC

BlueField Rshim Host Side Driver

▪ Driver source in BlueOS under src/drivers/rshim

▪ How to in BlueOS under Documentation/HOWTO-rshim

▪ To build:▪ From rshim directory:

• make -C /lib/modules/`uname -r`/build M=$PWD

• make -C /lib/modules/`uname -r`/build M=$PWD modules_install

▪ To install▪ modprobe the rshim, rshim_usb, rshim_net or rshim_pci , rshim_pci_net and rshim_pcie_lf drivers

▪ Can’t have both rshim_usb and rshim_pci running at the same time (SmartNIC use case).

▪ Tested on CentOS 7

▪ rshim_usb/rshim_net Available on Bluewhale BMC Beta release▪ Allow BMC to act as host to BlueField

BlueField TMFIFO ARM Side Driver

▪ TMFIFO driver on ARM side used for IP over USB/PCIe communication with host

▪ Network name on ARM side will be tmfifo_net0

▪ Driver is tmfifo

▪ Prebuilt tmfifo driver is included in the initramfs and install-xxx.bfb images provided in BlueOS.

▪ Driver source available in BlueOS under src/driver/tmfifo

▪ Note – Driver not include in the Centos by default. ▪ BlueOS Centos script installs the driver with the BlueField disk driver. MLNX_OFED also will install the tmfifo driver

▪ Make sure tmfifo source is available when upgrading kernels in Centos.

Booting Bluefield ARM – USB Connection and Drivers

❑Use USB cable to connect BlueField and the Host❑BlueField USB shows up as USB Endpoints on Host

# lsusb…Bus 003 Device 003: ID 22dc:0004 // Vendor ID & Product ID

# lsusb –vvBus 003 Device 003: ID 22dc:0004 Device Descriptor:

bEndpointAddress 0x01 EP 1 OUT // Bulk OUT (Boot)bEndpointAddress 0x82 EP 2 IN // Interrupt (TmFIFO)bEndpointAddress 0x03 EP 3 OUT // Bulk OUT (TmFIFO)bEndpointAddress 0x84 EP 4 IN // Bulk IN (TmFIFO)

USBBlueField

ARM

Cores

RShim

Rshim

Registers External Host(or BMC)

TM FIFO

BOOT FIFO

❑BOOT (host only):• rshim.ko• rshim_usb.koOnly driver needed to boot BlueField over BOOT FIFO

❑IP over USB (host side):• rshim_net.koWorks in tandem with Bluefield-side driver to enable network transfers over TM_FIFO

❑IP over USB (ARM side):• tmfifo.koWorks in tandem with host-side driver to enable network transfers over TM_FIFO

Booting Bluefield ARM – Boot over USB/PCIe

▪ Boot the image over USB or PCIe

▪ Cat the image over the rshim boot channel▪ cat “boot_image.bfb” > /dev/rshim0/boot

▪ Now watch for the boot messages over the UART of the Bluefield

▪ Host driver will “reach” into the BF over the rshim to allow the bootstream over USB or PCIe.

▪ Note pushing an image over USB is not a persistent boot, i.e. a reboot will not boot the same image.

Example Booting BlueField ARM – Non-Persistent Boot

BL1Boot ROM

BL2 BL31UEFIBL33

Linux Initramfs

ATF (ARM Trusted Firmware)

load load load

Reset

Single Image (BFB)

⓪

① ② ③

▪ Boot the BlueField where ATF/UEFI, Linux kernel and initramfs are included in a single bfb. Bfb will be pushed over rshim../build-bfb --bfb ../bin/default.bfb --kernel <location of your kernel>/Image --initramfs <location of your initramfs>/initramfs rshim -i rshim --no-gpt my_test_image.bfb

Notes: the “-i” here means use initramfs, the default bfb contains the ATF and UEFI

Example Booting BlueField ARM – Yocto eMMC Boot

BL1Boot ROM

BL2 BL31UEFIBL33

KernelRoot

Filesystem


load load load

Reset⓪

① ② ③

▪ Build the equivalent “default.bfb” that will installed in the eMMC boot partitionecho "ttyAMA1 earlycon=pl011,0x01800000 root=/dev/mmcblk0p2 rootwait" > bootarg

echo "/dev/mmcblk0p1/Image" > bootpath

echo “default” > acpi

./mlx-mkbfb --bl2 ../boot/bluewhale/bl2.bin --bl31 ../boot/bluewhale/bl31.bin --bl33 ../boot/BLUEFIELD_EFI.fd --boot-args=bootarg --boot-path=bootpath --boot-acpi=acpi test_default.bfb

EMMC0

eMMC Boot Partition eMMC User Data Partition

Test_default.bfb

Booting Bluefield ARM - PXE

▪ Network booting can be configured via the UEFI menu

BL1Boot ROM

BL2 BL31UEFIBL33

kernelRoot

Filesystem


load load load

Reset

Single Image (BFB)

⓪

① ② ③

PXE Network


Thank You

Documents

BlueField Overview · BlueOS Yocto Poky Kernel CentOS reference drivers Ubuntu commercial OS Mellanox Drivers & Packages MLNX_OFED driver ASAP2 NVMe-oF Kernel support & target reference