MIT Lincoln Laboratory SPP-HPEC09-1 BAM 12/5/2015 A Special-Purpose Processor System with Software-Defined Connectivity Benjamin Miller, Sara Siegal, James

MIT Lincoln LaboratorySPP-HPEC09-1BAM 04/21/23

A Special-Purpose Processor System with Software-Defined

Connectivity

Benjamin Miller, Sara Siegal, James Haupt, Huy Nguyen and Michael Vai

MIT Lincoln Laboratory

22 September 2009

This work is sponsored by the Navy under Air Force Contract FA8721-05-0002. Opinions, interpretations , conclusions and recommendations are those of the authors and are not necessarily endorsed by the United States Government.


Outline

• Introduction

• System Architecture

• Software Architecture

• Initial Results and Demonstration

• Ongoing Work/Summary


Why Software-Defined Connectivity?

• Modern ISR, COMM, EW systems need to be flexible

– Change hardware and software in theatre as conditions change

– Technological upgrade– Various form factors

• Want the system to be open– Underlying architecture

specific enough to reduce redundant software development

– General enough to be applied to a wide range of system components

E.g., different vendors

• Example: Reactive electronic warfare (EW) system

– Re-task components as environmental conditions change

– Easily add and replace components as needed before and during mission


Antenna Interface

Special Purpose Processor (SPP) System

• System representative of advanced EW architectures

– RF and programmable hardware, processors all connected through a switch fabric

LN

A

RF

Dis

trib

uti

on Rx 1

Rx 2

Rx R

RF

Dis

trib

uti

on

HP

A

FPGA 1 Processor 1

Enabling technology: bare-bone, low-latency pub/sub

middleware

Enabling technology: bare-bone, low-latency pub/sub

middleware

FPGA M

FPGA 2 Processor 2

Processor N

Switch Fabric

Thin Communications Layer (TCL) Middleware

Config. Mgr. Res. Mgr.

Switch Fabric

Proc. 1 Proc. 2OS

FPGA 1 FPGA M Rx RRx 1 Tx TTx 1

data processingalgorithms

OSProc. N

OS

Process PProcess 2

Process 1

configurebackplane

connections

manageinter-processconnections

Tx 1

Tx 1

Tx T


Mode 1: Hardwired

• Hardware components physically connected

• Connections through backplane are fixed (no configuration management)

• No added latency but inflexible



Switch Fabric

Proc. 1 Proc. 2OS

FPGA FPGA

RxRx TxTx

OSProc. NOS

FPGA FPGA

Process PProcess 2Process 1

connections tohardware fixed


Mode 2: Pub-Sub

• Everything communicates through the middleware– Hardware components have on-board processors running proxy

processes for data transfer

• Most flexible, but there will be overhead due to the middleware

ProcOS

Process 4

FPGA


Res. Mgr.

Proc ProcOS FPGA OS

ProcOS

Rx

Switch Fabric

FPGATx Proc

OS Rx

Process 3Process 2Proxy 1

Process PProcess 2

Process 1


Mode 3: Circuit Switching

• Configuration manager sets up all connections across the switch fabric

• May still be some co-located hardware, or some hardware that communicates via a processor through the middleware

• Overhead only incurred during configuration



Switch Fabric

Proc. 1 Proc. 2OS


OSProc. NOS

Process PProcess 2

Process 1


Today’s Presentation

• TCL middleware developed to support the SPP system– Essential foundation

• Resource Manager sets up (virtual) connections between processes



Switch Fabric

Proc. 1 Proc. 2OS


OSProc. NOS

Process PProcess 2

Process 1


Outline

• Introduction





MIT Lincoln LaboratorySPP-HPEC09-10

BAM 04/21/23

System Configuration

• 3 COTS boards connected through VPX backplane

– 1 Single-board computer, dual-core PowerPC 8641

– 1 board with 2 Xilinx Virtex- 5 FPGAs and a dual-core 8641

– 1 board with 4 dual-core 8641s– Processors run VxWorks

• Boards come from same vendor, but have different board support packages (BSPs)

• Data transfer technology of choice: Serial RapidIO (sRIO)

– Low latency important for our application

• Implement middleware in C++


BAM 04/21/23

BSP2

VPX + Serial Rapid I/OVPX + Serial Rapid I/O

Physical Interface

Operating System (Realtime: VxWorks; Standard: Linux)VxWorks

BSP1

System Model

HW (Rx/Tx, ADC, etc.)

VendorSpecific

ApplicationComponents

Signal Processing Lib

System ControlComponents


BAM 04/21/23

VPX + Serial Rapid I/O

Operating System (Realtime: VxWorks; Standard: Linux)

System Model

VendorSpecific

VPX + Serial Rapid I/O

VxWorks

BSP1 BSP2 HW (Rx/Tx, ADC, etc.)

TCL Middleware




Physical Interface


BAM 04/21/23

BSP1 BSP2

System Model

VPX + Serial Rapid I/OVPX + Serial Rapid I/O

Physical Interface

Operating System (Realtime: VxWorks; Standard: Linux)VxWorks

HW (Rx/Tx, ADC, etc.)

VendorSpecific

TCL Middleware





New components can easily be added by complying with the middleware API

New SW components

New HWcomponents


BAM 04/21/23

Outline

• Introduction






BAM 04/21/23

Switch Fabric

Publish/Subscribe Middleware

Process l1Process k

pu

blish

Process l2

Proc. l1 Proc. l2

Topic T Subscribers------------------

l1l2

Publishing application doesn’t need to know where the data is going . . .

. . . and the subscribers are unconcerned about where their data comes from

send toapplication

del

iver

deliver

TCL Middleware

Middleware acts as interface to both application and hardware/OS

Proc. k

no

tify

no

tify

OS OSOS

send

to l1

send

to l2

subscribers


BAM 04/21/23

Abstract Interfaces to Middleware

• Middleware must be abstract to be effective– Middleware developers are unaware of hardware-specific libraries– Users have to implement functions that are specific to BSPs

TCL Middleware

publisher/subscribermgmt

interface with application

interface with

hardware/OS

arrivalnotification send data to

subscribers

accept data from

publishers

How (exactly) do I execute a data transfer?

What data-transfer

technology am I using?


BAM 04/21/23

XML Parser

• Resource manager is currently in the form of an XML parser– XML file defines topics, publishers, and subscribers– Parser sets up the middleware and defines virtual network

topology



Switch Fabric

Proc. 1 Proc. 2

OS


OSProc. N

OS

Process PProcess 2

Process 1

setup.xml Parser


BAM 04/21/23

Builder

has ahas a

CustomBSP-Builder

de

riv

ed

fro

m

Interfaceto BSP

Middleware Interfaces

• Base classes– DataReader, DataReaderListener and DataWriter

interface with the application– Builder interfaces with BSPs

• Derive board- and communication-specific classes

has aDataReader

DataReaderListener

DataWriter

de

riv

ed

fro

m

SrioData-Reader

SrioData-ReaderListener

SrioData-Writerd

eri

ve

d f

rom

de

riv

ed

fro

m

Interface to Application

change with hardware

change with comm. tech.

has a


BAM 04/21/23

Builder

CustomBSP-Builder

de

riv

ed

fro

m

Interfaceto BSP

Builder

• Follows the Builder pattern in Design Patterns*

• Provides interface for sRIO-specific tasks– e.g., establish sRIO connections, execute data transfer

• Certain functions are empty (not necessarily virtual) in the base class, then implemented in the derived class with BSP-specific libraries

#include <math.h>#include “vendorPath/bspDma.h”...//member functions STATUS CustomBSPBuilder::performDmaTransfer(...){ return bspDmaTransfer(...);}...

#include <math.h>... //member functions STATUS Builder::performDmaTransfer(...){}...

*E. Gamma et. al. Design Patterns: Elements of Reusable Object-Oriented Software. Reading, Mass.: Addison-Wesley, 1995.


BAM 04/21/23

has aDataReader

DataReaderListener

DataWriter

der

ived

fro

m

SrioData-Reader

SrioData-ReaderListener

SrioData-Writerd

eriv

ed f

rom

der

ived

fro

m

Interface to Application

Publishers and Subscribers

• DataReaders, DataWriters and DataReaderListeners act as “Directors” of the Builder

– Tell the Builder what to do, Builder determines how to do it

• DataWriter used for publishing, DataReader and DataReaderListener used by subscribers

• Derived classes implement communication(sRIO)-specific, but not BSP-specific, functionality

– e.g., ring a recipient’s doorbell after transferring data

//member functionsvirtual STATUS DataWriter::write(message)=0;

virtual STATUS SrioDataWriter::write(message){... myBuilder->performDmaXfer(…);...}

Derived Builder type determined dynamically


BAM 04/21/23

Outline

• Introduction






BAM 04/21/23

Software-Defined ConnectivityInitial Implementation

<Topic><Name>Send</Name><ID>0</ID><Sources> <Source>

<SourceID>8</SourceID> </Source>

</Sources><Destinations> <Destination>

<DSTID>0</DSTID> </Destination></Destinations>

</Topic><Topic>

<Name>SendBack</Name><ID>1</ID><Sources> <Source>




</Topic>

<Topic><Name>Send</Name><ID>0</ID><Sources> <Source>




</Topic><Topic>

<Name>SendBack</Name><ID>1</ID><Sources> <Source>




</Topic>

• Experiment: Process-to-process data transfer latency

– Set up two topics– Processes use TCL to

send data back and forth– Measure round trip time

with and without middleware in place

Process 2

TCL Middleware

Process 1

Switch Fabric

Proc. 1 Proc. 2OS


OSProc. N

OS


BAM 04/21/23

Software-Defined ConnectivityCommunication Latency

• One-way latency ~23 us for small packet sizes

• Latency grows proportionally to packet size for large packets

• Reach 95% efficiency at 64 KB

• Overhead is negligible for large packets, despite increasing size

P1 P2


BAM 04/21/23

Demo 1: System Reconfiguration

Processor #1 Processor #2 Processor #3

Detect

Detect Transmit Process

TCL Middleware

Process Transmit

Configuration 1:

Configuration 2:

XML1

XML2

Objective: Demonstrate connectivity reconfiguration by simply replacing the configuration XML file

Detect Process Transmit

Detect Transmit Process


BAM 04/21/23

Transmit Proc #2

Receive Proc #1

Low-latency predefined connections allow quick response

Demo 2: Resource Management

Receive Proc #1 Transmit

TCL Middleware

Proc #2Control

I’ve detectedsignals!

I will process this newinformation!


BAM 04/21/23

Transmit Proc #2

Receive Proc #1


I need more help toanalyze the signals!

I will determine what totransmit in response!

Resource manager sets up new connections on demand to efficiently utilize available computing power


TCL Middleware

Proc #2Control

Working…


BAM 04/21/23

Transmit Proc #2

Receive Proc #1


Working…

Proc #1/Transmit are publisher/subscriber on topic TransmitWaveform

I will transmit theresponse!

I have determined anappropriate response!


TCL Middleware

Proc #2Control


BAM 04/21/23

Transmit Proc #2

Receive Proc #1


Done!

After finishing, components may be re-assigned


TCL Middleware

Proc #2Control

I will transmit theresponse!

I have determined anappropriate response!


BAM 04/21/23

Outline

• Introduction






BAM 04/21/23

Ongoing Work

Thin Communications Layer (Pub/Sub)

InterferenceMitigation

Software Modules

SW Module Timing

SW Module

ImageFormation

SW Module

Detection

SW ModuleFPGA

FPGA

FFTDIQ

DAC

ADC

IP Cores

FPGAModules

Control

Data PathPhysical Layer (e.g., Switch Fabric)

• Develop the middleware (configuration manager) to set up fixed connections

– Mode 3: Objective system

• Automate resource management- Dynamically reconfigure

system as needs change- Enable more efficient use of

resources (load balancing)

Process PProcess 2


Config. Mgr. Res. Mgr. Process 1

Switch Fabric

Proc. 1 Proc. 2OS


OSProc. N

OS


BAM 04/21/23

Summary

• Developing software-defined connectivity of hardware and software components

• Enabling technology: low-latency pub/sub middleware– Abstract base classes manage connections between nodes– Application developer implements only system-specific send and

receive code

• Encouraging initial results– At full sRIO data rate, overhead is negligible

• Working toward automated resource management for efficient allocation of processing capability, as well as automated setup of low-latency hardware connections

Documents

MIT Lincoln Laboratory SPP-HPEC09-1 BAM 12/5/2015 A Special-Purpose Processor System with Software-Defined Connectivity Benjamin Miller, Sara Siegal, James