NetFPGA Hands-on Training Day 1

Hands-on Training – June 18-19, 2012 1

NetFPGA Hands-on TrainingDay 1

Presented by: Adam Covington (Stanford University)

Indiana University, BloomingtonJune 18 - 19, 2012

http://NetFPGA.org


Tutorial Outline• Background

– Introduction– The NetFPGA Platform

• The Stanford Base Reference Router– Motivation: Basic IP review– Example: Reference Router running on the NetFPGA

• Infrastructure– Tree– Build System– Scripts

• The Life of a Packet Through the NetFPGA– Hardware Datapath – Interface to software: Exceptions and Host I/O

• Implementation– Module Template– User Data Path– Write Crypto NIC using a static key

• Simulation and Debug– Write and Run Simulations for Crypto NIC

• Concluding Remarks


Section I: Motivation


NetFPGA = Networked FPGA

A line-rate, flexible, open networking platform for teaching and research


NetFPGA 1G Board

NetFPGA consists of…

Four elements:

• NetFPGA board

• Tools + reference designs

• Contributed projects

• CommunityNetFPGA 10G Board


NetFPGA 1G NetFPGA 10G

4 x 1Gbps Ethernet Ports 4 x 10Gbps Ethernet Ports

4.5 MB ZBT SRAM64 MB DDR2 SDRAM

27 MB QDRII-SRAM288 MB RLDRAM-II

PCI PCI Express x8

Virtex II-Pro 50 Virtex 5 TX240T

NetFPGA Board Comparison


FPGAFPGA

MemoryMemory

1GE1GE

1GE1GE

1GE1GE

1GE1GE

NetFPGA board

PCI

CPUCPU MemoryMemory

NetFPGA Board

PC with NetFPGA

NetworkingSoftwarerunning on a standard PC

A hardware acceleratorbuilt with a Field Programmable Gate Arraydriving Gigabit network links


Tools + Reference Designs

Tools:• Compile designs• Verify designs• Interact with hardware

Reference designs:• Router (HW)• Switch (HW)• Network Interface Card (HW)• Router Kit (SW)• SCONE (SW)


Contributed Projects

More projects:http://netfpga.org/foswiki/NetFPGA/OneGig/ProjectTable

Project Contributor

OpenFlow switch Stanford University

Packet generator Stanford University

NetFlow Probe Brno University

NetThreads University of Toronto

zFilter (Sp)router Ericsson

Traffic Monitor University of Catania

DFA UMass Lowell


Community

Wiki• Documentation

– User’s Guide– Developer’s Guide

• Encourage users to contribute

Forums• Support by users for users• Active community - 10s-100s of posts/week


International Community

Over 1,000 users, using 2,000 cards at150 universities in 40 countries


NetFPGA’s Defining Characteristics

• Line-Rate– Processes back-to-back packets

• Without dropping packets • At full rate of Gigabit Ethernet Links

– Operating on packet headers • For switching, routing, and firewall rules

– And packet payloads• For content processing and intrusion prevention

• Open-source Hardware – Similar to open-source software

• Full source code available • BSD-Style License

– But harder, because • Hardware modules must meeting timing• Verilog & VHDL Components have more complex interfaces • Hardware designers need high confidence in specification of modules


Test-Driven Design

• Regression tests– Have repeatable results – Define the supported features– Provide clear expectation on functionality

• Example: Internet Router– Drops packets with bad IP checksum– Performs Longest Prefix Matching on destination address– Forwards IPv4 packets of length 64-1500 bytes– Generates ICMP message for packets with TTL <= 1– Defines how packets with IP options or non IPv4

… and dozens more … Every feature is defined by a regression test


Who, How, Why

Who uses the NetFPGA?– Teachers– Students– Researchers

How do they use the NetFPGA?– To run the Router Kit– To build modular reference designs

• IPv4 router• 4-port NIC• Ethernet switch, …

Why do they use the NetFPGA?– To measure performance of Internet systems– To prototype new networking systems


Summer Camp Objectives

• Overall picture of NetFPGA• How reference designs work• How you can work on a project

– NetFPGA Design Flow– Directory Structure, library modules and projects– How to utilize contributed projects

• Interface/Registers– How to verify a design (Simulation and Regression

Tests)– Things to do when you get stuck

AND… You can build your own projects!

16 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 16

Section II: Network review


Internet Protocol (IP)

Data

DataIP

Hdr

Eth Hdr

DataIP

Hdr

Data to betransmitted:

IP packets:

EthernetFrames:

DataIP

HdrData

IP Hdr

Eth Hdr

DataIP

HdrEth Hdr

DataIP

Hdr

…

…


Internet Protocol (IP)

Data

DataIP

Hdr…

16 3241

Options (if any)

Destination Address

Source Address

Header ChecksumProtocolTTL

Fragment OffsetFlagsFragment ID

Total Packet LengthT.ServiceHLenVer

20 b

ytes


Basic operation of an IP router

R3

A

B

C

R1

R2

R4 D

E

FR5

R5F

R3E

R3D

Next HopDestination

D


Basic operation of an IP router

A

B

C

R1

R2

R3

R4 D

E

FR5


Forwarding tables

Entry Destination Port

12⋮ 232

0.0.0.00.0.0.1⋮

255.255.255.255

12⋮12

~ 4 billion entries

Naïve approach:One entry per address

Improved approach:Group entries to reduce table sizeEntry Destination Port

12⋮50

0.0.0.0 – 127.255.255.255128.0.0.1 – 128.255.255.255

⋮248.0.0.0 – 255.255.255.255

12⋮12

IP address 32 bits wide → ~ 4 billion unique address


IP addresses as a line

0 232-1


12345

StanfordBerkeley

North AmericaAsia

Everywhere (default)

12345

All IP addresses

North AmericaAsia

BerkeleyStanford

Your computer My computer


Longest Prefix Match (LPM)


12345

StanfordBerkeley

North AmericaAsia


12345

Universities

Continents

Planet

DataTo:

Stanford

Matching entries:•Stanford•North America•Everywhere

Most specific


Longest Prefix Match (LPM)


12345

StanfordBerkeley

North AmericaAsia


12345

Universities

Continents

Planet

DataTo:

Canada

Matching entries:•North America•Everywhere

Most specific


Implementing Longest Prefix Match


12345

StanfordBerkeley

North AmericaAsia


12345

Most specific

Least specific

Searching

FOUND


Basic components of an IP router

Control Plane

Data Planeper-packet processing

SwitchingForwarding

Table

Routing Table

Routing Protocols

Management& CLI

Softw

areH

ardware

Queuing


IP router components in NetFPGA

SCONE

Routing Table

Routing Protocols

Management& CLI

Output PortLookup

ForwardingTable

InputArbiter

OutputQueues

Switching Queuing

Linux

Routing Table

Routing Protocols

Management& CLI

Router Kit

OR

Softw

areH

ardware

28 Hands-on Training – June 18-19, 2012 28

Section III: Example


Operational IPv4 router

Control Plane

Data Planeper-packet processing

Softw

areH

ardware

Routing Table

Routing Protocols

Management& CLI

SCONE

SwitchingForwarding

TableQueuing

Reference router

Java GUI


Streaming video


Streaming video

PC & NetFPGA(NetFPGA in PC)

NetFPGA runningreference router


Streaming video

Video streaming over shortest path

VideoclientVideo

server


Streaming video

VideoclientVideo

server

Link breaks


Streaming video

.1.1

.1.2.3.1

.30.2

.4.1

.4.2

.6.1.3.2

.7.1

.7.2

.9.1

.6.2

.10.1

.10.2

.12.1

.9.2

.13.1

.13.2

.15.1

.12.2

.16.1

.16.2.15.2

.28.1

.28.2.27.1

.30.1

.25.1

.25.2.24.1

.27.2

.22.1

.22.2.21.1

.24.2

.19.1

.19.2

.21.2.18.2

.18.1

.23.1


Observing the routing tables

Columns:•Subnet address•Subnet mask•Next hop IP•Output ports


Demo


Review

NetFPGA as IPv4 router:•Reference hardware + SCONE software•Routing protocol discovers topology

Demo:•Ring topology•Traffic flows over shortest path•Broken link: automatically route around failure


Section IV: Infrastructure


Infrastructure

• Tree structure

• NetFPGA package contents– Reusable Verilog modules– Verification infrastructure– Build infrastructure– Utilities– Software libraries


Tree Structure (1)

netfpga

binbin

liblib

projectsprojects

bitfilesbitfiles

(scripts for running simulations and setting up the environment)(scripts for running simulations and setting up the environment)

(contains the bitfiles for all projects that have been synthesized)(contains the bitfiles for all projects that have been synthesized)

(shared Verilog modules, libraries needed for simulation/synthesis/design)(shared Verilog modules, libraries needed for simulation/synthesis/design)

(user projects, including reference designs)(user projects, including reference designs)


Tree Structure (2)

lib

CC

javajava

MakefilesMakefiles

Perl5Perl5

pythonpython

scriptsscriptsverilogverilog

(common software and code for reference designs)(common software and code for reference designs)

(contains software for the graphical user interface)(contains software for the graphical user interface)

(makefiles for simulation and synthesis)(makefiles for simulation and synthesis)

(libraries to interact with reference designs, create test data, and manage simulations/regression tests)(libraries to interact with reference designs, create test data, and manage simulations/regression tests)

(common libraries to aid in regression tests)(common libraries to aid in regression tests)

(utility scripts – less commonly used than those in the bin directory)(utility scripts – less commonly used than those in the bin directory)

(modules that can be reused in designs)(modules that can be reused in designs)


Tree Structure (3)

projects/crypto_nic

docdoc

includeinclude

srcsrc

swsw

synthsynth

testtest

(project specific documentation)(project specific documentation)

(XML files defining project and any local modules, auto-generated Verilog register defines)(XML files defining project and any local modules, auto-generated Verilog register defines)

(non-library Verilog code used for synthesis and simulation)(non-library Verilog code used for synthesis and simulation)

(software elements of the project)(software elements of the project)

(project-specific .xco files to generate cores, Makefile to implement the design)(project-specific .xco files to generate cores, Makefile to implement the design)

(simulation and hardware tests)(simulation and hardware tests)

liblib (C/Perl defines for registers)(C/Perl defines for registers)


NetFPGA package contents

• Projects:– HW: router, switch, NIC, buffer sizing router– SW: router kit, SCONE

• Reusable Verilog modules• Verification infrastructure:

– simulate full board with PCI + physical interfaces– run tests against hardware– test data generation libraries (eg. packets)

• Build infrastructure• Utilities:

– register I/O, packaging, …• Software libraries


Reusable Verilog modules

Category Modules

I/O interfaces Ethernet MACCPU DMA queuesCPU register queues

MDIOPCI

Output queues SRAM-basedDRAM-based

BRAM-based

Output port lookup Router (CAM-based)Learning switch (CAM-based)

NICHardwired

Memory interfaces SRAM DRAM

Miscellaneous FIFOsGeneric register module

Rate limiter


Verification Infrastructure

• Simulation: nf_test.py sim– allows testing before synthesis– catches many bugs

• Hardware tests: nf_test.py hw– test synthesized hardware

• Test data generation libraries:– easily create test data:– many standard packet formats supported out of

the box– easily add support for custom formats


Build infrastructure

• Register system:– allocates memory to modules– generates “include” files for various languages

• Build/synthesis:– required shared modules documented XML

(shared with register system)– shared modules pulled in during synthesis– resultant bitfile checked for timing errors


Utilities

• Bitfile download: nf_download• Register I/O: regread, regwrite• Device querying: nf_info• SRAM dumping: lib/scripts/sram_dump


Software libraries

• Libraries for interfacing with NetFPGA:– C, Perl, Java, Python support


Section V: Life of a Packet


Reference Router Pipeline

• Five stages– Input– Input arbitration– Routing decision and

packet modification– Output queuing– Output

• Packet-based module interface

• Pluggable design

MACRxQMACRxQ

CPURxQCPURxQ

MACRxQMACRxQ

CPURxQCPURxQ

MACRxQMACRxQ

CPURxQCPURxQ

MACRxQMACRxQ

CPURxQCPURxQ

Input ArbiterInput Arbiter

Output Port LookupOutput Port Lookup

MACTxQMACTxQ

CPUTxQCPUTxQ

MACTxQMACTxQ

CPUTxQCPUTxQ

MACTxQMACTxQ

CPUTxQCPUTxQ

MACTxQMACTxQ

CPUTxQCPUTxQ

Output QueuesOutput Queues


Full System Components

Software

PCI Bus

NetFPGA

CPURxQCPURxQ

CPUTxQ

nf2_reg_grp

user data path

nf2c0 nf2c1 nf2c2 nf2c3 ioctl

MACTxQMACTxQ

MACRxQMACRxQ

Ethernet

CPURxQCPURxQ

CPUTxQ

CPURxQCPURxQ

CPUTxQ

CPURxQCPURxQ

CPUTxQ

MACTxQMACTxQ

MACRxQMACRxQ

MACTxQMACTxQ

MACRxQMACRxQ

MACTxQMACTxQ

MACRxQMACRxQ


port0 port2192.168.2.y192.168.1.x

Life of a Packet through the Hardware

IP packet


Inter-Module Communication

Using “Module Headers”:

IP Hdr

Eth Hdr

…

0

0

0

Last word of packet0x10

Last Module Hdry

……

Module Hdrx Contain information such as packet length, input port, output port, …

Data Word(64 bits)

Ctrl Word(8 bits)


Module i

Module i+1

data

Inter-Module Communication

ctrlwr

rdy


MAC Rx Queue

MAC Rx Queue


Rx Queue

Rx Queue

IP Hdr:IP Dst: 192.168.2.3,

TTL: 64, Csum:0x3ab4

Eth Hdr:Dst MAC = port 0,

Ethertype = IP

Data

0

0

0

Pkt length,input port = 0

0xff


Input Arbiter

Input Arbiter

Rx Q 0

Rx Q 1

…

Rx Q 7

Pkt

Pkt

Pkt


Output Port Lookup

Output Port

Lookup


Output Port

LookupIP Hdr:IP Dst: 192.168.2.3,


IP Hdr:IP Dst: 192.168.2.3,

TTL: 63, Csum:0x3ac2

Output Port Lookup

EthHdr: Dst MAC = 0Src MAC = x,

Ethertype = IP

Data

0

0

0

Pkt length,input port = 00xff

1- Check input port matches

Dst MAC

2- Check TTL, checksum

3- Lookup next hop IP & output port

(LPM)

4- Lookup next hop MAC address (ARP)

5- Add output port header

6- Modify MAC Dst and Src addresses

7-Decrement TTL and update

checksum

EthHdr: Dst MAC = nextHop Src MAC = port 4,

Ethertype = IP


output port = 4


Output Queues

Output Queues

OQ0

OQ4

OQ7


MAC Tx Queue

MAC Tx Queue


MAC Tx Queue

MAC Tx Queue

IP Hdr:IP Dst: 192.168.2.3,


IP Hdr:IP Dst: 192.168.2.3,

TTL: 63, Csum:0x3ac2

EthHdr: Dst MAC = nextHop Src MAC = port 4,

Ethertype = IP

Data

0

0

0


output port = 40xff


Exception Packets

• Example: TTL = 0 or TTL = 1

• Packet has to be sent to the CPU• Host generates an ICMP packet response• Difference starts at the Output Port Lookup



Ethernet

Exception Packet Path

CPURxQCPURxQ

CPUTxQCPUTxQ

CPURxQCPURxQ

CPUTxQCPUTxQ

CPURxQCPURxQ

CPUTxQCPUTxQ

CPURxQCPURxQ

CPUTxQCPUTxQ

nf2_reg_grpnf2_reg_grp

user data pathuser data path

MACTxQMACTxQ

MACRxQMACRxQ

MACTxQMACTxQ

MACRxQMACRxQ

MACTxQMACTxQ

MACRxQMACRxQ

MACTxQMACTxQ

MACRxQMACRxQ

Ethernet

Software

PCI Bus

NetFPGA


Output Port

LookupIP Hdr:IP Dst: 192.168.2.3,

TTL: 1, Csum:0x3ab4

Output Port Lookup

EthHdr: Dst MAC = 0,Src MAC = x,

Ethertype = IP

Data

0

0

0

Pkt length,input port = 00xff

1- Check input port matches

Dst MAC

2- Check TTL, checksum – EXCEPTION!

3- Add output port module


output port = 1


Output Queues

Output Queues

OQ0

OQ1

OQ2

OQ7


CPU Tx Queue

CPU Tx Queue


CPU Tx Queue

CPU Tx Queue

IP Hdr:IP Dst: 192.168.2.3,

TTL: 1, Csum:0x3ab4

EthHdr: Dst MAC = 0, Src MAC = x,

Ethertype = IP

Data

0

0

0


output port = 10xff


ICMP Packet

• Packet arrives at the CPU Rx Queue from the PCI Bus

• Same path as a packet from the MAC until it reaches the Output Port Lookup (OPL)

• The OPL module sees the packet is from the CPU Rx Queue 1 and sets the output port directly to 0

• The packet continues on the same path as the non-exception packet to the Output Queues and then MAC Tx queue 0



Ethernet

ICMP Packet Path

CPURxQCPURxQ

CPUTxQCPUTxQ

CPURxQCPURxQ

CPUTxQCPUTxQ

CPURxQCPURxQ

CPUTxQCPUTxQ

CPURxQCPURxQ

CPUTxQCPUTxQ

nf2_reg_grpnf2_reg_grp

user data pathuser data path

MACTxQMACTxQ

MACRxQMACRxQ

MACTxQMACTxQ

MACRxQMACRxQ

MACTxQMACTxQ

MACRxQMACRxQ

MACTxQMACTxQ

MACRxQMACRxQ

Ethernet

Software

PCI Bus

NetFPGA


NetFPGA-Host Interaction

• Linux driver interfaces with hardware– Packet interface via standard Linux network

stack

– Register reads/writes via ioctl system call with wrapper functions:

• readReg(nf2device *dev, int address, unsigned *rd_data);• writeReg(nf2device *dev, int address, unsigned *wr_data);

eg:readReg(&nf2, OQ_NUM_PKTS_STORED_0, &val);



NetFPGA to host packet transferNetFPGA to host packet transfer

PC

I Bu

sP

CI B

us

2. Interrupt notifies driver of packet arrival

2. Interrupt notifies driver of packet arrival

3. Driver sets up and initiates DMA transfer


1. Packet arrives – forwarding table sends to CPU queue

1. Packet arrives – forwarding table sends to CPU queue



NetFPGA to host packet transfer (cont.)NetFPGA to host packet transfer (cont.)

PC

I Bu

sP

CI B

us

4. NetFPGA transfers packet via DMA

4. NetFPGA transfers packet via DMA

5. Interrupt signals completion of DMA


6. Driver passes packet to network stack6. Driver passes packet to network stack



Host to NetFPGA packet transfersHost to NetFPGA packet transfers

PC

I Bu

sP

CI B

us



1. Software sends packet via network sockets

Packet delivered to driver

1. Software sends packet via network sockets

Packet delivered to driver





Register accessRegister access

PC

I Bu

sP

CI B

us

1. Software makes ioctl call on network socket

ioctl passed to driver

1. Software makes ioctl call on network socket

ioctl passed to driver

2. Driver performs PCI memory read/write

2. Driver performs PCI memory read/write



• Packet transfers shown using DMA interface

• Alternative: use programmed IO to transfer packets via register reads/writes– slower but eliminates the need to deal with

network sockets


Section VI: Example Project


Project: Cryptographic NIC

Implement a network interface card (NIC) that encrypts upon transmission and

decrypts upon reception


Cryptography

XOR function

XOR written as: ^ ⊻ ⨁XOR is commutative: (A ^ B) ^ C = A ^ (B ^ C)

A B A ^ B

0 0 0

0 1 1

1 0 1

1 1 0

XORing a value with itself always yields 0


Cryptography (cont.)

Simple cryptography:– Generate a secret key– Encrypt the message by XORing the message and key– Decrypt the ciphertext by XORing with the key

Explanation:

(M ^ K) ^ K = M ^ (K ^ K)

= M

= M ^ 0

Commutativity

A ^ A = 0



Example:

Message: 00111011

Key: 10110001

Message ^ Key: 10001010

Key: 10110001

Message ^ Key ^ Key: 00111011



Idea: Implement simple cryptography using XOR– 32-bit key– Encrypt every word in payload with key

Note: XORing with a one-time pad of the same length of the message is secure/uncrackable. See: http://en.wikipedia.org/wiki/One-time_pad

PayloadHeader

Key Key Key Key Key

⨁


Section VII: Implementation


Getting started with a new project (1)

• Projects:– Each design represented by a project

– Location: netfpga/projects/<proj_name>• netfpga/projects/crypto_nic

– Consists of:• Verilog source• Simulation tests• Hardware tests• Libraries• Optional software



– Normally:• copy an existing project as the starting point

– Today:• pre-created project

– Missing from pre-created project:• Verilog files (with crypto implementation)• Simulation tests• Hardware tests• Custom software



Typically implement functionality in one or more modules inside the user data path

MACRxQMACRxQ

CPURxQCPURxQ

MACRxQMACRxQ

CPURxQCPURxQ

MACRxQMACRxQ

CPURxQCPURxQ

MACRxQMACRxQ

CPURxQCPURxQ

Input ArbiterInput Arbiter

Output Port LookupOutput Port Lookup

MACTxQMACTxQ

CPUTxQCPUTxQ

MACTxQMACTxQ

CPUTxQCPUTxQ

MACTxQMACTxQ

CPUTxQCPUTxQ

MACTxQMACTxQ

CPUTxQCPUTxQ

Output QueuesOutput Queues

CryptoCrypto

Crypto module to encrypt and decrypt packets

User data path



– Shared modules included from netfpga/lib/verilog• Generic modules that are re-used in multiple projects• Specify shared modules in project’s include/project.xml

– Local src modules override shared modules

– crypto_nic:

Local Shared

user_data_path.vcrypto.v

Everything else


Exploring project.xml (1)

• Location: project/<proj_name>/include<?xml version="1.0" encoding="UTF-8"?><nf:project …>

<nf:name>Crypto NIC</nf:name>

<nf:description>NIC with basic crypto support</nf:description>

<nf:version_major>0</nf:version_major> <nf:version_minor>1</nf:version_minor> <nf:version_revision>0</nf:version_revision>

<nf:dev_id>0</nf:dev_id>

Short nameDescription

Version information•indicate bitfile version

Unique ID to identify projectSee: http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/DeviceIDList


Exploring project.xml (2) <nf:use_modules> core/io_queues/cpu_dma_queue core/io_queues/ethernet_mac core/input_arbiter/rr_input_arbiter core/nf2/generic_top core/nf2/reference_core core/output_port_lookup/nic core/output_queues/sram_rr_output_queues core/sram_arbiter/sram_weighted_rr core/user_data_path/reference_user_data_path core/io/mdio core/cpci_bus core/dma core/user_data_path/udp_reg_master core/io_queues/add_rm_hdr core/strip_headers/keep_length core/utils/generic_regs core/utils </nf:use_modules>

Shared modules toload from lib/verilog


Exploring project.xml (3) <nf:memalloc layout="reference"> <nf:group name="core1"> <nf:instance name="device_id" /> <nf:instance name="dma" base="0x0500000"/> <nf:instance name="mdio" /> <nf:instance name="nf2_mac_grp" count="4" /> <nf:instance name="cpu_dma_queue" count="4" /> </nf:group> <nf:group name="udp"> <nf:instance name="in_arb" /> <nf:instance name="crypto" /> <nf:instance name="strip_headers" /> <nf:instance name="output_queues" /> </nf:group> </nf:memalloc></nf:project>

Specify where to instantiate modules, the number of instances, and the memory addresses to use



Tasks:Set the project that we’ll be working with:

1. Add the following lines to the end of ~/.bashrc:export NF_DESIGN_DIR=$NF_ROOT/projects/crypto_nicexport

PERL5LIB=$NF_ROOT/lib/Perl5:$NF_DESIGN_DIR/lib/Perl5

2. Type: source ~/.bashrc

Copy reference files as starting points:

3. Copy the following files from netfpga/lib/verilog/core into netfpga/projects/crpyto_nic/src

user_data_path/reference_user_data_path/src/user_data_path.vmodule_template/src/module_template.v



Create crypto.v from module_template.v:

1. Rename the local module_template.v to crypto.v2. Change the module name inside crypto.v (first non-

comment line of the file)

3. Add the crypto module to the user data path


user_data_path.v (1)module user_data_path #( parameter DATA_WIDTH = 64, ... ) ( ... )

//------------------ Internal parameters ----------------------- ...

//----------------- Input arbiter wires/regs ------------------- ...

Module port declaration


user_data_path.v (2) //-------------- output port lut wires/regs -------------------- wire [CTRL_WIDTH-1:0] op_lut_in_ctrl; wire [DATA_WIDTH-1:0] op_lut_in_data; wire op_lut_in_wr; wire op_lut_in_rdy;

...

//------- output queues wires/regs ------ ...

Wire declarations for the output port lookup module.Duplicate this section, and replace op_lut with crypto


user_data_path.v (3) //--------- Connect the data path ----------- input_arbiter #( ... ) input_arbiter ( ... )

output_port_lookup #( ... ) output_port_lookup ( ... )

...

Module instantiations.

1.Duplicate the output_port_lookup instantiation 2.Rename to crypto3.Remove all parameters (inside the first set or parentheses)4.In the output_port_lookup instantiation, replace oq_ with crypto_5.In the crypto instantiation, replace op_lut_ with crypto_

We’ve inserted the new module into the pipeline



Run a simulation to verify changes:

1. nf_test.py sim --major nic --minor short

Now we can implement the crypto functionality


Implementing the Crypto Module (1)

• What do we want to encrypt?– IP payload only

• Plaintext IP header allows routing• Content is hidden

– Encrypt bytes 35 onward• Bytes 1-14 – Ethernet header• Bytes 15-34 – IPv4 header (assume no options)

– Assume all packets are IPv4 for simplicity



• State machine (draw on next page):– Module headers on each packet– Datapath 64-bits wide

• 34 / 8 is not an integer! • Inside the crypto module


Crypto Module State Diagram

Hint: We suggest 4 states (or 3 if you’re feeling adventurous)

SkipModuleHeaders



Implement your state machine inside crypto.v– Use a static key initially

Suggested sequence of steps:1. Create a static key value

• Constants can be declared in the module with localparam:localparam MY_EXAMPLE = 32’h01234567;

2. Implement your state machine without modifying the packet

3. Update your state machine to modify the packet by XORing the key and the payload

• Use two copies of the key to create a 64-bit value to XOR with data words


module_template.v (1)module module_template #( parameter DATA_WIDTH = 64, parameter CTRL_WIDTH = DATA_WIDTH/8, parameter UDP_REG_SRC_WIDTH = 2 ) ( ... )

//----------------------- Signals---------------------------- ...

//------------------ Local assignments ----------------------- ...

Module port declaration


module_template.v (2) //------------------------- Modules-------------------------------

fallthrough_small_fifo #( .WIDTH(CTRL_WIDTH+DATA_WIDTH), .MAX_DEPTH_BITS(2) ) input_fifo ( .din ({in_ctrl, in_data}), // Data in .wr_en (in_wr), // Write enable .rd_en (in_fifo_rd_en), // Read the next word .dout ({in_fifo_ctrl, in_fifo_data}), .full (), .nearly_full (in_fifo_nearly_full), .prog_full (), .empty (in_fifo_empty), .reset (reset), .clk (clk) );

Packet data dumped in a FIFO. Allows some “decoupling” between input and output.


module_template.v (3) generic_regs #( .UDP_REG_SRC_WIDTH (UDP_REG_SRC_WIDTH), .TAG (0), .REG_ADDR_WIDTH (1), .NUM_COUNTERS (0), .NUM_SOFTWARE_REGS (0), .NUM_HARDWARE_REGS (0) ) module_regs ( ... );

Generic registers.

Ignore for now – we’ll explore this later


module_template.v (4) //------------------------- Logic-------------------------------

always @(*) begin // Default values out_wr_int = 0; in_fifo_rd_en = 0;

if (!in_fifo_empty && out_rdy) begin out_wr_int = 1; in_fifo_rd_en = 1; end end

Combinational logic to read data from the FIFO. (Data is output to output ports.)

You’ll want to add your state in this section.


Inter-module Communication

Module i+1

`

Module i

datadata

ctrlctrlwrwrrdyrdy



Implement your state machine inside crypto.v– Use a static key initially

Suggested sequence of steps:1. Create a static key value

• Constants can be declared in the module with localparam:localparam MY_EXAMPLE = 32’h01234567;

2. Implement your state machine without modifying the packet

3. Update your state machine to modify the packet by XORing the key and the payload

• Use two copies of the key to create a 64-bit value to XOR with data words


Section VIII: Simulation and Debug


Testing: Simulation (1)

• Simulation allows testing without requiring lengthy synthesis process

• NetFPGA simulation environment allows:– Send/receive packets

• Physical ports and CPU– Read/write registers– Verify results

• Simulations run in ModelSim/VCS/ISim



• Simulations located in project/test• Multiple simulations per project

– Test different features• Example:

– crypto_nic/test/both_nic_short• Send one packet from CPU, expect packet out

physical port• Send one packet in physical port, expect packet to

CPU

Note: This test will not work once your crypto module is implemented!



Useful functions:Register access:

nftest_regwrite(addr, value)nftest_regread_expect(addr, expect)

Packet generation:make_IP_pkt(…) – see documentationencrypt_pkt(key, pkt)decrypt_pkt(key, pkt)

Packet transmission/reception:nftest_send_phy(interface, pkt)nftest_expect_phy(interface, pkt)nftest_send_dma(interface, pkt)nftest_expect_dma(interface, pkt)



Task:Implement tests for encryption and decryption

Modify the following tests:netfpga/projects/crypto_nic/test/both_crypto_encrypt/run.pynetfpga/projects/crypto_nic/test/both_crypto_decrypt/run.py

Look at both_nic_short as an example of creating IP packets and sending/receiving them


Running Simulations

• Use command nf_test.py– Required Parameter

• sim or hw (right now only use sim)– Optional parameters

• --major <major_name>• --minor <minor_name>• --gui (starts the default viewing environment)

both_crypto_encrypt

majormajor minorminor


Running Simulations

Non-GUI execution example:# 10756.00ns testbench.host32.service_interrupt: Info: Interrupt signaled# 10935 Host read 0x00000044 with cmd 0x6: Disconnect with Data, # 10995 CPCI Interrupt: DMA ingress xfer complete# 11175 Host read 0x00000148 with cmd 0x6: Disconnect with Data, # 11415 Host read 0x00000150 with cmd 0x6: Disconnect with Data, # 11475.00ns testbench.host32.service_interrupt: Info: DMA ingress transfer complete. # 11655 Host read 0x00000040 with cmd 0x6: Disconnect with Data, # Timecheck: 13645.00ns# 20100 Simulation has reached finish time - ending.# ** Note: $finish : /home/summercamp/netfpga/lib/verilog/core/testbench/target32.v# Time: 20100 ns Iteration: 0 Instance: /testbench/target32--- Simulation is complete. Validating the output.

Comparing simulation output for port 1 ...Port 1 matches [1 packets]Comparing simulation output for port 2 ...Port 2 matches [0 packets]

--- Test PASSED (test_nic_short) Test test_nic_short passed!------------SUMMARY---------------PASSING TESTS:

test_nic_shortFAILING TESTS: TOTAL: 1 PASS: 1 FAIL: 0


Running Simulations

GUI execution example:


Running Simulations

GUI execution example (cont)Try the following:

nf_test.py sim --major crypto --minor encrypt –gui

In the transcript window of the GUI:do wave.dorun 10us

You should see waveforms of packets going in and coming out of the crypto module


Running Simulations

• When running ModelSim interactively:– Click "no" when simulator prompts to finish

– Changes to code can be recompiled without quitting ModelSim:

• bash# cd /tmp/$(whoami)/test/<projname>; make model_sim• VSIM 5> restart -f; run -a

– Ensure $NF_DESIGN_DIR is correct


Section IX: Conclusion


Nick McKeown, Glen Gibb, Jad Naous, David Erickson, G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller,

Paul Hartke, Neda Beheshti, Sara Bolouki, James Zeng, Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo,

Sam D’Amico

Acknowledgments

NetFPGA Team at Stanford University (Past and Present):

NetFPGA Team at University of Cambridge (Past and Present):

Andrew Moore, David Miller, Martin Zadnik, Muhammad Shahbaz

All Community members (including but not limited to):

Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek, Yahsar Ganjali, Martin Labrecque, Jeff Shafer,

Eric Keller , Tatsuya Yabe, Bilal Anwer,Yashar Ganjali, Martin Labrecque

Ram Subramanian, Kees Vissers, Michaela Blott, Shep Siegel


Special thanks to our Partners:

Other NetFPGA Tutorials Presented At:

SIGMETRICS

Ram Subramanian, Patrick Lysaght, Veena Kumar, Paul Hartke, Anna Acevedo

Xilinx University Program (XUP)

See: http://NetFPGA.org/tutorials/


Thanks to our Sponsors:

• Support for the NetFPGA project has been provided by the following companies and institutions

Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in these materials do not necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this project.

Documents

NetFPGA Hands-on Training Day 1