OFED (iWarp) Enhancements Felix Marti, felix@chelsio.com

Open Fabrics Alliance WorkshopSonoma, April 2008

ChelsioChelsioCommunicationsCommunications

Verbs

2Verbs should remain/become transport neutralVerbs should remain/become transport neutral

OFED does not support all iWarp Verbs; MIA are Stag Alloc Stag Dealloc Fast Register Non-Shared Memory Region RDMA Read Invalidate Local Stag Invalidate Local Stag

User and Kernel space availability Currently Bind Memory Window Kernel only?

APIs need to enable ‘fast path’ QP operations Currently Bind MW not QP operation?`

Memory Handling Verbs

OFED consumers ask for better Memory Handling capabilities i.e. NFSoverRDMA Memory Handling Dilemma:

Table courtesy of Tom Talpey

3

iWarp Connection Setup

4

iWarp connections start life as TCP connections

Endpoints negotiate to upgrade to iWarp

Upgrade requires ‘initiator’ to send first iWarp PDU

Some (many) ULPs/Apps don’t adhere to the requirement – ahhh, fix the ULPs/Apps, pleeze

iWarp Connection Setup (2)

OFED only solution – do we need standardization (IETF)?OFED only solution – do we need standardization (IETF)?5

Proposal (referred to as Peer 2 Peer support): Negotiate p2p using MPA private data MPA initiator advertises if it supports p2p MPA responder advertises if it wants RTR and what

type (OB RDMA Read, RDMA Write or Send) MPA responder defers ‘initiating iWarp operations’

until RTR is received MPA initiator sends RTR after successfully

processing the MPA start response

Enhance RDMA-CM API to allow ULP/App to enable/disable p2p

Latest Chelsio iWarp Projects

Chelsio is heavily investing into iWarp/OFED!Chelsio is heavily investing into iWarp/OFED!6

Peer 2 Peer Connection upgrade

OpenMPI (and the other MPIs)

RDS

NFSoverRDMA

Future

Call To Action! Help Out! Maybe we can donate HW?Call To Action! Help Out! Maybe we can donate HW?7

ULPs/Apps should migrate to RDMA-CM to take advantage of iWarp i.e. ib_* tests shipping in OFED

As OFED becomes more transport neutral, life becomes easier

[4-tuple sharing with the host stack]

Thank You!

8