Bridging the Software/Hardware Forwarding Divide

Daekyeong Moon Jad Naous Junda Liu Kyriakos Zarifis Martin Casado

Teemu Koponen Scott Shenker Lee Breslau

1 IntroductionAs network speeds increase, and network requirementsdiversify, networking hardware vendors are being pressedon two distinct fronts. To remain competitive, they mustcontinually improve their cost/performance ratio at a pacefaster than Moores Law. Over the past fifteen years,enterprise network speeds have increased by three ordersof magnitude, from 10Mbps to 10Gbps, requiring that thecost of a unit of bandwidth (a gigabit port, or Gport) dropprecipitously. At the same time, networks are expected toaddress myriad application, management and securityrequirements that were unheard of fifteen years ago,creating pressure to develop network forwarding devicesthat can flexibly accommodate these new demands as theyarise. This trend is reflected in the many open platforminitiatives launched by system vendors [10, 16, 22].

Despite significant research and engineering efforts,the industry has failed to simultaneously achieve lowcost/performance and high flexibility.1 There are threebasic approaches to building network forwarding devicescurrently pursued in the field and, as we relate below, theyoffer quite different tradeoffs between these two goals.

In the commercial world, where cost concerns domi-nate, the most prevalent approach to building networkforwarding devices (this includes both switches androuters, but hereafter we will use the term switchas shorthand) involves embedding the basic forwardinglogic in ASICs. There are commodity forwardingchips (from, for example, Broadcom, Marvell, andFulcrum) that cost as little as $20/10Gport, a price-point that would have been unthinkable only a few yearsago. Multi-layer switches using these commodity chipssupport 240Gbps (24x10Gbps) of capacity for under$100/10Gport, and 640Gbps (64x10Gbps) chipsets willbe generally available this year which should lower thecost even further. However, one cannot modify thehardware forwarding algorithm without respinning thechip. This forces packet forwarding to evolve on hardwareUC Berkeley, Stanford University, Nicira Networks,

International Computer Science Institute (ICSI), AT&T Research1As we clarify later, we are not addressing network appliances that

provide deep packet inspection. Our concern here is only for packetforwarding decisions: to which output port does this packet go, and howshould its header be rewritten? When we refer to flexibility, we meanthe flexibility in forwarding algorithms, not general programmabilityfor sophisticated packet processing.

design timescales, which are glacially slow compared tothe rate at which network requirements are changing.

In the research community, where flexibility isparamount, there has been renewed interest in purelysoftware-based switches (i.e., switches runningcommodity operating systems on general-purpose CPUs);see, for example, [11]. These designs have the requisiteflexibility, since new forwarding algorithms merelyrequire additional programming, but their port-densityand cost/performance remains poor. For example,software routers on general CPUs have been shownto support 10Gbps of trivial forwarding for minimumsize packets [11], achieving roughly $1000/10Gport, anorder of magnitude more than the ASIC-based switches.While prevalent in low-speed wireless devices, puresoftware switches have made almost no penetration intothe high-speed, high-fanout wireline market.

The industry has tried to bridge the divide betweenthese two extremes with network processors, whosegoal was to provide programmable functionality at near-commodity hardware costs. Unfortunately, networkprocessors have not realized this goal, as designershave yet to find a sweet-spot in the tradeoff betweenhardware simplicity and flexible functionality. Thecost/performance ratios have lagged well behind com-modity networking chips and the interface providedto software has proven hard to use, requiring protocolimplementors to painstakingly contort their code to theidiosyncrasies of the particular underlying hardware toachieve reasonable performance.2 These two problems(among others) have prevented network processors fromdislodging the more narrowly targeted commodity packet-forwarding hardware that dominates the market today.One area in which network processors are widely used isin network appliances, which go beyond mere forwardingto do deep packet inspection; but even here, afterinitial success, network processors are starting to losemarket share as the forwarding performance of general-purpose CPUs has significantly improved. Today manymarket-leading middleboxes (such as load balancers) areimplemented on standard x86 platforms.

In this paper, we try again to bridge thesoftware/hardware divide, proposing a hybrid design

2However, CAFE [20] is a promising approach for datacenterforwarding.

1

that attempts to combine the high speed and low costof hardware with the superior flexibility of software.We motivate our approach by noting that currenthardware-accelerated packet-forwarding implements theforwarding logic in hardware. In contrast, in our approachall forwarding decisions are first made in software andthereafter imitated by hardware. The hardware uses aclassification engine to match software-made decisionswith the incoming packets to which they apply (e.g.,all packets destined for the same prefix). Thus, thehardware does not need to understand the logic of packetforwarding, it merely stores the results of forwardingdecisions (previously made by software) and appliesthem to packets with the appropriate headers. In short,hardware caches the decisions made by software andexecutes them at high-speed.

Similarly, the forwarding software is not tied to aparticular hardware implementation. Instead, forwardingalgorithms are programmed against a high-level API,which increases portability and reduces the complexity ofimplementation. This forwarding software is not speed-critical, so it can be written in a high-level language (inour implementations we have used Python).

For lack of a better term, we call this decision-cachingapproach Software-Defined Forwarding (SDF), sinceall forwarding decisions are made in software. Todemonstrate the viability of SDF, we built a completesystem in hardware and software. On top of it, wehave implemented a number of conventional forwardingalgorithms (e.g., L2 Ethernet forwarding and L3 LPMforwarding), ported an existing code base (XORP), aswell as implemented several recently proposed algorithms(SEATTLE, i3, Chord). We have also implementeda virtualization layer in software that allows multipleforwarding algorithms to operate in parallel on the samehardware; doing so did not require any change to thehardware forwarding model.

It is important at this point to clarify the relationshipbetween this work and ongoing work on OpenFlow [21]and NOX [14].3 The goal of OpenFlow is to providean API to the hardware that is sufficiently general fornetwork innovation. The goal of NOX is to build acentralized network operating system on top of thishardware interface. Our project lies somewhere inbetween.

In contrast to OpenFlow, we are not focusing onthe definition of the hardware interface, but instead areinvestigating how to build the hardware and softwarefor a router-like system in which the forwarding logic isinsulated from the hardware. This requires us to focus ona high-level API instead of just the low-level hardware

3This discussion also applies to Orphal [22]; there are importanttechnical differences between Orphal and OpenFlow but they are notrelevant for our discussion here.

interface. We envision OpenFlow as a natural candidatefor the underlying hardware interface to our higher-layersoftware forwarding layer, but dont limit ourselves tothat choice since our point is more general.

In contrast to the NOX, we are not advocating acentralized management paradigm, but only whethersingle-box routers and switches could be built in a farmore flexible manner.4 To this end, we provide trace-based performance numbers of this approach, which canbe seen as evidence that an OpenFlow-like hardwareinterface provides sufficient performance for a varietyof todays forwarding tasks.

This approach was previously discussed in [8], but thestudy provided here is far more detailed and comprehen-sive. Our approach also has similarities with flow caching,and we explain the difference shortly. Because ourapproach uses packet classification as a basic primitive,we rely on the vast body of work in this area.

2 Goals, Limitations, and ApplicabilityBefore delving into design details and performanceresults, we first articulate SDFs goals, limitations andapplicability.

Goals: The immediate goal of the SDF approach isto build switches with more forwarding flexibility byhaving all forwarding decisions determined completelyin software and then imitated by hardware. This willallow networks to deploy new forwarding algorithms(such as to support a new protocol) without changing theirnetworking hardware. In fact, as we discuss later, thiswill enable networks to run several different forwardingalgorithms in parallel on the same switch.

The longer term goal is to create an environment whereboth hardware and software designers can independentlyfocus on their respective tasks. Hardware designerscan focus on building decision caches with higherspeeds, greater fanouts, larger memory, lower power andlower cost. Similarly, software designers can addressemerging networking requirements by implementingnew forwarding algorithms against a simple and fixedhardware interface. This strict separation of concernsshould engender rapid progress in both hardware andsoftware.

Limitations: We reiterate that the SDF approach onlysupports standard header-based forwarding and does notapply to more complex networking behaviors such asdeep-packet inspection or arbitrary packet modification(like encryption).

More specifically, the forwarding decisions can dependonly on the header of the incoming packet (where the

4The NOX approach is sometimes referred to as Software-DefinedNetworking; our use of the term Software-Defined Forwardingemphasizes this distinction between our narrow focus on forwarding ina single box, and NOXs emphasis on global network abstractions.

2

definition of header is the first n bits of the packet,where n is flexible) and on the internal state of theforwarding logic implementation. The output packetscan differ from the incoming packet in the content (andthe length) of the header, but this new header contentcan not depend on any forwarding logic state modifiedper packet (that is, for all packets matching a particularentry, the headers must be modified in the same way).Thus, the model does not include, for example, en/de-capsulation, en/de-cryption, nor complex modificationsto the packet payload. Such operations requiring uniquemodifications per packet are handled outside the model,by a logical output port to which the packet is forwardedto. We note this is consistent with the model commonlyused in routers and switches of using logical interfacesfor tunneling (e.g., IPsec or GRE).

While the packet processing model is limited, we havebeen surprised by how broadly it applies to traditional andnewly proposed forwarding paradigms. For example, theSDF approach can implement aggregation, basic tagging,address overwriting (e.g., VLAN, or layered addressing),dynamic learning and filtering, all without changes to thehardware forwarding model.

Applicability: The SDF approach is most relevant insettings that need a high degree of flexibility. Research isan obvious arena where it would be important to deploynew forwarding algorithms, running at high speeds oncommercial-grade equipment, merely by rewriting theforwarding software. In addition, the ability to run severalforwarding algorithms in parallel, as is envisioned inGENI and other testbeds, would be very valuable.

It is an open question whether production networksneed this degree of flexibility. Some have predicted thatthe future of networking lies in the ability to virtualizelow-level packet transport, in which case this flexibilitywould be crucial. But it is also true that current forwardingalgorithms evolve slowly, which suggests that perhapsflexibility is not relevant to production networks. Thequestion is whether this slow evolution reflects theinability to evolve more quickly or the lack of a needto. We dont pretend to know the answer.

Since SDF, as described so far, relies on decision-caching, it would be natural to assume that its appli-cability is limited to regimes where caching is effective.However, as we describe later, SDF can also be run in aproactive mode, where all software forwarding decisionsare made proactively and stored in hardware. In this case,SDF functions at hardware speeds.

There are some cases where, in order to reduce the staterequired to cache decisions, it would be advantageousto augment the hardware with a few basic computa-tional primitives such as decrement by 1, computea checksum, or compute a hash that can be appliedto a specified set of bits. These primitives are trivial

to implement in hardware, and would only be requiredin cases where performance was critical (e.g., Internetbackbone). We discuss this more fully in Section 4.

3 OverviewIn this section we present a high-level overview of ourapproach, leaving design details for Section 6 and adescription of our implementation for Section 7. Belowwe first discuss the system functionality and components,then walk through the steps in forwarding a packet.

3.1 System Functionality and Components

Figure 1 provides a high-level view of the proposedforwarding architecture: it contains three components:the system software, the forwarding hardware, and theforwarding software.

In SDF, a forwarding device accepts an (inport, in-packet) pair, and then outputs one or more (outport,outpacket) pairs. The hardware performs packet headerlookup, header rewriting, and forwarding (passing thepacket from the input port to the appropriate outputport). In our implementation, a central component ofthe hardware is a wide TCAM, which performs packetclassification based on the packet header (e.g., the first576 bits of the packet); and, if a match is found, thehardware sends the packet to the specified output port(s)after modifying the packet headers. The hardware alsomaintains hit counters for TCAM entries but it does notmanage the entries without instructions from the systemsoftware; in particular, it does not implement inactivitytimeouts of any sort.

Forwarding logic (in the form of a forwarding applica-tion) is built on the API provided by the system software.The system software is responsible for translating theforwarding applications decisions into entries in thepacket classification engine (a TCAM in our design); itis also responsible for revoking these entries when therelevant internal forwarding application state changes.More specifically, for a forwarding decision the systemrecords a) the incoming port, b) the relevant header bits ofthe incoming packet (i.e., which header bit should matchand what the bit values should be in order for the decisionto apply), c) the outgoing ports, and d) the associated

System Software

Forwarding Application

Forwarding Hardware

Figure 1: Packets can take one of two paths through the system. Apacket matching an entry in the hardware is immediately forwarded outthe appropriate port(s). A received packet without a matching entry inthe hardware is passed to the forwarding application, which makes aforwarding decision and sends the packet out. The system software thenstores this decision in the hardware.

3

outgoing packet headers. It also records what forwardingapplication internal state the decision was based on, so itcan revoke the classification engine entry correspondingto the decision when the internal state changes. Forexample, with L3 forwarding, the forwarding decisiondepends on the matching (software) FIB entry, and if thatchanges the TCAM entry would be revoked.

The forwarding application is written in a high-levelsuch as C, Java, or even Python. Our goal is to provide aprogramming model which approaches that of a puresoftware router. However, due to the difficulty ininferring state dependencies precisely and efficiently, inour current implementation the forwarding applicationhas the responsibility to explicitly mark which header bitsand internal state it relied on in a forwarding decision. Wediscuss the API in more detail in Section 6.

3.2 Forwarding Steps

We now step through a simplified IP forwarding decisionto illustrate hardware state setup and revocation. Forclarity we ignore TTL decrement and header checksumrecomputation (and discuss them in Section 4).

1. A packet is received on port0.

2. Packet classification hardware checks the incomingport and the packet header against its lookup table.Assuming there is no matching entry, the hardwareforwards the packet to the system software whichpasses it on to the forwarding application.

3. The forwarding application checks for the Ethernettype and reads the destination IP address.

4. The forwarding application looks into its software-FIB and finds a matching entry.

5. The forwarding application reads the destinationMAC from the ARP cache and overwrites thedestination MAC in the packet header with it.

6. The forwarding app sends the packet out of port1.

7. The system software intercepts the packet, savingthe new packet header with any changes it may have.

8. The system software creates a hardware packetclassification entry which matches on the Ethernettype, and the destination IP address. It creates andassociates with the entry an action of overwriting theheader with the new header (which has a modifieddestination MAC), and the action of sending thepacket out of port1.

9. The system software also maintains a mapping fromthe dependent software-FIB and ARP entries to thenew hardware forwarding entry.

10. Assuming a second packet is received to the samedestination IP address (whether or not from thesame transport connection), the hardware will match

on the Ethernet type and destination IP address,overwrite the header with the correct destinationMAC, and forward it out of port1.

11. If the FIB or ARP entries are modified (e.g., routingupdate or timeout), the system software removes theassociated forwarding entry from the hardware.

3.3 Sound Familiar?

Superficially, SDF seems much like flow or route caching,a technique popular a decade ago (see [17] for a recentrevival of the concept). Roughly, flow caching expects thesoftware to use the first packet of every network flow tomake a complex forwarding decision (such as IP lookupusing LPM) after which the flow is added to a flow cache.The cache entry is only valid for the duration of that flow.However, due to the small average flow sizes (e.g., 10packets), hit rates tend not to be sufficient for overcomingthe difference between hardware and software processingspeeds (i.e., the switch is limited by the rate at whichsoftware can forward, because it must handle roughly oneout of every ten packets). Our approach differs from theclassic flow and route caching in three essential ways.

First, we dont couple hardware state to network flows,but to forwarding decisions (which usually apply to manyflows and do not time out, but instead are explicitlyrevoked as needed). Thus, the forwarding hardwareexperiences vastly lower miss rates, similar to othersoft state commonly found in networks today such asL2 learning tables and ARP caches. We explore theperformance of decision caching in Section 5.

Second, flow caching is generally tied to a particularforwarding mechanism (traditionally LPM) and was notdesigned to increase the flexibility of the system, merelyto reduce state requirements. In contrast, our approachis intended to support arbitrary forwarding logic overarbitrary headers with the same simple hardware.

Finally, the SDF approach is not limited to reactivelypopulating the hardware forwarding state after a cachemiss, as is done in route/flow caching. As we describelater in Section 6, our proposed implementation supportsproactively pushing decisions into the hardware beforeany cache miss occurs. Doing so avoids all caching-related performance issues and allows the system toforward at hardware speeds. Thus, while we believeSDF is capable of running reactively in many produc-tion deployment environments, it can also be used inenvironments where caching is not applicable.

To relate our approach to OpenFlow, note that Open-Flow defines a low-level protocol interface to the hard-ware while SDF defines a high-level programming API.Forwarding applications should be written to the SDFinterface, and in turn SDF could use OpenFlow asan interface to manage the switch hardware, or coulduse the chip vendors native SDK. Writing forwarding

4

applications directly to OpenFlow would limit them toOpenFlow-based switches and require them to manuallymanage the flow table: insert entries, keep track of statechanges, and revoke entries as needed.

4 Example Use CasesThe goal of SDF is to allow software to define a widerange of forwarding behaviors, all of which can thenbe accelerated by the same underlying hardware. Toillustrate the kinds of forwarding that can be supported,below we describe several example use cases.

4.1 Accelerating Existing Software Routers

We first briefly explain how existing software routers suchas XORP and Quagga can integrate with SDF. Softwarerouters typically consist of a control plane (which imple-ments a set of routing/switching protocols) that exportsthe resulting FIB to a forwarding plane such as XORPsForwarding Engine Abstraction. To port a software routerto SDF, it is sufficient to provide the control plane with aforwarding plane implementation built on the APIs SDFprovides (which we describe in Section 6). This SDF-based forwarding plane implementation stores the FIBit receives from the control plane and makes forwardingdecisions when invoked by the SDF system software. Aslong as the software router has a well-defined forwardingengine abstraction, implementing the forwarding enginebased on SDFs APIs is easy.

To demonstrate the simplicity of the task, we modifiedXORPs FEA to push its FIB entries over TCP to aPython forwarding application built on top of the SDFAPIs. In our prototype implementation, adding therequired TCP integration mostly involved copy-pastingfrom the existing FEA source code, while the forwardingapplication required only 220 lines of Python.

Thus, we think SDF offers an extremely easy path toprovide existing software routers hardware acceleration.

4.2 Virtualization

There is a growing research literature (see [1, 3, 26] for asmall sampling) on network virtualization, which is theability to run separate network architectures in paralleleach within their own slice of the network. These slicescan implement their own routing and forwarding algo-rithms, and essentially operate as independent networks.5

To create slices, the traffic must be partitioned in someway so that it is clear which packets belong to whichslices. This demultiplexing criterion can be defined interms of input ports, VLANs or other packet header fields.

5We are using the term network virtualization as it is primarilyused in the research community. In the commercial world, networkvirtualization does not involve different forwarding algorithms but ismostly focused on sharing physical infrastructure, managing logical linktopology, and supporting virtual server mobility.

Forwarding Hardware

Master SystemSoftware

Pkts Pkts(De)Muxing

System Software

ForwardingApplication

Slice 1

System Software

ForwardingApplication

Slice 2

Resource Allocation

Virtualization platform

System Software

ForwardingApplication

Slice N

Slices

Figure 2: Supporting network virtualization on SDF.

The key challenge in network virtualization is tosupport separate forwarding algorithms on the samehardware infrastructure. Note that it is simple to doso in software by having separate forwarding processes.When a packet arrives at a software switch, it is sentto the appropriate forwarding process based on thedemultiplexing criterion, and that process then makesthe forwarding decision for that packet.

Initial attempts at producing the same virtualizationbehavior in hardware were cumbersome and expensive,involving separate network processors for each slice. Thisis because traditional hardware-based packet forwardingembeds the forwarding logic directly in the hardware, andit is difficult to implement several different forwardinglogics simultaneously in the same ASIC.

However, because SDF uses software for all forwardingdecisions, and these decisions are merely imitated inhardware, supporting multiple forwarding algorithms isstraightforward (as long as each forwarding algorithmis compatible with the SDF forwarding model). This isbecause the ability to imitate a decision based on a packetheader is architecture-neutral. Also, the demultiplexingis based on either the input port or the packet header, andthus demultiplexing is codified in the resulting hardwaredecision entry. That is, SDF treats the demultiplexing aspart of the overall forwarding decision.6

To do this in a manner in which each forwardingapplication gets its fair share of system resources, thesystem software should segment the TCAM between eachof the applications. This requires either the applicationsbe trusted to provide a unique segment ID, or for thesystem software to be able to determine which applicationmade a particular forwarding decision. Of course this isonly a first order protection measure as errant applicationscould still consume disproportionate amounts of CPU orcontrol bandwidth. Protecting against such cases requiresisolation measures, which are orthogonal to SDF and wellunderstood by the research community (see e.g., [24]).

6This virtualization approach is similar to FlowVisor: these twopieces of work were done simultaneously. The FlowVisor work exploresthe applications of virtualization while the study here focuses on thesoftware API and merely identifies virtualization as a natural andstraightforward byproduct of software-defined networking.

5

Figure 2 depicts our virtualization prototype basedon Linux-VServer [19]. Each forwarding applicationruns in its own slice, on top of a local copy of systemsoftware. These system software instances in the slicesare then connected to the master system software instancein the root slice, which maintains a table of demul-tiplexing keys (e.g., VLAN IDs) to uniquely identifythe destination slice for incoming packets requiring aforwarding decision. The master system software isalso responsible for ensuring fair resource allocation. Inour prototype, we assume administrator configures theslices, their demultiplexing key and resource allocation apriori. We have used our prototype to run several differentforwarding algorithms simultaneously, including XORP,Chord, i3, and VL2 [13], and have achieved hardwarespeeds and good isolation.

Network virtualization doesnt impose hard require-ments on the platform virtualization; we only assume thata) the master system software runs somewhere, isolatedfrom the forwarding slices, within the physical host,and b) there is a communication channel between themaster system software and the system software runningvirtualized forwarding applications. Therefore, dependingon the platform virtualization approach, the mastersystem software may run as a Unix process in a specialmanagement virtual machine or even within the virtualmachine monitor itself. Similarly, the interconnectionbetween the master system software and slices can use thecommunication mechanism most suitable to the chosenplatform virtualization solution.

4.3 Implementing Forwarding Algorithms

In this section, we explore the suitability of our APIfor implementing various protocols. We describe theimplementations of two standard protocols (L2 learningand IP forwarding), as well as three research proposals(SEATTLE, i3, Chord). All implementations were builtin and tested on our prototype described in Section 7.

4.3.1 L2 Learning Switch

An L2 learning switch operates by maintaining a mappingbetween MAC addresses and the physical ports on whichthey can be reached. These mappings are learned bywatching the source addresses of packets as they traversethe forwarding software. Learned addresses expire aftersome period of inactivity, and each entry is updated whenthe switch receives a frame with the same source addressthrough a different port (e.g., during host movement).Incoming packets for which there is no learned MACaddress are flooded.

When a packet arrives and there is a mapping forits destination MAC address, an entry is created in theTCAM with the destination MAC address field markedas a dependency. We also mark the incoming port as

a dependency to ensure that we can see packets fromother hosts on the network. If the switch does not finda mapping, the cached forwarding decision is simplyto broadcast the packet. To revoke the entry whenthe mapping changes, we annotate the decision with aparticular state identifier associated to the mapping entry.

4.3.2 IPv4 and IPv6 Forwarding

IP forwarding proved to be a challenging forwardingalgorithm to support on our platform. The challengesstem from both the contents of the IP header as well asfrom the complicated lookup algorithm involved, LPM.

To forward IPv6 packets, the forwarding software hasto inspect the destination address and modify the TTL.Thus, the dependencies are the address and TTL, andthe rewritten packet header modifies the TTL. IPv4 ismore complicated, because in addition the forwardingsoftware must verify the IP header checksum and thenrecompute the checksum after the TTL has been modified and therefore, it needs to mark the entire packet headeras a dependency (because the entire packet header isinvolved in the header checksum recomputation). Theobvious remedy is to support checksumming and TTLdecrementing in hardware, as is currently done. This canbe done by providing some basic arithmetic primitives(decrementing, checksum, hash, etc.) that can be appliedto specified sets of bits (i.e., the hardware does not have tounderstand where the TTL field is). The SDF API couldbe extended with these additional operation-specific callsto convey enough information to the system software fromthe forwarding application to imitate these operations inhardware, assuming the hardware supports them.

LPM as a lookup algorithm is challenging for our APIdue to its implicit use of a conflict resolution scheme(longest match) for deciding the correct forwarding action.In our Python implementation of a simplified IPv4 router,the router pre-loads a FIB (from a RIB obtained fromRouteViews) and builds a trie. When it is invoked bycache misses, it looks up the trie to figure out the nexthop address. Since this next-hop result is valid until theFIB changes again for that entry, the software marks thedestination IP address and TTL, and associates the sentpacket with the corresponding IP-to-next hop mappingentry in the FIB data structure. When updating the FIB,the software creates a new trie and checks whether the newtrie yields the same next hop for each IP address in theold trie. If the result differs for an address, the softwarerevokes the corresponding old entry and constructs a newone, otherwise using the old entry in the new trie. Tomaintain the correctness of the forwarding, the routerimplementation calls revoke() for all FIB entries sharingthe same prefix as the newly added prefix.

In our approach, the software is oblivious to the

6

existence of the TCAM.7 Further, because the software isdeterministically making a forwarding decision over thepacket headers, and the system software is revoking thesedecisions as the system state changes, the TCAM shouldnever contain conflicting entries. The trade-off of usingnon-conflicting entries is that there is the potential forsubstantial hardware state explosion which is generallyameliorated through the use of priorities. For example,the default entry in LPM) conflicts with every other entry.In our approach, this would require every non-conflictingdestination address prefix which matches the default routeto be added as a separate hardware entry.

Although, SDF doesnt assume the use of priorities inhardware classification rules, the system software couldmaintain the header bit values and the masks of cacheddecisions in an order suitable for TCAMs to reduce thenumber of TCAM entries using the following algorithm:

1. First consider only decisions not having a dont carebit set in the beginning of the search mask. Assignthese decisions to groups based on their longest, firstconsecutive exact match; the group having the longestexact match of all will have the highest priority andthe group with least exact match bits will have thelowest priority.

2. Then assign all the remaining headers (which all havea dont care bit or more set in the beginning of theirmasks) to groups based on the number of consecutivedont care bits; the decisions in a group with onlyone dont care bit get a priority number just belowthe lowest priority assigned for group in the step one.The group with most dont care bits gets the lowestpriority of all groups processed in the steps 1 and 2.

3. Repeat steps 1 and 2, recursively, for each group tosort the entries within the group. Repeat until nocached decision has search mask bits left.

However, this does not solve the problem of determin-ing how to update TCAM entries without requiring therewriting of all entries, which could result in relativelylong periods of time (even seconds) when TCAMs cannotbe used in packet forwarding. We dont present afast update algorithm here, but view the algorithmsfor performing incremental updates developed for IPaddresses [27] and for general packet classification [28]as excellent starting points.

4.3.3 Floodless in SEATTLE

We also implemented SEATTLE [18] for our prototype.Like i3, SEATTLE demonstrates SDF can support hard-ware forwarding speeds of nonstandard protocols.

SEATTLE performs host discovery without resorting toflooding [18]. It does so by forming a link-layer DHT, that

7Modulo API calls aiding the system in tracking the header bits andinternal application state used in forwarding decisions.

stores information about host locations (i.e., associatedswitches), IP addresses, and MAC addresses. Briefly,when a switch S1 detects a directly connected host H1, itlearns H1s MAC address and IP address from traffic andpublishes the IPH1 (MACH1 , S1) and the MACH1 S1 mappings in the DHT. When a remote host H2 thensends a packet to H1, H2s local switch S2 resolves H1saddress and location through the DHT, and tunnels thepacket to S1 on behalf of H2. If the mapping changes(e.g., host migration or NIC replacement), S1 performsDHT operations to update the mapping.

We limit the description of our implementation tolocation resolution since address resolution is handled ina similar manner in SEATTLE. On receipt of an Ethernetframe from a host, our Python forwarding softwarefirst publishes the MACSRC Switch mapping if theaddress has not been published before. Next, if it findsan entry in its local mapping table, it sends the frameto the cached location (i.e., the associated switch) viathe shortest path. To do so, the switch encapsulates theframe by sending it out of a logical port which maintainsa tunnel to the destination switch. The software marks thedestination MAC of the received packet as a dependencythereby ensuring that the decision remains cached untilthe mapping changes.

If there is no mapping entry found, the forwardingsoftware initiates the DHT lookup process, and marks thedestination address in the packet header as a dependency;then the software sends the packet to a special null port;this caches the decision to drop frames destined to theaddress. This decision is also annotated with state identi-fiers corresponding to the (pending) mapping lookup sentto the DHT; once the reply arrives (or software timeoutswhile waiting for it), the cached decision will be revoked.On receipt of a DHT message, our implementationperforms the appropriate DHT operation (i.e., insert,delete, lookup). Note that since DHT messages arecontrol messages, they are not subject to caching.

Finally, when the software receives an outer frame, itinspects the encapsulating header to determine whetherit requires further forwarding. If the frame requiresforwarding, it marks the destination address of theouter header and forwards the packet further withouten/decapsulation. Since this decision is valid untilthe topology changes, the sent frame is annotated bythe relevant entries in the data structure holding theinformation about topology. If the frame does not requireforwarding to another switch (i.e., the frame is addressedto the switch itself), the switch marks the destinationaddress of the encapsulating header and sends the frameto a decapsulation port (coupled to the actual physicaloutput port connected destined to a host) removing theencapsulation header before sending the packet out onthe wire. Forwarding the packet to the decapsulation

7

port signals the system software to include hardware-implemented decapsulation in the cached decision.

4.3.4 i3 and Chord

We used our prototype to implement another non-standarddesign, i3 [29]. In i3 servers register listenerstriggersin the network and obtain trigger identifiers, whichthey publish to allow senders to find them. In ourimplementation, trigger identifiers are published in aChord DHT [30]. To send a packet to a server registeredwith i3, the sender looks up the server in the DHT bytraversing Chords ring until it finds the servers triggeridentifier, which it can use to forward the packet to thetrigger. The trigger then forwards the packet to the server.

Weve implemented both Chord and i3 forwarding onSDF. This required matching packet headers below thetransport layer at the Chord and i3 layers, demonstratingthe flexibility of the SDF approach. Both Chord and i3forwarding require only exact matches on their respectiveidentifier fields. Specifically, Chords cached decisionsinclude the 8 bit packet type field, the 160 bit Chordidentifier, and the 8 bit Chord TTL. For i3, the cacheddecisions include the 32 bit identifier stack size (thenumber of stacked identifiers) and a 256 bit i3 identifier.The required TCAM search mask length is therefore wellbeyond the standard maximum of 576 bits.

5 Performance in Reactive ModeOur design contains both a software component (theforwarding application and the system software, runningon a general CPU) and hardware component (based ona classification engine, in our case a TCAM). If thesystem is run in proactive mode, and the hardware canstore enough state to capture all possible forwardingdecisions, then the system will run at hardware forwardingspeeds. This is how we imagine the system will be runin performance-critical scenarios (such as the Internetbackbone). If the system is run in reactive mode, aswould be appropriate for most experimental uses whereperformance is not critical but a simple and convenientprogrammatic API is essential, then the resulting speeddepends on how many packets are handled by software,and the relative speed of the hardware and softwareforwarding actions. This is our focus in this section.

While the ability of commodity CPUs to performpacket forwarding has increased significantly over thelast few years, TCAMs remain significantly faster. Forexample, an 8-core PC supports forwarding capacitiesof 4.9 million packets/s [12], while modern TCAMscan achieve rates of 150200 million packets/s.8 Even

8The classification rate largely defines the maximum hardwareforwarding speed since applying packet modifications is easier thanclassification. Similarly, the high-speed switch fabric connecting portsis not a limiting factor.

special purpose general processors which include networkhardware on die (e.g. [9]) are an order of magnitudeslower than TCAMs.

In order to marry a slower software path with a muchfaster hardware path in a manner that takes full advantageof the system, the ratio of the number of packets processedby hardware versus software must be commensurate withthe differential between the speeds of the two layers.9

For purposes of analyzing these issues we assumesoftware forwarding performs two orders of magnitudeslower than the forwarding engine implemented in thehardware. This assumes that the switch employs one ormore high-speed multicore CPUs (as is true for manynewer switches [2]), and is a conservative estimate ofthe speed ratio given the TCAM and CPU figures citedabove. With this assumption, in order to fully realize theforwarding potential of our system, the cache hit ratesmust exceed 99%. The hit rates depend on the nature ofthe traffic, and the size of the cache.

We now analyze hit rates using standard L2 and L3forwarding algorithms on real network traces to seewhether a 99% hit rate is a reasonable expectationfor feasible cache sizes. We used the set of tracesdescribed in Table 1, and we assumed the naive hardwareimplementation of a managed TCAM in which eachmatching rule fills up an entry. The system uses LRUwhen replacing cache entries and we assume that theFIB does not change during the period of analysis.10

We warmed up the cache for half of the trace, and thenmeasured the hit-rate for the remainder of the trace.

L2 learning. Our MAC learning implementation uses15 second timeouts. For analysis, we used the first tracein Table 1, and the resulting cache hit rates are shownin the left-hand graph of Figure 3. The cache miss rateis less than 1% for cache sizes larger than 64, and isnearly 0.01% for a cache with 256 entries. Note thatthese TCAM state sizes are significantly smaller thantodays commercial Ethernet switches, which commonlyhold 1 million Ethernet addresses (6MB of TCAM). Thus,SDF in reactive mode would easily reach the requisite99% hit rate with very moderate state requirements inthis particular enterprise setting. We have looked at otherenterprise traces, and found similarly encouraging results.

IPv4 forwarding. We next explore the cache behaviorof standard IPv4 forwarding, assuming that the hardwaresupports TTL decrementing and checksum recomputation(so the forwarding decision depends only on the destina-tion address). We only had one trace with an associatedFIB, which is the ISP trace in Table 1. In order to obtain

9In addition, the bus between the hardware and software must not besaturated, and the hardware should be able to update the classificationrules as fast as the system software updates them, and these too dependon the cache hit rate.

10Routes to popular destinations are quite stable [25].

8

L2 trace L3 tracesName Enterprise OC12 [4] OC48 [5] OC192-A [6] OC192-B [7] OC192-C [7] ISP ISP2Source LBNL CAIDA AMPTH CAIDA Equinix Chicago Equinix Chicago Equinix San Jose ISP Europe ISP USADate Dec 2004 Jan 2007 Aug 2002 May 2008 Feb 2009 Mar 2009 Mar 2005 Jan 2010Duration 1 hour 2 hours 1 hour 1 hour 1 hour 1 hour NA 1 hour# packets 1,977,405 29,092,430 419,720,983 741,205,934 111,839,231 1,551,424,452 17,526,284 598,534,072Anonymized Yes Yes Yes Yes Yes Yes No No

Table 1: Traces used in the cache hit-rate evaluations. All the traces except the last contained full packet headers and timestamps. The last traceonly included a series of destination addresses without timestamps. However, because this last trace was not anonymized (but the last few bits wereelided), we could use the accompanying FIB to make accurate forwarding decisions; for the other traces we had to assume that forwarding decisionswere done at either the /16 or /24 granularity.

0.0001

0.001

0.01

0.1

1

16 32 64 128 256

Mis

s r

ate

(lo

g s

ca

le)

The number of cache entries (log scale)

L2 switch

0.001

0.010

0.100

1.000

10K 20K 40K 100K 200K 400K

Mis

s r

ate

(lo

g s

ca

le)

The number of cache entries (log scale)

LPM/16/24

Figure 3: Simulations of cache miss rates for (left) L2 learning with the enterprise trace and (right) LPM using the ISP trace.

conservative results, we used no warm-up period, so everycache entry experiences at least one miss. The cache hitrates of this analysis are shown in the right-hand graph ofFigure 3. There are three curves: one assuming that theforwarding decisions are based on the FIB, one assumingthat the forwarding decisions are based on /16 prefixes,and one assuming that the forwarding decisions are basedon /24 prefixes. For now we focus on the FIB-based curve,since that represents what would happen in practice. Here,the miss rate is less than 1% when the cache holds morethan about 64k entries, which is a small amount of statefor an ISP-class router. We do not know the links speedfor this trace, so to investigate hit-rates on very high-speedlinks we used several traces from CAIDA. For these traceswe did not have an associated FIB (because the traceswere anonymized); we therefore measured the hit-ratesassuming that the forwarding decisions were done oneither a /16 or /24 granularity. The data from the ISP traceindicates that, in that example, the /16 granularity is abetter estimate of the hit rate, and that the /24 granularityis very conservative (roughly two orders of magnitudehigher miss rate than the FIB results for large caches).

0.000010.000100.001000.010000.100001.00000

8K 16K 32K 64K 128K

Mis

s ra

te (l

og s

cale

)

The number of cache entries (log scale)

LPM/16/24

Figure 4: LPM using the ISP2 trace.

To support our assumption, we repeated the sameexperiment on the ISP2 trace. The trace was collected bya large domestic ISP and aggregates 10-30K downstreamcustomers. The trace contains IP addresses that are

not anonymized and has a matching FIB.11 Figure 4presents that the result is consistent with the previousmeasurement.

Figure 5 shows the results for the OC-48 trace and theOC-192C trace (the results for the other OC-192 tracesare qualitatively similar). When using /16 forwarding,even tiny cache sizes (2k entries for OC-48 and 1kentries for OC-192; the 1k point lies off the graph inFigure 5) are sufficient to achieve 1% miss rates. For/24 forwarding (which the ISP trace suggests is a verypessimistic estimate), cache sizes of 20k entries and 40kentries are required for 1% miss rates, which is againquite reasonable for ISP-class routers.

If these traces are representative, these results suggestonly moderate-sized caches can achieve sufficiently highcache hit rates in both enterprise and ISP networks.

Performance under stressful conditions. We now dis-cuss two situations where cachings performance may getstressed: when the software FIB changes and when theswitch is under attack. First, when the FIB changes, thiswill change a large number of forwarding entries. Intraditional flow caching, this requires flushing the cacheand enduring a high miss rate until the cache is warmed upagain. However, in our approach, when the routing statechanges, the application can push updated decisions to thehardware after revoking the invalid decisions, rather thanmerely revoking them. This does require a fast channelto update the hardware component, which may be thelimiting factor, but there is no need for a long cachewarmup period after a routing event.

Second, in a denial-of-service attack, attackers caninject spurious traffic in the hope of dislodging validcache entries, which would increase the miss rate. Oncethe miss rate goes over 1% (assuming the two order of

11The ISP generously has run our simulator on the trace and we havebeen reported only the cache hit rate result.

9

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

1K 2K 4K 10K 20K 40K 100K

Mis

s r

ate

(lo

g s

ca

le)

The number of cache entries (log scale)

/16/24

1e-06

1e-05

0.0001

0.001

0.01

0.1

10K 20K 40K 100K 200K 400K

Mis

s r

ate

(lo

g s

ca

le)

The number of cache entries (log scale)

/16/24

Figure 5: Cache miss rates with forwarding on the /16 and /24 granularities for the OC-48 (left) and the OC192-C trace (right).

0.001

0.01

0.1

10K 20K 40K 100K 200K 400K

Mis

s r

ate

(lo

g s

ca

le)

Cache sizes (log scale)

w/o attackw/ attack

0.0001

0.001

0.01

0.1

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Cac

he m

iss

rate

(log

)

The ratio of DoS traffic to total traffic

16384 entries65536 entries

Figure 6: Cache miss rates with (and without) random DoS traffic comprising 50% of the traffic for the ISP trace (left) and cache missrates withvarious random DoS traffic rate for the ISP2 trace(right).

magnitude difference between line speeds and the CPU),then some of the cache-miss packets will be dropped. Wenow investigate how effective such an attack might be,assuming the cache uses a LRU replacement policy.12

Using the ISP trace in Table 1 (again, with no warm-up period), we examine the effect of a DoS attack thatsends traffic with random destinations. The left graph inFigure 6 compares the miss rate with no attack to the missrate when the attack traffic comprises half of the totaltraffic. With an attack of this magnitude, overloadingthe link is likely to be more of a problem of cachemisses, but we wanted to look at an extreme scenario.Somewhat to our surprise, this massive attack only hasa very limited impact on the miss rate; this is becausethe long-tailed distribution of network destinations makesit unlikely that any of the frequently used cache entriesis ever evicted. In fact, for the larger cache sizes, themiss rate with the attack traffic is lower than without theattack traffic (this is because the random addresses tendto fall on the large prefixes, whereas the trace traffic moresystematically explores the address space). The rightgraph in Figure 6 also conforms to the assumption andindicates that random DoS attack could hardly impose animpact on the cache performance.

We dont claim to have solved all performance prob-lems associated with caching. However, if caching ismainly used in environments where performance is notcritical, then these issues discussed above (and otherperformance issues) may not be fatal. While the proactivemode is clearly safer, the reactive mode may still be

12The network can take more explicit measures against such attacks,such as rate-limiting the number of new flow entries from any port,which would limit the scope of such an attack. However here we justfocus on the attack traffics impact on the cache miss rate.

an attractive choice considering how small the staterequirements are to achieve low miss rates.

6 System DesignIn this section we first describe the API we designedwhile implementing our system. This API reflects lessonslearned while implementing a range of protocols, but it isonly one example of many possible software interfaces.We then discuss the system software design, followed byan overview of the hardware design.

6.1 API

Basic operations. For basic packet sending and receiv-ing, our API is modeled around a standard datagramsocket interface. To gain access to traffic, a forwardingapplication invokes listen() while providing a callback tobe signaled on each new received packet for which thereis no matching hardware entry and which is ready for theapplication to retrieve using receive(). On packet receipt,the forwarding application can then process the packetinternally (e.g., if the packet is control traffic), or it canmake a forwarding decision, modify the packet, and sendit out of one or more ports through a call to send().

In addition to basic sending and receiving, the APIprovides methods for the forwarding software to declarethe state dependencies of the forwarding decision. Thisincludes methods for annotating which header bits andwhat application state the decision relied on, and signalingwhen the application state has changed (to induce arevocation of any cached state). Modifications to packetsare mimicked by the system software by simply copyingthe modified outgoing header bits of each sent packet andforcing an overwrite of them for every match. Becausethese additional methods comprise the portion of the API

10

Cor

eFunction signature Descriptionlisten(callback) Register a callback to be signaled on every packet ready to be received.read() Receive a packet; return values are (in port, in pkt).send(out pkt, final) Send a packet. If final is true, the system knows the decision is complete.

InPk

t

mark(in pkt, bit pos, len) Mark packet header bits as used in a forwarding decision.mark ingress(in pkt) Mark the ingress port as used in a forwarding decision.unmark(in pkt, bit pos, len) Unmark matching bit markings, if any.clear(in pkt) Cancel all earlier bit markings.new in pkt(in port) Prepare a new synthetic inbound packet. Used in proactive decisions.

Out

Pkt

new out pkt(in pkt) Prepare a new outbound packet. If it is to be forwarded and not self-generated a reference to a received packet is required to copy the packet.

register(out pkt, state id) Register application state used in a decision resulting in this packet.add destination(out pkt, port) Add a destination port to send the packet to.clear(out pkt) Cancel all earlier state bindings, bit modifications, and port additions.

Con

trol

new state id() Construct a state identifier to store into application state data structures.removed(state id) Inform system about changed application state.get ports() Obtain a list of identifiers of ports the system has.add port(type), del port(id) Create/remove a virtual port. Type defines the type of tunneling/encryption

(possibly implemented in hardware) to use.set properties(id, key, val) Set port properties. For example, a port representing an IPsec tunnel requires

configuring encryption, as well as a tunnel end-point.get properties(id, key) Get a current port property value.get stats(port id) Get the latest port statistics. Format of statistics is port type specific.

Table 2: Forwarding API. The Core calls are to receive/send packets, InPkt/OutPkt are for packet operations. Control calls are for control plane.

which constrains the programmer beyond the traditionalsoftware model, we describe them in more detail below.For a complete summary of the API, see Table 2.

Annotating dependencies. It is difficult to build asystem software layer that can automatically withoutany help from the developer infer header and state depen-dencies without incurring exorbitant runtime overheads.Rather than aiming at full transparency, we chose insteadto put the onus on the programmer to explicitly markheaders and state used in forwarding decisions.

This is done through so-called state identifiers. Stateidentifiers are used to track application state that isused to make forwarding decisions. The canonicalexample of such state is the routing table. For eachuniquely identifiable component of system state usedfor forwarding decisions, such as a RIB entry, theforwarding application must request and associate a newstate identifier from the API.

This identifier is used to signal state changes to thesystem. For example, when removing an entry from aninternal data structure representing the forwarding table,the application is responsible for calling the removed()call on the associated state identifier. This will triggerthe system to revoke all hardware entries associated withthat particular entry. If instead of deleting the entry, theapplication modifies it, the application still must obtain anew identifier to replace the old one which is now invalid.

While making a forwarding decision for a receivedpacket, the application is required to register() any state

it uses in a decision; since the state identifiers areconveniently stored in the data structures, the applicationcan register() the state used into the outbound packet (ab-straction) while traversing the application data structuresand making the forwarding decision. Once the applicationthen send()s the packet, the system can record the stateidentifiers related to the decision to be cached. If theapplication subsequently removes the data structure entry(and informs the API about it), the system can identify thecached decisions to remove from hardware, and hence,maintain the correctness of the forwarding function.

Similarly to annotating state, in order to provide bit-level dependency information for the packet header, theapplication uses the mark() method to mark the header bitsit used in making a forwarding decision on a given packet.For headers in which the forwarding application markedthe bits, the system software will generate a TCAM entrywhose values matched those of of the packet. All otherheader bits are set to dont care.

The system software can infer packet header mod-ifications by observing the packets as they are sentby the application. This works so long as for eachincoming packet, the resulting outgoing packets aremodifications of the original packet. Unfortunately, thisis not always the case. In certain protocols, incomingpackets require intermediate packets to be sent beforeforwarding decisions are made. For example, if the MACaddress is not cached for a given next hop IP address, thepacket is queued and an ARP packet is sent instead. To aid

11

in such scenarios, our API considers the special of caseof no packet header bits nor state dependencies markedas a sign not to cache anything. Hence, the applicationcan respond to the ARP request by itself.

Finally, the forwarding application must send thepacket to one or more ports. As with standard forwardingsoftware, ports are represented via a programmaticabstraction. When a forwarding application sends apacket out of a port, the system software uses the informa-tion annotated by the application during the forwardingdecision and sets up a hardware entry correspondingly.

Prioritizing control protocols. Networking forward-ing software can often be divided into a control planeand a data plane. Under this model, the data plane isresponsible for performing packet lookups against one ormore data structures which are maintained by the controlplane. The control plane maintains the data structure(s)by sending control traffic between participating nodes.

In our approach both standard network traffic andcontrol traffic share the (limited) bandwidth between thehardware and the CPU. Further, on failure conditions (e.g.,link failure), many hardware entries may be invalidatedcausing a flood of packets to the CPU. This has thepotential of starving out the control traffic when it is mostneeded. To prevent such cases, the API provides a methodfor setting up entries in hardware that have prioritizedservice to the CPU. Thus, during failure conditionssuch as routing instabilities, the control traffic is givenprecedence allowing the system to converge quickly.

The entries for control traffic have the same express-ibility as other hardware entries (they can be defined overany header field or incoming port). However, instead ofsending to an egress port, the forwarding application cansend the control traffic packets to a logical control port,which represents ingress queues given weighted prioritycompared to the queue reserved for sending data planepackets from hardware to CPU.

The method for identifying control traffic is identicalto other forwarding operations. If the data plane makes adecision to send a packet to the logical control port, acorresponding entry to the hardware is put in place by thesystem software. Generally these entries are maintainedin hardware for the lifetime of the forwarding application.

Proactive decisions. In the reactive mode, the APIoperates in pull mode: whenever the system software re-ceives a packet from the hardware requiring a forwardingdecision, it asks the application for the decision. Whilethis allows the application to remain oblivious to thedecision replacement process, the simplicity comes witha cost: a new decision about a a revoked decision canonly be made when a new packet requiring the old entryarrives. This causes some additional forwarding delay forthis first packet, as it takes some time for the decision to

be made and the entry installed, but for some applicationsthis is of little concern. However, if the stream of packetsthat would use this revoked entry exceeds the capacity ofthe bus and/or the CPU, the system has to drop packetsuntil the new entry has been inserted into the TCAM.

For these demanding environments and applications,the API provides a mechanism to proactively push re-placement decision(s) at the same time the correspondingTCAM entries are removed. This is done by havingthe application create an appropriate synthetic packet (orpackets, as necessary) whenever an entry is revoked, soa new entry is installed at the same time the old one isrevoked. The system software does not forward thesesynthetic packets, it only uses them to construct andenqueue update(s) to the TCAM.

6.2 System Software

The system software is responsible for implementing andexposing the programmatic API used by the forwardingapplication, and directly managing the hardware. Whiledesigning the API, we tried to strike a balance betweenrelatively unconstrained program design, and the abilityto efficiently implement the system software. The APIitself imposes few restrictions on how the system softwarerealizes the required functionality for applications.

Language support. One can consider the proposedAPI as the low-level Sockets API for routers/switches;in a manner similar to the Sockets API, how the func-tionality is exposed for applications depends on theprogramming language and development environmentin question. Moreover, how the API relates to otherinterfaces provided for the forwarding applications alsodepends on the chosen language and environment. Forexample, for C/C++ applications, it is natural to align ourAPI next to the Sockets API and provide it as a part of thebasic system libraries.

Application portability. From an application perspec-tive, the system software implementation is irrelevantas long as the applications language is supported andthe different API implementations for the same languageremain binary compatible. In Section 7, we discussour implementation in which the system software isimplemented entirely in user-space providing both a Cand Python interface, both backed by an FPGA-basedhardware forwarding plane.

6.3 Hardware

At a component-level, the idealized hardware design is thestandard high-speed, high-fanout distributed forwardingarchitecture (as shown in Figure 7). A system-level CPUmanages the shared state between CPUs local to each linecard. The line card-local CPUs directly manage a packetclassification engine, which is used by the high-speedpacket processing logic to perform the lookups. The line

12

Figure 7: A distributed forwarding architecture.

cards also contain high-speed random access memory tostore the forwarding actions; the packet processing logicuses the random access memory to retrieve the requiredmodifications for packets, as well as to determine thedestination port(s) to forward the packet to. All line cardsare interconnected via a high-speed backplane.

A less scalable version can be implemented in a mannersimilar to commodity switches and routers available today.In such an instantiation, a control CPU controls a singlehardware forwarding engine (consisting of classificationengine and high-speed packet processing logic). This isthe approach we took with our prototype implementationwhich we describe in Section 7.

In these designs, the feasibility and implications ofperforming packet classifications using wide search masksare most essential. While our design does not dictatea particular packet classification component, for thefollowing discussion we consider the use of TCAMs sincethey are widely used in networking hardware.13

The largest TCAMs on the market today are 36Mbits insize and have an entry with of 288 or 576 bits. Assumingstandard protocols, a 576 bit entry is sufficient for lookupscovering the Ethernet header, the IP header (withoutoptions), as well as the TCP/UDP headers, while 288bits is sufficient for both the Ethernet and IP headers.Given these wide entries, the maximum number of rules aTCAM chip can store is either 128k or 64k, respectivelyfor 288 and 576 bit entries. In addition, TCAMs aredesigned to support stacked use in backs of 4 or 8,allowing for the (expensive) possibility of building aforwarding engine which contains 512K, 576-bit entries.

While TCAMs cant match header values against 576bit wide entries in a single clock cycle, they still manageextremely high throughput. For example, to match 576bits, a modern TCAM typically requires 4 clock cycles.Therefore, at 200MHz a chip which sells for roughly$300 can support 40Gbps of bandwidth. A TCAM with288 bit wide entries uses less for lookups, and therefore,can sustain 80Gbps. Given our design, if the additionalhardware logic does not introduce additional overhead, awide TCAM (576 bits) can support four 10Gbps ports, ora more standard configuration of 24, 1Gbps ports.

The main complexity of the system software is the

13Some of the information here are gathered from private discussionswith a TCAM vendor. Public information regarding commerciallyavailable TCAMs are available at [15, 23].

optimal management of entries in TCAMs and forwardingactions in SRAMs on line cards; i.e., keeping the cardsbusy forwarding without dedicating extensive periods oftime to updating the entries in TCAMs and in SRAMs. Aslong as the TCAM entries are non-conflicting, the systemsoftware can implement a simple cache replacementpolicy and remove the least used entries as necessary; thisis possible since as the order of non-conflicting TCAMentries doesnt matter. If the number of required TCAMentries is less than the number of TCAM entries availablein total, the task is even simpler. The maximum updatespeed is determined by bandwidth available on CPU bustowards line card(s) (and by the CPU itself).

A commonly raised concern with the use of TCAMs(besides price) is power consumption. It is true TCAMsconsume more power and are more expensive than eitherDRAM or SRAM, however a single TCAM can sustainconsiderable traffic. When compared to a commodityprocessor (which has been argued as a flexible alternativeto standard switching hardware [11, 12]), TCAM poweruse is far more modest. For instance, in practice a TCAMcan consume up to 30 watts, which is significantly lessthan the typical power consumption of a Pentium 4 at2.8GHz rated at 68 watts. Certainly when measured interms of power consumed per packet forwarded, TCAMsare far more attractive than commodity processors.

7 ImplementationWe implemented a prototype system to validate the design,as well as to better understand the implications of ourdesign decisions. We implemented the API with twolanguage bindings, C and Python. We choose to supportthe latter because we believe it emphasizes the flexibilityof the approach. In particular, it demonstrates that wecan use a (very) high-level language for implementingforwarding algorithms and still achieve wire forwardingspeeds. Indeed, even though Python is a scriptinglanguage and performs roughly 20 times slower than anequivalent C program, we were able to get hit rates whichallowed us to take full advantage of the hardware datapath.

The Python forwarding applications run on a Pythoninterpreter embedded in a C system software imple-mentation. The system software manages two linecards we implemented (for now, we dont support theirsimultaneous use): one running as a process in user-space and written in C (and running in the same hostas the system software, connected over TCP); and oneimplemented as a NetFPGA device. The user-space linecard uses the host network interfaces; its implementationis straightforward and not of independent interest. Wedescribe the NetFPGA implementation below.

NetFPGA prototype. We implemented a NetFPGAbased hardware line card with a simulated TCAMwith 32 x 512 bit wide matching entries and four 1Gbps

13

Figure 8: Packet processing pipeline of the NetFPGA prototype. Userdata path refers to custom logic on the FPGA.

Ethernet ports. Figure 8 depicts the packet processingpipeline implemented in our NetFPGA prototype. Thepipeline is organized in five stages. The first stage, theRx Queues, reads packets from the CPU or from theEthernet. The Input Arbiter stage selects which of the RxQueues to service, and pushes the packet to the OutputPort Lookup module. The Output Port Lookup modulelooks for matches and does any replacements requiredon the packet. The packet is then stored in the OutputQueues module until the output port is ready to receive it.The Tx Queues are responsible for delivering the packetfrom the Output Queues to the CPU or to the Ethernet.

As a packet arrives into the Output Port Lookup moduleit gets stored in an input FIFO and at the same time issent to a rule selector that creates 64-bit words to matchagainst. The module then reads the rule from the memoryand compares it to the output of the rule selector using themask read from the memory. Each packet match requiresnine 64-bit rule words to be matched. The first wordcontains the input port and the other eight are packet datawords. To simplify the matching, the memory is organizedin 512 byte blocks. Entries are written vertically intomemory so that the first memory block contains the firstrule word (64 bits) of both match data and match mask, ofall 32 entries. Similarly, the subsequent memory blockscontain the remaining eight rule words. Therefore, asthe words read from the packet (header) arrive from therule selector, they can be matched a block-by-block to allrule words. However, we note this simple implementationdoesnt scale to larger entry numbers well; as the numberof entries grows, the number of required clock cycles tomatch a packet increases accordingly.

The memory is also used to store the lookup results:the replacement data, the replacement mask, and thehit packet and byte counters. The entries are writtenhorizontally such that each memory word contains allthe data for a single match. When a match is found, theOutput Port Lookup module reads the packet out of theinput FIFO, and uses the result from the match to replacethe first 64 bytes of the packet headers as required. If amatch is not found, the module reads the packet from theinput FIFO and sends it to the CPU over PCI.

Microbenchmarks. The table below contains latencyand throughput rates for cache entry insertion and deletion

for our NetFPGA prototype. The entry insertion is farslower because it requires writing the full lookup entry,the header modifications, and it requires overwriting thepacket and byte counts. Deletion on the other hand simplyoverwrites the enable bit for the entry.

Overhead Latency ThroughputEntry insert 300 bytes 55.5 ms 18,018/sEntry delete 4 bytes 740 ns 1,351,351/s

We also performed throughput testing for the data pathusing a hardware packet generator which can perform linespeed testing. Tests were performed with full line rateon all four ports. Results include the Ethernet CRC, theinter-packet gap, and the packet preamble.

Packet size (bytes) 64 128 512 1500Throughput 3.8G 3.8G 3.8G 3.8G

8 Concluding CommentsThe SDF approach we have presented here containsnothing new; it merely glues together two well-knowncomponents (software decisions and hardware classi-fication) into a complete forwarding solution. Wehave designed a high-level API that allows forwardingsolutions to be developed in a high-level languageindependent of the low-level hardware implementation.The approach can be implemented on top of OpenFlowor other similar hardware interfaces, or could be porteddirectly to commercial switch SDKs.

Our preliminary work suggests that this combina-tion of known components provides a flexible packetforwarding platform that, because it is based on stan-dard hardware (e.g., TCAMs), should offer competitivecost/performance. We think that SDF may prove quiteuseful in building experimental networks. Its relevanceto production networks is more questionable, as the needfor flexibility in that context remains an open question.

References[1] T. Anderson, L. Peterson, S. Shenker, and J. Turner. Overcoming

the Internet Impasse through Virtualization. Computer, 38(4):3441, 2005.

[2] Arista Networks. http://www.aristanetworks.com.[3] A. Bavier, N. Feamster, M. Huang, L. Peterson, and J. Rexford. In

VINI Veritas: Realistic and Controlled Network Experimentation.In SIGCOMM, 2006.

[4] http://www.caida.org/data/passive/passive_2007_dataset.xml.

[5] http://www.caida.org/data/passive/passive_oc48_dataset.xml.

[6] http://www.caida.org/data/passive/passive_2008_dataset.xml.

[7] http:// www.caida.org/data/passive/passive_2009_dataset.xml.

[8] M. Casado, T. Koponen, D. Moon, and S. Shenker. RethinkingPacket Forwarding Hardware. In HotNets, Oct. 2008.

[9] Cavium Networks. http://www.caviumnetworks.com/.[10] ControlPoint Developers Alliance. http://www.

fulcrummicro.com/cda/.

14

http://www.aristanetworks.comhttp://www.caida.org/data/passive/passive_2007_dataset.xmlhttp://www.caida.org/data/passive/passive_2007_dataset.xmlhttp://www.caida.org/data/passive/passive_oc48_dataset.xmlhttp://www.caida.org/data/passive/passive_oc48_dataset.xmlhttp://www.caida.org/data/passive/passive_2008_dataset.xmlhttp://www.caida.org/data/passive/passive_2008_dataset.xmlhttp://www.caida.org/data/passive/passive_2009_dataset.xmlwww.caida.org/data/passive/passive_2009_dataset.xmlhttp://www.caviumnetworks.com/http://www.fulcrummicro.com/cda/http://www.fulcrummicro.com/cda/

[11] M. Dobrescu et al. RouteBricks: Exploiting Parallelism To ScaleSoftware Routers. In SOSP, 2009.

[12] N. Egi, A. Greenhalgh, M. Handley, M. Hoerdt, F. Huici,and L. Mathy. Towards High Performance Virtual Routers onCommodity Hardware. In CoNEXT, 2008.

[13] A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim,P. Lahiri, D. A. Maltz, P. Patel, and S. Sengupta. VL2: A Scalableand Flexible Data Center Network. In SIGCOMM, 2009.

[14] N. Gude, T. Koponen, J. Pettit, B. Pfaff, M. Casado, N. McKeown,and S. Shenker. NOX: Towards an Operating System for Networks.SIGCOMM Comput. Commun. Rev., 38(3):105110, 2008.

[15] Integrated Device Technology (IDT). http://idt.com/.[16] J. Kelly, W. Araujo, and K. Banerjee. Rapid Service Creation

using the JUNOS SDK. In PRESTO, 2009.[17] C. Kim, M. Caesar, A. Gerber, and J. Rexford. Revisiting Route

Caching: The World Should Be Flat. In PAM, 2009.[18] C. Kim, M. Caesar, and J. Rexford. Floodless in Seattle:

A Scalable Ethernet Architecture for Large Enterprises. InSIGCOMM, 2008.

[19] Linux-VServer. http://www.linux-vserver.org/.[20] G. Lu, Y. Shi, C. Guo, and Y. Zhang. CAFE: A Configurable

pAcket Forwarding Engine for Data Center Networks. In PRESTO,2009.

[21] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar,L. Peterson, J. Rexford, S. Shenker, and J. Turner. OpenFlow:Enabling Innovation in Campus Networks. ACM SIGCOMM CCR,38(2):6974, 2008.

[22] J. C. Mogul et al. Orphal: API Design Challenges for Open RouterPlatforms on Proprietary Hardware. In HotNets, 2008.

[23] Netlogic Microsystems. http://netlogicmicro.com/.[24] L. Peterson, A. Bavier, M. E. Fiuczynski, and S. Muir. Experiences

Building PlanetLab. In OSDI, 2006.[25] J. Rexford, J. Wang, Z. Xiao, and Y. Zhang. BGP Routing Stability

of Popular Destinations. In IMW, 2002.[26] G. Schaffrath et al. Network Virtualization Architecture: Proposal

and Initial Prototype. In VISA, 2009.[27] D. Shah and P. Gupta. Fast Incremental Updates on Ternary-CAMs

for Routing Lookups and Packet Classification. In HotI, 2000.[28] H. Song and J. S. Turner. Fast Filter Updates in TCAMs for Packet

Classification. In GLOBECOM, 2006.[29] I. Stoica, D. Adkins, S. Zhuang, S. Shenker, and S. Surana. Internet

Indirection Infrastructure. In SIGCOMM, 2002.[30] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and

H. Balakrishnan. Chord: A Scalable Peer-to-peer Lookup Servicefor Internet Applications. In SIGCOMM, 2001.

15

http://idt.com/http://www.linux-vserver.org/http://netlogicmicro.com/

IntroductionGoals, Limitations, and ApplicabilityOverviewSystem Functionality and ComponentsForwarding StepsSound Familiar?

Example Use CasesAccelerating Existing Software RoutersVirtualizationImplementing Forwarding AlgorithmsL2 Learning SwitchIPv4 and IPv6 ForwardingFloodless in SEATTLEi3 and Chord

Performance in Reactive ModeSystem DesignAPISystem SoftwareHardware

ImplementationConcluding Comments