Designing Extensible IP Router Software · OSPF IS−IS router manager BGP4+ RIB Management Functions Unicast Routing Multicast Routing RIB = routing information base FEA = forwarding

Designing Extensible IP Router Software

Mark Handley†∗ Eddie Kohler‡∗ Atanu Ghosh∗ Orion Hodson∗ Pavlin Radoslavov∗∗International Computer Science Institute †University College, London ‡UCLA

{mjh, kohler, atanu, hodson, pavlin}@xorp.org

ABSTRACT

Many problems with today’s Internet routing infrastruc-ture—slow BGP convergence times exacerbated by timer-based route scanners, the difficulty of evaluating new pro-tocols—are not architectural or protocol problems, butsoftware problems. Router software designers have tack-led scaling challenges above all, treating extensibility andlatency concerns as secondary. At this point in the In-ternet’s evolution, however, further scaling and securityissues require tackling latency and extensibility head-on.

We present the design and implementation of XORP,an IP routing software stack with strong emphases on la-tency, scaling, and extensibility. XORP is event-driven,and aims to respond to routing changes with minimaldelay—an increasingly crucial requirement, given risingexpectations for Internet reliability and convergence time.The XORP design consists of a composable frameworkof routing processes, each in turn composed of modularprocessing stages through which routes flow. Extensibil-ity and latency concerns have influenced XORP through-out, from IPC mechanisms to process arrangements tointra-process software structure, and leading to novel de-signs. In this paper we discuss XORP’s design and im-plementation, and evaluate the resulting software againstour performance and extensibility goals.

1 INTRODUCTION

The Internet has been fabulously successful; previouslyunimagined applications frequently arise, and changingusage patterns have been accommodated with relativeease. But underneath this veneer, the low-level proto-cols that support the Internet have largely ossified, andstresses are beginning to show. Examples include secu-rity and convergence problems with BGP routing [18],deployment problems with multicast [10], QoS, and IPv6,and the lack of effective defense mechanisms against de-nial-of-service attacks. The blame for this ossificationhas been placed at various technical and non-technicalpoints in the Internet architecture, from limits of lay-ered protocol design [4] to the natural conservatism of

Orion Hodson is currently at Microsoft Research.This material is based upon work supported by the National ScienceFoundation under Grant No. 0129541. Any opinions, findings, and con-clusions or recommendations expressed in the material are those of theauthor(s) and do not necessarily reflect the views of the National Sci-ence Foundation.

commercial interests [9]; suggested solutions have in-cluded widespread overlay networks [23, 24] and activenetworking [6, 30]. But less attention has been paid toa simple, yet fundamental, underlying cause: the lack ofextensible, robust, high-performance router software.

The router software market is closed: each vendor’srouters will run only that vendor’s software. This makesit almost impossible for researchers to experiment in realnetworks, or to develop proof-of-concept code that mightconvince network operators that there are alternatives tocurrent practice. A lack of open router APIs additionallyexcludes startup companies as a channel for change.

The solution seems simple in principle: router soft-ware should have open APIs. (This somewhat resemblesactive networks, but we believe that a more conservativeapproach is more likely to see real-world deployment.)Unfortunately, extensibility can conflict with the otherfundamental goals of performance and robustness, andwith the sheer complexity presented by routing protocolslike BGP. Relatively few software systems have robust-ness and security goals as stringent as those of routers,where localized instability or misconfiguration can rip-ple throughout the Internet [3]. Routers must also jugglehundreds of thousands of routes, which can be installedand withdrawn en masse as links go up and down. Thislimits the time and space available for extensions to run.Unsurprisingly, then, existing router software was notwritten with third-party extension in mind, so it doesn’tgenerally include the right hooks, extension mechanismsand security boundaries.

We therefore saw the need for a new suite of routersoftware: an integrated open-source software router plat-form running on commodity hardware, and viable both inresearch and production. The software architecture wouldhave extensibility as a primary goal, permitting experi-mental protocol deployment with minimal risk to exist-ing services. Internet researchers needing access to routersoftware would share a common platform for experimen-tation, and get an obvious path to deployment for free.The loop between research and realistic real-world ex-perimentation would eventually close, allowing innova-tion to take place much more freely. We have made sig-nificant progress towards building this system, which wecall XORP, the eXtensible Open Router Platform [13].

This paper focuses on the XORP control plane: rout-ing protocols, the Routing Information Base (RIB), net-

NSDI ’05: 2nd Symposium on Networked Systems Design & ImplementationUSENIX Association 189

work management software, and related user-level pro-grams that make up the vast majority of software on arouter today. This contrasts with the forwarding plane,which processes every packet passing through the router.Prior work on component-based forwarding planes hassimultaneously achieved extensibility and good perfor-mance [16, 26], but these designs, which are based on theflow of packets, don’t apply directly to complex protocolprocessing and route wrangling. XORP’s contributions,then, consist of the strategies we used to break the controlplane, and individual routing protocols, into componentsthat facilitate both extension and good performance.

For example, we treat both BGP and the RIB as net-works of routing stages, through which routes flow. Par-ticular stages within those networks can combine routesfrom different sources using various policies, or notifyother processes when routes change. Router functionalityis separated into many Unix processes for robustness. Aflexible IPC mechanism lets modules communicate witheach other independent of whether those modules arepart of the same process, or even on the same machine;this allows untrusted processes to be run entirely sand-boxed, or even on different machines from the forward-ing engine. XORP processes are event-driven, avoidingthe widely-varying delays characteristic of timer-baseddesigns (such as those deployed in most Cisco routers).Although XORP is still young, these design choices arestable enough to have proven their worth, and to demon-strate that extensible, scalable, and robust router softwareis an achievable goal.

The rest of this paper is organized as follows. Af-ter discussing related work (Section 2), we describe ageneric router control plane (Section 3) and an overviewof XORP (Section 4). Sections 5 and 6 describe par-ticularly relevant parts of the XORP design: the rout-ing stages used to compose the RIB and routing proto-cols like BGP and our novel inter-process communica-tion mechanism. The remaining sections discuss our se-curity framework; present a preliminary evaluation, whichshows that XORP’s extensible design does not impact itsperformance on macro-benchmarks; and conclude.

2 RELATED WORK

Previous work discussed XORP’s requirements and high-level design strategy [13]; this paper presents specificsolutions we developed to achieve those requirements.We were inspired by prior work on extensible forward-ing planes, and support Click [16], one such forwardingplane, already.

Individual open-source routing protocols have longbeen available, including routed [29] for RIP, OSPFd [20]for OSPF, and pimd [14] for PIM-SM multicast routing.However, interactions between protocols can be prob-lematic unless carefully managed. GateD [21] is perhaps

the best known integrated routing suite, although it beganas an implementation of a single routing protocol. GateDis a single process within which all routing protocolsrun. Such monolithic designs are fundamentally at oddswith the concept of differentiated trust, whereby moreexperimental code can be run alongside existing serviceswithout destabilizing the whole router. MRTD [28] andBIRD [2], two other open-source IP router stacks, alsouse a single-process architecture. In the commercial world,Cisco IOS [7] is also a monolithic architecture; experi-ence has shown that this significantly inhibits networkoperators from experimenting with Cisco’s new protocolimplementations.

Systems that use a multi-process architecture, per-mitting greater robustness, include Juniper’s JunOS [15]and Cisco’s most recent operating system IOS XR [8].Unfortunately, these vendors do not make their APIs ac-cessible to third-party developers, so we have no idea iftheir internal structure is well suited to extensibility. Theopen-source Zebra [31] and Quagga [25] stacks use mul-tiple processes as well, but their shared inter-process APIis limited in capability and may deter innovation.

Another important distinguishing factor between im-plementations is whether a router is event-driven or usesa periodic route scanner to resolve dependencies betweenroutes. The scanner-based approach is simpler, but has arather high latency before a route change actually takeseffect. Cisco IOS and Zebra both use route scanners, with(as we demonstrate) a significant latency cost; MRTDand BIRD are event-driven, but this is easier given a sin-gle monolithic process. In XORP, the decision that ev-erything is event-driven is fundamental and has been re-flected in the design and implementation of all protocols,and of the IPC mechanism.

3 CONTROL PLANE FUNCTIONAL OVERVIEW

The vast majority of the software on a router is control-plane software: routing protocols, the Routing Informa-tion Base (RIB), firewall management, command-line in-terface, and network management—and, on modern rout-ers, much else, including address management and “mid-dlebox” functionality. Figure 1 shows a basic functionalbreakdown of the most common software on a router.The diagram’s relationships correspond to those in XORPand, with small changes, those in any router. The rest ofthis section explores those relationships further.

The unicast routing protocols (BGP, RIP, OSPF, andIS-IS) are clearly functionally separate, and most routersonly run a subset of these. However, as we will see later,the coupling between routing protocols is fairly complex.The arrows on the diagram illustrate the major flows ofrouting information, but other flows also exist.

The Routing Information Base (RIB) serves as theplumbing between routing protocols. Protocols such as

NSDI ’05: 2nd Symposium on Networked Systems Design & Implementation USENIX Association190

PIM−SM

RIP

FEA

Forwarding Plane

IGMP/MLD

CLI SNMP...

OSPF

IS−IS

routermanager

BGP4+

RIB

Management Functions

Unicast Routing

Multicast Routing

RIB = routing information baseFEA = forwarding engine abstraction

FIGURE 1—Typical router control plane functions

RIP and OSPF receive routing information from remoterouters, process it to discover feasible routes, and sendthese routes to the RIB. As multiple protocols can supplydifferent routes to the same destination subnet, the RIBmust arbitrate between alternatives.

BGP has a more complex relationship with the RIB.Incoming IBGP routes normally indicate a nexthop routerfor a destination, rather than an immediate neighbor. Ifthere are multiple IBGP routes to the same subnet, BGPwill typically need to know the routing metrics for eachchoice so as to decide which route has the nearest exit(so-called “hot potato” routing). Thus, BGP must exam-ine the routing information supplied to the RIB by otherrouting protocols to make its own routing decisions.

A key instrument of routing policy is the process ofroute redistribution, where routes from one routing pro-tocol that match certain policy filters are redistributedinto another routing protocol for advertisement to otherrouters. The RIB, as the one part of the system that seeseveryone’s routes, is central to this process.

The RIB is thus crucial to the correct functioning ofa router, and should be extended only with care. Routingprotocols may come and go, but the RIB should ideallybe general enough to cope with them all; or failing that, itshould support small, targeted extensions that are easilychecked for correctness.

The Forwarding Engine Abstraction (FEA) providesa stable API for communicating with a forwarding en-gine or engines. In principle, its role is syntactic, andmany single-platform routers leave it out, communicat-ing with the forwarding plane directly.

PIM-SM (Protocol Independent Multicast—SparseMode [12]) and IGMP provide multicast routing func-tionality, with PIM performing the actual routing andIGMP informing PIM of the existence of local receivers.

PIM contributes routes not to the RIB, but directly viathe FEA to the forwarding engine. Thus, the FEA’s inter-face is important for more than just the RIB. However,PIM does use the RIB’s routing information to decide onthe reverse path back to a multicast source.

The “Router Manager” holds the router configura-tion and starts, configures, and stops protocols and otherrouter functionality. It hides the router’s internal structurefrom the user, providing operators with unified manage-ment interfaces for examination and reconfiguration.

Our goal is a router control plane that provides all thisfunctionality, including all the most widely used routingprotocols, in a way that encourages extensibility. At thispoint, we do not automatically protect operators frommalicious extensions or experimental code. Instead, oursoftware architecture aims to minimize extension foot-print, making it feasible for operators to check the codethemselves. This requires a fundamental design shift fromthe monolithic, closely-coupled designs currently preva-lent. In Section 7 we will discuss in more detail our cur-rent and future plans for XORP’s security framework.

4 XORP OVERVIEW

The XORP control plane implements this functionalitydiagram as a set of communicating processes. Each rout-ing protocol and management function is implementedby a separate process, as are the RIB and the FEA. Pro-cesses communicate with one another using an extensi-ble IPC mechanism called XORP Resource Locators, orXRLs. This blurs the distinction between intra- and inter-process calls, and will even support transparent commu-nication with non-XORP processes. The one importantprocess not represented on the diagram is the Finder,which acts as a broker for IPC requests; see Section 6.2.(XORP 1.0 supports BGP and RIP; support for OSPFand IS-IS is under development.)

This multi-process design limits the coupling betweencomponents; misbehaving code, such as an experimen-tal routing protocol, cannot directly corrupt the mem-ory of another process. Performance is a potential down-side, due to frequent IPCs; to address it, we implementedvarious ways to safely cache IPC results such as routes(Section 5.2.1). The multi-process approach also servesto decouple development for different functions, and en-courages the development of stable APIs. Protocols suchBGP and RIP are not special in the XORP design—theyuse APIs equally available to all. Thus, we have confi-dence that those APIs would prove sufficient, or nearlyso, for most experimental routing protocols developed inthe future.1

We chose to implement XORP primarily in C++, be-cause of its object orientation and good performance. Re-alistic alternatives would have been C and Java. When westarted implementing XORP, the choice was not com-


pletely clear cut, but we’ve become increasingly satis-fied; for example, extensive use of C++ templates allowscommon source code to be used for both IPv4 and IPv6,with the compiler generating efficient implementationsfor both.

Each XORP process adopts a single-threaded event-driven programming model. An application such as a rout-ing protocol, where events affecting common data comefrom many sources simultaneously, would likely havehigh locking overhead; but, more importantly, our ex-perience is that it is very hard for new programmers tounderstand a multi-threaded design to the point of be-ing able to extend it safely. Of course, threaded programscould integrate with XORP via IPC.

The core of XORP’s event-driven programming modelis a traditional select-based event loop based on theSFS toolkit [19]. Events are generated by timers and filedescriptors; callbacks are dispatched whenever an eventoccurs. Callbacks are type-safe C++ functors, and allowfor the currying of additional arguments at creation time.

When an event occurs, we attempt to process thatevent to completion, including figuring out all inter-processdependencies. For example, a RIP route may be usedto resolve the nexthop in a BGP route; so a RIP routechange must immediately notify BGP, which must thenfigure out all the BGP routes that might change as a re-sult. Calculating these dependencies quickly and efficientlyis difficult, introducing strong pressure toward a periodicroute scanner design. Unfortunately, periodic scanningintroduces variable latency and can lead to increased loadbursts, which can affect forwarding performance. Sincelow-delay route convergence is becoming critical to ISPs,we believe that future routing implementations must beevent-driven.

Even in an event-driven router, some tasks cannotbe processed to completion in one step. For example,a router with a full BGP table may receive well over100,000 routes from a single peer. If that peering goesdown, all these routes need to be withdrawn from allother peers. This can’t happen instantaneously, but a flap-ping peer should not prevent or unduly delay the process-ing of BGP updates from other peers. Therefore, XORPsupports background tasks, implemented using our timerhandler, which run only when no events are being pro-cessed. These background tasks are essentially coopera-tive threads: they divide processing up into small slices,and voluntarily return execution to the process’s mainevent loop from time to time until they complete.

We intend for XORP to run on almost any modernoperating system. We initially provide support, includingFEA support, for FreeBSD and Linux, and for FreeBSDand Linux running Click as a forwarding path. Windowssupport is under development.

incoming routesfrom neighboringrouters

outgoing routesto neighboringrouters

state machinefor neighboringrouter

table of routes

bestroutes

to RIB



FIGURE 2—Abstract routing protocol

5 ROUTING TABLE STAGES

From the general process structure of the XORP controlplane, we now turn to modularity and extensibility withinsingle processes, and particularly to the ways we dividerouting table processing into stages in BGP and the RIB.This modularization makes route dataflow transparent,simplifies the implementation of individual stages, clari-fies overall organization and protocol interdependencies,and facilitates extension.

At a very high level, the abstract model in Figure 2can represent routing protocols such as RIP or BGP. (Link-state protocols differ slightly since they distribute all rout-ing information to their neighbors, rather than just thebest routes.) Note that packet formats and state machinesare largely separate from route processing, and that allthe real magic—route selection, policy filtering, and soforth—happens within the table of routes. Thus, from asoftware structuring point of view, the interesting part isthe table of routes.

Unfortunately, BGP and other modern routing proto-cols are big and complicated, with many extensions andfeatures, and it is very hard to understand all the interac-tions, timing relationships, locking, and interdependen-cies that they impose on the route table. For instance,as we mentioned, BGP relies on information from intra-domain routing protocols (IGPs) to decide whether thenexthop in a BGP route is actually reachable and whatthe metric is to that nexthop router. Despite these depen-dencies, BGP must scale well to large numbers of routesand large numbers of peers. Thus, typical router imple-mentations put all routes in the same memory space asBGP, so that BGP can directly see all the informationrelevant to it. BGP then periodically walks this jumborouting table to figure out which routes win, based onIGP routing information. This structure is illustrated inFigure 3. While we don’t know how Cisco implementsBGP, we can infer from clues from Cisco’s command lineinterface and manuals that it probably works somethinglike this.

Unfortunately, this structure makes it very hard toseparate functionality in such a way that future program-mers can see how the pieces interact or where it is safe tomake changes. Without good structure we believe that itwill be impossible for future programmers to extend our


incoming BGProutes from neighboringrouters

BGP routes

Other routes

BGP state machine for neighbor

RIP state machinefor neighbor

BGP state machine for neighbor

RIP state machinefor neighbor

incoming RIProutes from neighboringrouters

RIPDecisionProcess

bestBGProutes

bestRIProutes

outgoing BGP routes to neighboring BGProuters

outgoing RIP routes to neighboring RIProuters

BGPDecisionProcess

Best routesto forwarding engine

FIGURE 3—Closely-coupled routing architecture

software without compromising its stability.Our challenge is to implement BGP and the RIB in

a more decoupled manner that clarifies the interactionsbetween modules.

5.1 BGP Stages

The mechanism we chose is the clear one of data flow.Rather than a single, shared, passive table that stores in-formation and annotations, we implement routing tablesas dynamic processes through which routes flow. Thereis no single routing table object, but rather a network ofpluggable routing stages, each implementing the sameinterface. Together, the network stages combine to im-plement a routing table abstraction. Although unusual—to our knowledge, XORP is the only router using thisdesign—stages turn out to be a natural model for routingtables. They clarify protocol interactions, simplify themovement of large numbers of routes, allow extension,ease unit testing, and localize complex data structure ma-nipulations to a few objects (namely trees and iterators;see Section 5.3). The cost is a small performance penaltyand slightly greater memory usage, due to some dupli-cation between stages. To quantify this, a XORP routerholding a full backbone routing table of about 150,000routes requires about 120 MB for BGP and 60 MB forthe RIB, which is simply not a problem on any recenthardware. The rest of this section develops this stage de-sign much as we developed it in practice.

To a first approximation, BGP can be modeled as thepipeline architecture, shown in Figure 4. Routes come infrom a specific BGP peer and progress through an in-coming filter bank into the decision process. The bestroutes then proceed down additional pipelines, one foreach peering, through an outgoing filter bank and thenon to the relevant peer router. Each stage in the pipelinereceives routes from upstream and passes them down-stream, sometimes modifying or filtering them along theway. Thus, stages have essentially the same API, andare indifferent to their surroundings: new stages can be

DecisionProcess

IGP routinginformationfrom RIB

Best routesto RIB

BestRoutesto Peers

PeerIn

FilterBank

FilterBank

PeerOut

PeerIn

FilterBank

PeerIn

FilterBank

FilterBank

PeerOut

FilterBank

PeerOut

RoutesfromBGPPeers

RoutestoBGPPeers

FIGURE 4—Staged BGP architecture

added to the pipeline without disturbing their neighbors,and their interactions with the rest of BGP are constrainedby the stage API.

The next issue to resolve is where the routes are ac-tually stored. When a new route to a destination arrives,BGP must compare it against all alternative routes to thatdestination (not just the previous winner), which dictatesthat all alternative routes need to be stored. The naturalplace might seem to be the Decision Process stage; butthis would complicate the implementation of filter banks:Filters can be changed by the user, after which we need tore-run the filters and re-evaluate which route won. Thus,we only store the original versions of routes, in the PeerIn stages. This in turn means that the Decision Processmust be able to look up alternative routes via calls up-stream through the pipeline.

The basic interface for a stage is therefore:

• add route: A preceding stage is sending a new routeto this stage. Typically the route will be dropped,modified, or passed downstream to the next stage un-changed.

• delete route: A preceding stage is sending a deletemessage for an old route to this stage. The deletionshould be dropped, modified, or passed downstreamto the next stage unchanged.

• lookup route: A later stage is asking this stage to lookup a route for a destination subnet. If the stage cannotanswer the request itself, it should pass the requestupstream to the preceding stage.

These messages can pass up and down the pipeline, withthe constraint that messages must be consistent. Thereare two consistency rules: (1) Any delete route messagemust correspond to a previous add route message; and(2) the result of a lookup route should be consistent withprevious add route and delete route messages sent down-stream. These rules lessen the stage implementation bur-den. A stage can assume that upstream stages are consis-tent, and need only preserve consistency for downstreamstages.

For extra protection, a BGP pipeline could includestages that enforced consistency around possibly-erro-


neous experimental extensions, but so far we have notneeded to do this. Instead, we have developed an extraconsistency checking stage for debugging purposes. Thiscache stage, just after the outgoing filter bank in the out-put pipeline to each peer, has helped us discover manysubtle bugs that would otherwise have gone undetected.While not intended for normal production use, this stagecould aid with debugging if a consistency error is sus-pected.

5.1.1 Decomposing the Decision Process

The Decision Process in this pipeline is rather complex:in addition to deciding which route wins, it must getnexthop resolvability and metric information from theRIB, and fan out routing information to the output peerpipeline branches and to the RIB. This coupling of func-tionality is undesirable both because it complicates thestage, and because there are no obvious extension pointswithin such a macro-stage. XORP thus further decom-poses the Decision Process into Nexthop Resolvers, asimple Decision Process, and a Fanout Queue, as shownin Figure 5.

DecisionProcess

Best routesto RIB

BestRoutesto Peers

NexthopResolver

NexthopResolver

NexthopResolver

IGP routinginformationfrom RIB

NexthopLookup

FanoutQueue

PeerIn

FilterBank

PeerIn

FilterBank

PeerIn

FilterBank

FilterBank

PeerOut

FilterBank

PeerOut

FilterBank

PeerOut

FIGURE 5—Revised staged BGP architecture

The Fanout Queue, which duplicates routes for eachpeer and for the RIB, is in practice complicated by theneed to send routes to slow peers. Routes can be receivedfrom one peer faster than we can transit them via BGPto other peers. If we queued updates in the n Peer Outstages, we could potentially require a large amount ofmemory for all n queues. Since the outgoing filter banksmodify routes in different ways for different peers, thebest place to queue changes is in the fanout stage, afterthe routes have been chosen but before they have beenspecialized. The Fanout Queue module then maintains asingle route change queue, with n readers (one for eachpeer) referencing it.

The Nexthop Resolver stages talk asynchronously tothe RIB to discover metrics to the nexthops in BGP’sroutes. As replies arrive, it annotates routes in add routeand lookup route messages with the relevant IGP metrics.

Routes are held in a queue until the relevant nexthop met-rics are received; this avoids the need for the DecisionProcess to wait on asynchronous operations.

5.1.2 Dynamic Stages

The BGP process’s stages are dynamic, not static; newstages can be added and removed as the router runs. Wemade use of this capability in a surprising way when weneeded to deal with route deletions due to peer failure.When a peering goes down, all the routes received bythis peer must be deleted. However, the deletion of morethan 100,000 routes takes too long to be done in a singleevent handler. This needs to be divided up into slices ofwork, and handled as a background task. But this leadsto a further problem: a peering can come up and go downin rapid succession, before the previous background taskhas completed.

To solve this problem, when a peering goes down wecreate a new dynamic deletion stage, and plumb it in di-rectly after the Peer In stage (Figure 6).

PeerIn

FilterBank

PeerIn

FilterBank

Before peering goes down:

DeletionStage

After peering goes down:

FIGURE 6—Dynamic deletion stages in BGP

The route table from the Peer In is handed to the deletionstage, and a new, empty route table is created in the PeerIn. The deletion stage ensures consistency while grad-ually deleting all the old routes in the background; si-multaneously, the Peer In—and thus BGP as a whole—is immediately ready for the peering to come back up.The Peer In doesn’t know or care if background dele-tion is taking place downstream. Of course, the deletionstage must still ensure consistency, so if it receives anadd route message from the Peer In that refers to a prefixthat it holds but has not yet got around to deleting, thenfirst it sends a delete route downstream for the old route,and then it sends the add route for the new route. Thishas the nice side effect of ensuring that if the peeringflaps many times in rapid succession, each route is heldin at most one deletion stage. Similarly, routes not yetdeleted will still be returned by lookup route until afterthe deletion stage has sent a delete route message down-stream. In this way none of the downstream stages evenknow that a background deletion process is occurring—all they see are consistent messages. Even the deletionstage has no knowledge of other deletion stages; if thepeering bounces multiple times, multiple dynamic dele-tion stages will be added, one for each time the peer-


ing goes down. They will unplumb and delete themselveswhen their tasks are complete.

We use the ability to add dynamic stages for manybackground tasks, such as when routing policy filters arechanged by the operator and many routes need to be re-filtered and reevaluated. The staged routing table designsupported late addition of this kind of complex function-ality with minimal impact on other code.

5.2 RIB StagesOther XORP routing processes also use variants of thisstaged design. For example, Figure 7 shows the basicstructure of the XORPRIB process. Routes come into theRIB from multiple routing protocols, which play a simi-lar role to BGP’s peers. When multiple routes are avail-able to the same destination from different protocols, theRIB must decide which one to use for forwarding. Aswith BGP, routes are stored only in the origin stages, andsimilar add route, delete route and lookup routemessagestraverse between the stages.

OriginTable

OriginTable

Routesfrom RIP

StaticRoutes

Routesfrom IBGP

Routesfrom EBGP Redistributed

RoutesRoutes to BGP’s NexthopLookup stage

Routes toForwardingEngine

OriginTable

OriginTable

MergeStage

MergeStage

ExtIntStage

RedistStage

RegisterStage

RegisterStage

FIGURE 7—Staged RIB architecture

Unlike with BGP, the decision process in the RIB isdistributed as pairwise decisions between Merge Stages,which combine route tables with conflicts based on apreference order, and an ExtInt Stage, which composesa set of external routes with a set of internal routes. InBGP, the decision stage needs to see all possible alter-natives to make its choice; the RIB, in contrast, makesits decision purely on the basis of a single administra-tive distance metric. This single metric allows more dis-tributed decision-making, which we prefer, since it bettersupports future extensions.

Dynamic stages are inserted as different watchers reg-ister themselves with the RIB. These include RedistStages, which contain programmable policy filters to re-distribute a route subset to a routing protocol, and Regis-ter Stages, which redistribute routes depending on prefixmatches. This latter process, however, is slightly morecomplex than it might first appear.

5.2.1 Registering Interest in RIB RoutesA number of core XORP processes need to be able totrack changes in routing in the RIB as they occur. For

example, BGP needs to monitor routing changes that af-fect IP addresses listed as the nexthop router in BGProutes, and PIM-SM needs to monitor routing changesthat affect routes to multicast source addresses and PIMRendezvous-Point routers. We expect the same to be trueof future extensions. This volume of registrations putspressure on the Register Stage interface used to registerand call callbacks on the RIB. In monolithic or shared-memory designs centered around a single routing tablestructure, a router could efficiently monitor the structurefor changes, but such a design cannot be used by XORP.We need to share the minimum amount of informationbetween the RIB and its clients, while simultaneouslyminimizing the number of requests handled by the RIB.

What BGP and PIM want to know about is the rout-ing for specific IP addresses. But this list of addressesmay be moderately large, and many addresses may berouted as part of the same subnet. Thus when BGP asksthe RIB about a specific address, the RIB informs BGPabout the address range for which the same answer ap-plies.

128.16.0.0/16

128.16.128.0/17128.16.192.0/18.

128.16.0.0/18

128.16.0.0/18 128.16.128.0/18

Routesin RIB

RelevantRouting infosent to BGP

Interested in128.16.32.1

Interested in128.16.160.1

FIGURE 8—RIB interest registration

Figure 8 illustrates this process. The RIB holds routesfor 128.16.0.0/16, 128.16.0.0/18, 128.16.128.0/17 and128.16.192.0/18. If BGP asks the RIB about address 128.16.32.1, the RIB tells BGP that the matching route is 128.16.0.0/18, together with the relevant metric and nexthoprouter information. This address also matched 128.16.0.0/16, but only the more specific route is reported. If BGPlater becomes interested in address 128.16.32.7, it doesnot need to ask the RIB because it already knows thisaddress is also covered by 128.16.0.0/18.

However, if BGP asks the RIB about address 128.16.160.1, the answer is more complicated. The most spe-cific matching route is 128.16.128.0/17, and indeed theRIB tells BGP this. But 128.16.128.0/17 is overlayedby 128.16.192.0/18, so if BGP only knew about 128.16.128.0/17 and later became interested in 128.16.192.1, itwould erroneously conclude that this is also covered by128.16.128.0/17. Instead, the RIB computes the largestenclosing subnet that is not overlayed by a more specificroute (in this case 128.16.128.0/18) and tells BGP that itsanswer is valid for this subset of addresses only. Should


the situation change at any later stage, the RIB will send a“cache invalidated” message for the relevant subnet, andBGP can re-query the RIB to update the relevant part ofits cache.

Since no largest enclosing subnet ever overlaps anyother in the cached data, RIB clients like BGP can usebalanced trees for fast route lookup, with attendant per-formance advantages.

5.3 Safe Route Iterators

Each background stage responsible for processing a largerouting table, such as a BGP deletion stage, must remem-ber its location in the relevant routing table so that itcan make forward progress on each rescheduling. TheXORP library includes route table iterator data struc-tures that implement this functionality (as well as a Pa-tricia Tree implementation for the routing tables them-selves). Unfortunately, a route change may occur whilea background task is paused, resulting in the tree nodepointed to by an iterator being deleted. This would causethe iterator to hold invalid state. To avoid this problem,we use some spare bits in each route tree node to holda reference count of the number of iterators currentlypointing at this tree node. If the route tree receives arequest to delete a node, the node’s data is invalidated,but the node itself is not removed immediately unless thereference count is zero. It is the responsibility of the lastiterator leaving a previously-deleted node to actually per-form the deletion.

The internals of the implementation of route trees anditerators are not visible to the programmer using them.All the programmer needs to know is that the iteratorwill never become invalid while the background task ispaused, reducing the feature interaction problem betweenbackground tasks and event handling tasks.

6 INTER-PROCESS COMMUNICATION

Using multiple processes provides a solid basis for re-source management and fault isolation, but requires theuse of an inter-process communication (IPC) mechanism.Our IPC requirements were:

• to allow communication both between XORP pro-cesses and with routing applications not built usingthe XORP framework;

• to use multiple transports transparently, including intra-process calls, host-local IPC, and networked commu-nication, to allow a range of tradeoffs between flexi-bility and performance;

• to support component namespaces for extensibilityand component location for flexibility, and to providesecurity through per-method access control on com-ponents;

• to support asynchronous messaging, as this is a natu-ral fit for an event-driven system; and

• to be portable, unencumbered, and lightweight.

During development we discovered an additional re-quirement, scriptability, and added it as a feature. Beingable to script IPC calls is an invaluable asset during de-velopment and for regression testing. Existing messagingframeworks, such as CORBA [22] and DCOM [5], pro-vided the concepts of components, component address-ing and location, and varying degrees of support for al-ternative transports, but fell short elsewhere.

We therefore developed our own XORP IPC mecha-nism. The Finder process locates components and theirmethods; communication proceeds via a naturally script-able base called XORP Resource Locators, or XRLs.

6.1 XORP Resource Locators

An XRL is essentially a method supported by a compo-nent. (Because of code reuse and modularity, most pro-cesses contain more than one component, and some com-ponents may be common to more than one process; so theunit of IPC addressing is the component instance ratherthan the process.) Each component implements an XRLinterface, or group of related methods. When one com-ponent wishes to communicate with another, it composesan XRL and dispatches it. Initially a component knowsonly the generic component name, such as “bgp”, withwhich it wishes to communicate. The Finder must re-solve such generic XRLs into a form that specifies pre-cisely how communication should occur. The resultingresolved XRL specifies the transport protocol family tobe used, such as TCP, and any parameters needed forcommunication, such as hostname and port.

The canonical form of an XRL is textual and human-readable, and closely resembles Uniform Resource Lo-cators (URLs [1]) from the Web. Internally XRLs areencoded more efficiently, but the textual form permitsXRLs to be called from any scripting language via a sim-ple call xrl program. This is put to frequent use in all ourscripts for automated testing. In textual form, a genericXRL might look like:

finder://bgp/bgp/1.0/set local as?as:u32=1777

And after Finder resolution:

stcp://192.1.2.3:16878/bgp/1.0/set local as?as:u32=1777

XRL arguments (such as “as” above, which is an Au-tonomous System number) are restricted to a set of coretypes used throughout XORP, including network addresses,numbers, strings, booleans, binary arrays, and lists ofthese primitives. Perhaps because our application domainis highly specialized, we have not yet needed support formore structured arguments.


As with many other IPC mechanisms, we have an in-terface definition language (IDL) that supports interfacespecification, automatic stub code generation, and basicerror checking.

6.2 Components and the Finder

When a component is created within a process, it in-stantiates a receiving point for the relevant XRL proto-col families, and then registers this with the Finder. Theregistration includes a component class, such as “bgp”;a unique component instance name; and whether or notthe caller expects to be the sole instance of a particu-lar component class. Also registered are each interface’ssupported methods and each method’s supported proto-col families. This allows for specialization; for example,one protocol family may be particularly optimal for aparticular method.

When a component wants to dispatch an XRL, it con-sults the Finder for the resolved form of the XRL. In re-ply, it receives the resolved method name together witha list of the available protocol families and argumentsto bind the protocol family to the receiver. For a net-worked protocol family, these would typically includethe hostname, receiving port, and potentially a key. Onceresolved, the dispatcher is able to instantiate a sender forthe XRL and request its dispatch. XRL resolution resultsare cached, and these caches are updated by the Finderwhen entries become invalidated.

In addition to providing resolution services, the Finderalso provides a component lifetime notification service.Components can request to be notified when another com-ponent class or instance starts or stops. This mechanismis used to detect component failures and component restarts.

6.3 Protocol Families

Protocol families are the mechanisms by which XRLs aretransported from one component to another. Each pro-tocol family is responsible for providing argument mar-shaling and unmarshaling facilities as well as the IPCmechanism itself.

Protocol family programming interfaces are small andsimple to implement. In the present system, there arethree protocol families for communicating between XORPcomponents: TCP, UDP, and intra-process, which is forcalls between components in the same process. Thereis also a special Finder protocol family permitting theFinder to be addressable through XRLs, just as any otherXORP component. Finally, there exists a kill protocolfamily, which is capable of sending just one messagetype—a UNIX signal—to components within a host. Weexpect to write further specialized protocol families forcommunicating with non-XORP components. These willeffectively act as proxies between XORP and unmodifiedXORP processes.

7 SECURITY FRAMEWORK

Security is a critical aspect of building a viable extensibleplatform. Ideally, an experimental protocol running on aXORP router could do no damage to that router, whetherthrough poor coding or malice. We have not yet reachedthis ideal; this section describes how close we are.

Memory protection is of course the first step, andXORP’s multi-process architecture provides this. The nextstep is to allow processes to be sandboxed, so they cannotaccess important parts of the router filesystem. XORPcentralizes all configuration information in the RouterManager, so no XORP process needs to access the filesys-tem to load or save its configuration.

Sandboxing has limited use if a process needs to haveroot access to perform privileged network operations. Toavoid this need for root access, the FEA is used as a re-lay for all network access. For example, rather than send-ing UDP packets directly, RIP sends and receives pack-ets using XRL calls to the FEA. This adds a small cost tonetworked communication, but as routing protocols arerarely high-bandwidth, this is not a problem in practice.

This leaves XRLs as the remaining vector for dam-age. If a process could call any other XRL on any otherprocess, this would be a serious problem. By default wedon’t accept XRLs remotely. To prevent local circumven-tion, at component registration time the Finder includesa 16-byte random key in the registered method name ofall resolved XRLs. This prevents a process bypassing theuse of the Finder for the initial XRL resolution phase,because the receiving process will reject XRLs that don’tmatch the registered method name.

We have several plans for extending XORP’s secu-rity. First, the Router Manager will pass a unique secretto each process it starts. The process will then use this se-cret when it resolves an XRL with the Finder. The Finderis configured with a set of XRLs that each process is al-lowed to call, and a set of targets that each process is al-lowed to communicate with. Only these permitted XRLswill be resolved; the random XRL key prevents bypass-ing the Finder. Thus, the damage that can be done by anerrant process is limited to what can be done through itsnormal XRL calls. We can envisage taking this approacheven further, and restricting the range of arguments thata process can use for a particular XRL method. Thiswould require an XRL intermediary, but the flexibilityof our XRL resolution mechanism makes installing suchan XRL proxy rather simple. Finally, we are investigat-ing the possibility of running different routing processesin different virtual machines under the Xen [11] virtualmachine monitor, which would provide even better iso-lation and allow us to control even the CPU utilization ofan errant process.


0

2000

4000

6000

8000

10000

12000

14000

0 5 10 15 20 25

Per

form

ance

(X

RLs

/sec

)

Number of XRL arguments

XRL performance for various communication families

Intra-ProcessTCPUDP

FIGURE 9—XRL performance results

8 EVALUATION

As we have discussed, the XORP design is modular, ro-bust and extensible, but these properties will come atsome cost in performance compared to more tightly cou-pled designs. The obvious concern is that XORP mightnot perform well enough for real-world use. On previousgenerations of hardware, this might have been true, butwe will show below that it is no longer the case.

The measurements are performed on a relatively low-end PC (AMD Athlon 1100MHz) running FreeBSD 4.10.At this stage of development we have put very little ef-fort into optimizing the code for performance, but wehave paid close attention to the computation complex-ity of our algorithms. Nevertheless, as we show below,even without optimization the results clearly demonstrategood performance, and the advantage of our event-drivendesign.

8.1 XRL Performance Evaluation

One concern is that the XRL IPC mechanism might be-come a bottleneck in the system. To verify that it is not,the metric we are interested in is the throughput we canachieve in terms of number of XRL calls per second.

To measure the XRL rate, we send a transaction of10000 XRLs using a pipeline size of 100 XRLs. Ini-tially, the sender sends 100 XRLs back-to-back, and thenfor every XRL response received it sends a new request.The receiver measures the time between the beginningand the end of a transaction. We evaluate three com-munication transport mechanisms: TCP, UDP and Intra-Process direct calling where the XRL library invokes di-rect method calls between a sender and receiver insidethe same process.1

1To allow direct comparison of Intra-Process against TCP and UDP,both sender and receiver are running within the same process. When werun the sender and receiver on two separate processes on the same host,the performance is very slightly worse.

In Figure 9 we show the average XRL rate and itsstandard deviation for TCP, UDP and Intra-Process trans-port mechanisms when we vary the number of argumentsto the XRL. These results show that our IPC mechanismcan easily sustain several thousands of XRLs per sec-ond on a relatively low-end PC. Not surprisingly, for asmall number of XRL arguments, the Intra-Process per-formance is best (almost 12000 XRLs/second), but for alarger number of arguments the difference between Intra-Process and TCP disappears. It is clear from these resultsthat our argument marshalling and unmarshalling is notterribly optimal, but despite this the results are quite re-spectable. In practice, most commonly used XRLs havefew arguments. This result is very encouraging, becauseit demonstrates that typically the bottleneck in the systemwill be elsewhere.

The UDP performance is significantly worse becauseUDP was our first prototype XRL implementation, anddoes not pipeline requests. For normal usage, XORP cur-rently uses TCP and does pipeline requests. UDP is in-cluded here primarily to illustrate the effect of requestpipelining, even when operating locally.

8.2 Event-Driven Design Evaluation

To demonstrate the scaling properties of our event-drivendesign, we present some BGP-related measurements. Rout-ing processes not under test such as PIM-SM and RIPwere also running during the measurements, so the mea-surements represent a fairly typical real-world configura-tion.

First, we perform some measurements with an emptyrouting table, and then with a routing table containing afull Internet backbone routing feed consisting of 146515routes. The key metric we care about is how long it takesfor a route newly received by BGP to be installed into theforwarding engine.

XORP contains a simple profiling mechanism whichpermits the insertion of profiling points anywhere in thecode. Each profiling point is associated with a profilingvariable, and these variables are configured by an exter-nal program xorp profiler using XRLs. Enabling a profil-ing point causes a time stamped record to be stored, suchas:

route ribin 1097173928 664085 add 10.0.1.0/24

In this example we have recorded the time in seconds andmicroseconds at which the route “10.0.1.0/24” has beenadded. When this particular profiling variable is enabled,all routes that pass this point in the pipeline are logged.

If a route received by BGP wins the decision process,it will be sent to its peers and to the RIB (see Figure 1).When the route reaches the RIB, if it wins against routesfrom other protocols, then it is sent to the FEA. Finally,


0

1

2

3

4

5

0 50 100 150 200 250 300

Tim

e (m

s)

Route

Introduce 255 routes to a BGP with no routes

Queued for RIBSent to RIB

Arrived at RIBQueued for FEA

Sent to FEAArrived at FEA

Entering KernelProfile Point Avg SD Min Max

Entering BGP - - - -Queued for transmissionto the RIB 0.226 0.026 0.214 0.525Sent to RIB 0.280 0.026 0.267 0.582Arriving at the RIB 1.036 0.132 0.964 2.285Queued for transmissionto the FEA 1.139 0.132 1.066 2.389Sent to the FEA 2.234 1.903 1.965 31.663Arriving at FEA 2.876 1.918 2.580 32.443Entering kernel 3.374 1.980 3.038 33.576

FIGURE 10—Route propagation latency (in ms), no initial routes

0

1

2

3

4

5

0 50 100 150 200 250 300

Tim

e (m

s)

Route

Introduce 255 routes to a BGP with 146515 routes (same peering)

Queued for RIBSent to RIB

Arrived at RIBQueued for FEA

Sent to FEA

Arrived at FEA

Entering Kernel

Profile Point Avg SD Min Max


FIGURE 11—Route propagation latency (in ms), 146515 initial routes and same peering

the FEA will unconditionally install the route in the ker-nel or the forwarding engine.

The following profiling points were used to measurethe flow of routes:

1. Entering BGP

2. Queued for transmission to the RIB

3. Sent to the RIB

4. Arriving at the RIB

5. Queued for transmission to the FEA

6. Sent to the FEA

7. Arriving at the FEA

8. Entering the kernel

One of the goals of this experiment is to demonstratethat routes introduced into a system with an empty rout-ing table perform similarly to a system with a full BGPbackbone feed of 146515 routes. In each test we intro-duce a new route every two seconds, wait a second, andthen remove the route. The BGP protocol requires that

the next hop is resolvable for a route to be used. BGPdiscovers if a next hop is resolvable by registering inter-est with the RIB. To avoid unfairly penalizing the emptyrouting table tests, we keep one route installed during thetest to prevent additional interactions with the RIB thattypically would not happen with the full routing table.

The results are shown in Figures 10–12. In the firstexperiment (Figure 10) BGP contained no routes otherthan the test route being added and deleted. In the secondexperiment (Figure 11) BGP contained 146515 routesand the test routes were introduced on the same peeringfrom which the other routes were received. In the thirdexperiment (Figure 12) BGP contained 146515 routesand the test routes were introduced on a different peeringfrom which the other routes were received, which exer-cises different code-paths from the second experiment.

All the graphs have been cropped to show the mostinteresting region. At the tables indicate, one or two routestook as much as 90ms to reach the kernel. This appearsto be due to scheduling artifacts, as FreeBSD is not a re-altime operating system.

The conclusion to be drawn from these graphs is that


0

1

2

3

4

5

0 50 100 150 200 250 300

Tim

e (m

s)

Route

Introduce 255 routes to a BGP with 146515 routes (different peering)

Queued for RIB

Sent to RIBArrived at RIB

Queued for FEA

Sent to FEA

Arrived at FEA

Entering Kernel

Profile Point Avg SD Min Max


FIGURE 12—Route propagation latency (in ms), 146515 initial routes and different peering

routing events progress to the kernel very quickly (typi-cally within 4ms of receipt by BGP). Perhaps as impor-tantly, the data structures we use have good performanceunder heavy load, therefore the latency does not signif-icantly degrade when the router has a full routing table.The latency is mostly dominated by the delays inherentin the context switch that is necessitated by inter-processcommunication. We should emphasize that the XRL in-terface is pipelined, so performance is still good whenmany routes change in a short time interval.

We have argued that an event driven route process-ing model leads to faster convergence than the traditionalroute scanning approach. To verify this assertion we per-formed a simple experiment, shown in Figure 13. We in-troduced 255 routes from one BGP peer at one secondintervals and recorded the time that the route appearedat another BGP peer. The experiment was performed onXORP, Cisco-4500 (IOS Version 12.1), Quagga-0.96.5,and MRTD-2.2.2a routers. It should be noted that thegranularity of the measurement timer was one second.

This experiment clearly shows the consistent behav-ior achieved by XORP, where the delay never exceedsone second. MRTD’s behavior is very similar, which isimportant because it illustrates that the multi-process ar-chitecture used by XORP delivers similar performance toa closely-coupled single-process architecture. The Ciscoand Quagga routers exhibit the obvious symptoms of a30-second route scanner, where all the routes received inthe previous 30 seconds are processed in one batch. Fastconvergence is simply not possible with such a scanner-based approach.

Teixeira et al demonstrate [27] that even route changeswithin an AS can be adversely affected by the delay in-troduced by BGP route scanners. In real ISP networks,the found delays of one to two minutes were commonbetween an IGP route to a domain border router chang-ing, and the inter-domain traffic flowing out of a domainchanging its exit router. During this delay, they show that

0

5

10

15

20

25

30

35

40

0 50 100 150 200 250 300

Del

ay b

efor

e ro

ute

is p

ropa

gate

d (s

)

Route arrival time (s)

BGP route latency induced by a router

XORPMRTdCisco

Quagga

FIGURE 13—BGP route flow

transient forwarding loops can exist, or traffic may beblackholed, both of which may have significant impacton customers. Thus fast convergence is clearly of highimportance to providers, and can only become more sowith the increase in prominence of real-time traffic.

8.3 Extensibility Evaluation

The hardest part of our design to properly evaluate isits extensibility. Only time will tell if we really have theright modularity, flexibility, and APIs. However, we canoffer a number of examples to date where extensibilityhas been tested.

Adding Policy to BGP

We implemented the core BGP and RIB functionalityfirst, and only then thought about how to configure pol-icy, which is a large part of any router functionality. Ourpolicy framework consists of three new BGP stages andtwo new RIB stages, each of which supports a commonsimple stack language for operating on routes. The de-tails are too lengthy for this paper, but we believe this


framework allows us to implement almost the full rangeof policies available on commercial routers.

The only change required to pre-existing code wasthe addition of a tag list to routes passed from BGP tothe RIB and vice versa. Thus, our staged architecture ap-pears to have greatly eased the addition of code that isnotoriously complex in commercial vendors’ products.

What we got wrong was the syntax of the commandline interface (CLI) template files, described in [13], usedto dynamically extend the CLI configuration language.Our original syntax was not flexible enough to allow user-friendly specification of the range of policies that weneed to support. This is currently being extended.

Adding Route Flap Damping to BGP

Route flap damping was also not a part of our originalBGP design. We are currently adding this functionality(ISPs demand it, even though it’s a flawed mechanism),and can do so efficiently and simply by adding anotherstage to the BGP pipeline. The code does not impactother stages, which need not be aware that damping isoccurring.

Adding a New Routing Protocol

XORP has now been used as the basis for routing re-search in a number of labs. One university unrelated toour group used XORP to implement an ad-hoc wirelessrouting protocol. In practice XORP probably did not helpthis team greatly, as they didn’t need any of our existingrouting protocols, but they did successfully implementtheir protocol. Their implementation required a singlechange to our internal APIs to allow a route to be spec-ified by interface rather than by nexthop router, as thereis no IP subnetting in an ad-hoc network.

9 CONCLUSIONS

We believe that innovation in the core protocols support-ing the Internet is being seriously inhibited by the na-ture of the router software market. Furthermore, littlelong term research is being done, in part because re-searchers perceive insurmountable obstacles to experi-mentation and deployment of their ideas.

In an attempt to change the router software landscape,we have built an extensible open router software plat-form. We have a stable core base running, consisting ofaround 500,000 lines of C++ code. XORP is event-driven,giving fast routing convergence, and incorporates a multi-process design and novel inter-process communicationmechanisms that aid extensibility, and allow experimen-tal software to run alongside production software.

In this paper we have presented a range of innovativefeatures, including a novel staged design for core pro-tocols, and a strong internal security architecture geared

around the sandboxing of untrusted components. We alsopresented preliminary evaluation results that confirm thatour design scales well to large routing tables while main-taining low routing latency.

In the next phase we need to involve the academiccommunity, both as early adopters, and to flesh out thelong list of desirable functionality that we do not yetsupport. If we are successful, XORP will become a trueproduction-quality platform. The road ahead will not beeasy, but unless this or some other approach to enable In-ternet innovation is successful, the long-run consequencesof ossification will be serious indeed.

ACKNOWLEDGMENTS

We thank the other members of the XORP project, in-cluding Fred Bauer (Linux), Andrea Bittau (routing pol-icy), Javier Cardona (SNMP support), Adam Greenhalgh(initial BGP implementation), Luigi Rizzo (fast forward-ing), Bruce M. Simpson (Windows), and Marko Zec (Clickintegration and fast lookups). We gratefully acknowledgethose who financially supported XORP, including the ICSICenter for Internet Research, Intel Corporation, the Na-tional Science Foundation (ANI-0129541), and MicrosoftCorporation. We also sincerely thank Scott Shenker forbelieving in the work against all odds.

NOTES1We have some confirmation of this: a group implementing an ad-

hoc routing protocol found that XORP’s RIB supported their applica-tion with just one trivial interface change [17].

REFERENCES

[1] T. Berners-Lee, L. Masinter, and M. McCahill. Uniformresource locators. RFC 1738, Internet Engineering TaskForce, December 1994.

[2] Bird project. The BIRD Internet Routing Daemon (Website). http://bird.network.cz/.

[3] V. J. Bono. 7007 explanation and apology, 1997.http://www.merit.edu/mail.archives/nanog/1997-04/msg00444.html.

[4] Robert Braden, Ted Faber, and Mark Handley. From pro-tocol stack to protocol heap—Role-based architecture. InProc. 1st Workshop on Hot Topics in Networks (HotNets-I), October 2002.

[5] Nat Brown and Charlie Kindel. Distributed ComponentObject Model Protocol – DCOM 1.0. Online, Novem-ber 1998. Expired IETF draft draft-brown-dcom-v1-spec-02.txt.

[6] Kenneth L. Calvert, James Griffioen, and Su Wen.Lightweight network support for scalable end-to-end ser-vices. In Proc. ACM SIGCOMM 2002 Conference, pages265–278, August 2002.


[7] Cisco Systems. Cisco IOS software.http://www.cisco.com/public/sw-center/sw-ios.shtml.

[8] Cisco Systems. Cisco IOS XR software.http://www.cisco.com/en/US/products/ps5845/index.html.

[9] National Research Council. Looking Over the Fence atNetworks. National Academy Press, 2001.

[10] Christophe Diot, Brian Neil Levine, Bryan Lyles, HassanKassem, and Doug Balensiefen. Deployment issues forthe ip multicast service and architecture. IEEE Network,January/February 2000.

[11] B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, I. Pratt,A. Warfield, P. Barham, and R. Neugebauer. Xen and theart of virtualization. In Proc. 18th ACM Symposium onOperating Systems Principles, October 2003.

[12] D. Estrin, D. Farinacci, A. Helmy, D. Thaler, S. Deer-ing, M. Handley, V. Jacobson, C. Liu, P. Sharma,and L. Wei. Protocol Independent Multicast—Sparse Mode (PIM-SM): Protocol specification.RFC 2362, Internet Engineering Task Force, June1998. ftp://ftp.ietf.org/rfc/rfc2362.txt.

[13] M. Handley, O. Hodson, and E. Kohler. XORP: An openplatform for network research. In Proc. 1st Workshop onHot Topics in Networks (HotNets-I), October 2002.

[14] A. Helmy. Protocol indendent multicast-sparsemode (pim-sm) implementation document, 1996.http://netweb.usc.edu/pim/pimd/docs/.

[15] Juniper Networks. JunOS software.http://www.juniper.net/products/junos/.

[16] Eddie Kohler, Robert Morris, Benjie Chen, John Jan-notti, and M. Frans Kaashoek. The Click modular router.ACM Trans. on Computer Systems, 18(3):263–297, Au-gust 2000.

[17] Thomas Kunz. Implementing bcast (implementation re-port). http://www.sce.carleton.ca/wmc/code.html, March2004.

[18] Zhuoqing Morley Mao, Ramesh Govindan, GeorgeVarghese, and Randy H. Katz. Route flap damping ex-acerbates Internet routing convergence. In Proc. ACMSIGCOMM 2002 Conference, August 2002.

[19] David Mazieres. A toolkit for user-level file systems. InProc. USENIX 2001 Annual Technical Conference, pages261–274, June 2001.

[20] John Moy. OSPF Complete Implementation. Addison-Wesley, December 2000.

[21] NextHop Technologies. GateD releases (Web site).http://www.gated.org/.

[22] Object Management Group. Common Object RequestBroker Architecture Specification 3.0.3, March 2004.http://www.omg.org/cgi-bin/doc?formal/04-03-12.

[23] Larry Peterson, Tom Anderson, David Culler, and Timo-thy Roscoe. A blueprint for introducing disruptive tech-nology into the Internet. In Proc. 1st Workshop on HotTopics in Networks (HotNets-I), October 2002.

[24] Larry Peterson, Scott Shenker, and Jonathan Turner.Overcoming the internet impasse through virtualization.In Proc. 3rd Workshop on Hot Topics in Networks(HotNets-III), November 2004.

[25] Quagga project. Quagga Routing Suite (Web site).http://www.quagga.net/.

[26] Tammo Spalink, Scott Karlin, Larry Peterson, andYitzchak Gottlieb. Building a robust software-basedrouter using network processors. In Proc. 18th ACM Sym-posium on Operating Systems Principles, pages 216–229,October 2001.

[27] R. Teixeira, A. Shaikh, T. Griffin, and J. Rexford. Dy-namics of hot-potato routing in ip networks. In Proc.2004 ACM SIGMETRICS Conference on Measurementand Modeling of Computer Systems, June 2004.

[28] University of Michigan and Merit Network.MRT: Multi-threaded Routing Toolkit (Web site).http://www.mrtd.net/.

[29] Unix Manual Pages. routed - network RIP and router dis-covery routing daemon.

[30] David Wetherall. Active network vision and reality:lessons from a capsule-based system. In Proc. 17th ACMSymposium on Operating Systems Principles, pages 64–79, Kiawah Island, South Carolina, December 1999.

[31] Zebra project. GNU Zebra routing software (Web site).http://www.zebra.org/.


Documents

Designing Extensible IP Router Software · OSPF IS−IS router manager BGP4+ RIB Management Functions Unicast Routing Multicast Routing RIB = routing information base FEA = forwarding