New Interdomain Routing Architectures

Jennifer Rexford

Well, BGP Has Some Problems

• Instability– Not guaranteed to converge to a unique,

stable solution (e.g., oscillation and bi-stable solutions)

• Slow convergence– Path exploration can take a very long time

• Non-linearity– Small changes can have large effects (e.g.,

intradomain changes leading to BGP changes)

• Feature interaction– Unexpected side effects (e.g., the interaction

of route-flap damping and path exploration)

Well, BGP Has Some More Problems

• Scalability challenges– Large memory, processing, and message-

passing overhead, dependent on behavior in other ASes

• Security vulnerabilities– BGP update messages are relatively easy to

forge with bogus prefix, AS path, or other attributes

• Difficult to configure– Operators must express their (many) goals in

an indirect and contorted manner• Difficult to troubleshoot

– Hard to infer the root cause of a routing change, or even the AS(es) responsible

But Wait, There’s More!

• No performance guarantees– BGP announcements do not include any

information about the performance of the path

• Limited flexibility for path selection– Only single-path routing on destination prefix

• Limited control over path selection– Chosen path is a byproduct of the composition

of local routing policies in many different ASes

• Forwarding path may not match AS path– Routers may deflect packets to other paths

(e.g., route aggregation, misconfiguration, and malice)

And, Perhaps the Biggest Problem of All…

• Hard to change– BGP is the glue that holds the disparate

parts of the Internet together– So, hard to change BGP in a fundamental

ways– … or is it?

Why is BGP Such a Mess?

• Absence of underlying models– Protocol designed without a design for the

decision process or policy language – BGP models (e.g., SPP) came much later

• Incremental evolution of the protocol– Many additions to BGP over the years– New route attributes and decision-process

steps• Rapid growth of the Internet

– Many ASes, and much more complex topology• Commercialization of the Internet

– More complex routing policies and competition– Heightened importance of security concerns

What’s a Networking Researcher to Do?

• Characterize the existing routing system– Measurement, modeling, simulation

• Improved operational practices– Best common practices for configuration

• Timer settings, route filtering, features to use, …

– Methods and tools for complicated tasks• Traffic engineering, config checking, root-cause

analysis

• Incremental changes to BGP– Better implementations of the protocol– New route attributes for greater flexibility– Graceful handling of failures, config changes, …

What’s a Networking Researcher to Do?

• Build on top of the existing routing system– Overlay networks

• Propose new architectures anyway– Identify and evaluate new and better solutions– … even if we can’t readily deploy them today

• Explore incremental-deployment approaches– Meta solutions that enable new architectures– With the goal of leading to a better architecture

• Create models of the fundamental limits– Maybe the problem BGP is solving is really

hard…

Key Ingredients of Architectural Proposals

• More control at the edge– Source routing and multi-path routing

• Negotiation between ASes– Explicit coordination for path selection

• Design for the common case– Handle common routing policies well (e.g.,

HLP)

• Servers computing the routes– Greater visibility and control than any one

router

• Multiple simultaneous architectures– Virtualization to support customized solutions

Source Routing

• The ultimate in flexibility– Sender determines path for each packet– At the router level, or at the AS level

• At the cost of…– Overhead of propagating topology information– Need for fast path changes after a failure– Lost control for intermediate routers/ASes

A

B

D

F

C

EADECF

ABEF

Source Routing Deployment Challenges

• ASes have little incentive to cooperate– No business relationship with ASes far away– Don’t want to relinquish control over routing– So, source routing is typically disabled

• Policy-compliant source routing– Limiting what sources routes can be used

• Progress on limited forms of source routing– Overlay networks

• Source routing on an overlay topology

– Multi-homing route control • “One hop source routing”

Multipath Routing

• Expose more paths to ASes– Multiple paths through same next-hop AS– Giving ASes greater control over path

selection

• Extreme case: export all paths– High protocol overhead– Lost control for intermediate ASes

• Compromise: export selective paths– Under control of intermediate ASes– Based on their routing policies, pricing, etc.

Exposing Extra Paths

• Example: AS A doesn’t like AS E– Default routes both go through E– Good to learn alternate route that avoids E

A

B

D

F

C

E

CF*CEFCBEF

EF *ECF

BEF*BCF

DEF*DABEF

ABEF*ADEF

BCFF*

Encapsulation to Use Non-Default Paths

• Direct packet along alternate route– Destination-based forwarding not enough– Encapsulate the packet to egress point

A

B

D

F

C

E

d d

e

e

Inter-AS Negotiation

• Directives– Tell other AS what to do– E.g., which link to use

• Suggestions– Tell other AS your

wishes– Which link you prefer

used

• Cooperation– Negotiate where to send– Across flows and time– Inbound and outbound– Trade small losses for

larger gain (“win win”)Customer A

Customer B

multiplepeeringpoints

Provider A

Provider B

Early-exit routing

Inter-AS Negotiation

• But, how to do it?– What info to

exchange?– How to prioritize the

many choices?– How prevent cheating?

• Open research territory– Some initial work on

exchanging preferences

– E.g., ASes exchange preferences, and iteratesCustomer A

Customer B

multiplepeeringpoints

Provider A

Provider B

Early-exit routing

Revisiting the Division of Labor

• Routers– Forward packets: yes– Compute paths: maybe not

• End users or ASes– Dictate path properties: yes– Control path selection: maybe not

• Intermediate ASes (e.g., ISPs)– Forward packets: yes– Business relationships: yes– Compute end-to-end paths: maybe not

Removing Routing From Routers

• Should routers forward packets: yes– Must be done at high speed– … in line-card hardware on fast routers– So, needs to be done on the routers

• Should routers compute routes: no– Hard to do in a distribution fashion– Difficult to make load-sensitive routing stable– Lacking complete information for good

decisions– Not flexible enough for end users– Difficult to extend over time

Routing As a Service (RAS)

• Goal: third parties pick end-to-end paths for clients to satisfy diverse user objectives

• Forwarding infrastructure– Basic routing (e.g., default routing)– Primitives for inserting forwarding-table entries

• Routing Service Provider (RSP)– Contracts ISPs for service (e.g., virtual links)– Selects end-to-end routes on behalf of clients– Competes with other selectors for customers

• End host– Queries RSP to pick path & install forwarding

state

Routing As a Service (RAS)

RSP

ISP

ISP

ISP

user

Routing Control Platform (RCP)

• Goal: Move beyond today’s artifacts, while remaining compatible with the legacy routers

• Incentive compatibility: phased evolution– Intelligent route reflector in a single AS– Learning BGP routes directly from neighbor ASes– Interdomain routing between RCPs

• Backwards compatibility: BGP to the routers– Using BGP to “push” answers to the routers– No need to change the legacy routers at all– Keep message format and router implementation

iBGP

eBGP

RCPiBGP

eBGP

RCP

AS 3AS 2AS 1

iBGP

Physicalpeering

Inter-AS ProtocolRCP RCP RCP

How Does This Differ From Overlays

• Overlays: circumventing the underlay– Host nodes throughout the network– Logical links between the host nodes– Active probes to observe the performance– Direct packets through good intermediate

nodes

• Routing services: controlling the underlay– Servers collect data directly from the routers– Servers compute forwarding tables for the

routers– Data packets do not go through the servers– Like an overlay for managing the underlayMaybe some combination of the two makes sense?

Concurrent Architectures Better than One

• Multiple customized architectures in parallel– Multiple virtual routers on a single platform– Resource isolation in CPU, forwarding table,

bandwidth– Programmability for different protocols and

mechanisms (routing, forwarding, addressing, …)

Cabo: Applications Within an Single ISP

• Customized virtual networks– Security for online banking– Fast-convergence for VoIP and gaming– Specialized handling of suspicious traffic

• Testing and deploying new protocols– Evaluate on a separate virtual network– Rather than in a dedicated test lab– Large scale and early-adopter traffic

• Leasing virtual components to others– ISPs have unused node and link capacity– Can allow others to construct services on top

Cabo: Enabling Economic Refactoring

• Infrastructure providers: Maintain routers, links, data centers, other physical infrastructure

• Service providers: Offer end-to-end services (e.g., layer 3 VPNs, SLAs, etc.) to users

Infrastructure Providers Service Providers

Today: ISPs try to play both roles, and cannot offer end-to-end services

Key Ingredients of Incremental Deployability

• Backwards compatibility– Technically possible to deploy the solution– E.g., change anything but the message format

• Incentive compatibility– Offer concrete benefits to early adopters– Generate incentives for others to follow– Do not rely on complete participation

• QoS, multicast, secure routing, IPv6, …• Need creativity on incremental

deployment

Lessons on Incremental Deployment

• Adding on top: tunneling in the data plane– Circumvent destination-based forwarded– Traverse routers

• Adding on top: servers in the control plan– Tricking the routers into doing the server’s

bidding– Implementing new functionality on the servers

• Sneaking in on the side: virtualization– Running additional virtual networks in parallel– Offering new functionality for special

applications– … while continuing to support “legacy” Internet

Discussion

• Tussles between stake-holders– Users and ISPs

• Division of function– Data, control, and management planes– End host, edge routers, and core routers

• How much better could BGP be?– While still being an interdomain protocol

with control distributed across Autonomous Systems?

– With an even cleaner slate than that?– Where should the economics be???