Wresting Control from BGP: Scalable Fine-grained Route Control

UCSD / AT&T ResearchUsenix —June 22, 2007

Dan Pei, Tom Scholl, Aman Shaikh, Alex C. Snoeren,

Kobus van der Merwe

Patrick Verkaik

What is route control?

ISP

Routecontrol

?

?

Traffic

destination

Traffic

• ISPs need to control selection of alternate paths• … in response to dynamic network conditions

• ISPs need to control selection of alternate paths• … in response to dynamic network conditions

measurement

• Historically routing provides simple connectivity• But demands are changing: gaming, DDoS defense

• Historically routing provides simple connectivity• But demands are changing: gaming, DDoS defense

IRSCP: enabling route control

• IRSCP: Intelligent Route Service Control Point– Next step after RCP [Caesar et al.]

• Enable route control for inter-domain traffic:

– Automated, based on network conditions– Scalable to demands of large ISP– No changes to existing ISP infrastructure

Simple example: load balancing

Customer ISP

Traffic

Egress router R1 Ingress routerR5

• Border Gateway Protocol (BGP)• BGP defaults to hot potato routing• Effectively: shortest path routing

Overloaded link

Traffic on customer access links is balanced

R3

R2

R4

• Simple route control objective• Customers are asking for this• Yet BGP can’t do it!

• Simple route control objective• Customers are asking for this• Yet BGP can’t do it!

Customer 1

Why does BGP do that?

ISP 1

ISP 2

Customer 2

External BGP (eBGP) session

Internal BGP (iBGP) session

ISP router

External router

Select

routeSelec

t route

Select

route

Select

route

dest d

dest d

R1

R3

R2R4

• Hundreds of ISP routers making local decisions• Hot potato default policy:

– Ensures consistency– ..but precludes control over routing

• Hundreds of ISP routers making local decisions• Hot potato default policy:

– Ensures consistency– ..but precludes control over routing

IRSCP route control

• Abstraction: “for this destination, direct traffic from this ISP router to that ISP router”

• Automated, based on network conditions:route control application

• Backwards compatible, consistent and scalable

• Selective: allow default BGP decision where desired

Outline

• IRSCP architecture

• Routing consistency

• Implementation

• Evaluation

Customer 1

Route control abstraction

ISP 1

ISP 2

Customer 2Traffic

Egress links

RC App

iBGP session

RC App

IRSCP

Assign-ments

• RC App assigns egress links to ISP routers

• Speak BGP to routers• …using IRSCP

AssignRoute?

?

R4

R1

R3R2

Route failover

• What if a route for egress link fails?– RC application at relatively slow

timescale

– IRSCP must fail over instantly

• So application sends a list of egress links for each ISP router– We call this a ranking

Customer 1

Rankings

ISP 1

ISP 2

Customer 2

Traffic

RC App

IRSCP

Rankings

Rankings R1

R4

R3

R2

12

12

R2 R4

R3R1 Select route for

R1 and R3

Select route for

R2 and R4

• Rankings map traffic from ingress to egress arbitrarily

• And allow route fail-over at routing time-scale

• Rankings map traffic from ingress to egress arbitrarily

• And allow route fail-over at routing time-scale

What about scalability?

• IRSCP talks to many thousands of routers• Responsible for route decision for each ISP

router:– Computation– Single point of failure

• Maintaining BGP session for each router :– State per session– Each eBGP session may add a route

2-3 GB sufficient

Distributed IRSCP

Customer 1

Distributed IRSCP

ISP 1

ISP 2

Customer 2

• Multiple IRSCP servers:– To distribute BGP sessions– For geographic diversity

• Routers may peer with several IRSCP servers• IRSCP servers are replicas: exchange all routes

IRSCP

IRSCP

IRSCP

IRSCP session

R4

R1

R3R2

Outline

• IRSCP architecture

• Routing consistency

• Implementation

• Evaluation

Consistency

Forwarding anomalies:

Deflection

Traffic

R1

R2

R3

R2

Looping

R1

R3

Rankings must be “consistent”Rankings must be “consistent”

123

123

R1

R3

R2

R4

Example of inconsistent rankings

RC application checks consistency constraints on rankings

No anomalies

TrafficR1R2

Rankings

R3 R4Deflection

Outline

• IRSCP architecture

• Routing consistency

• Implementation

• Evaluation

Expl. rankeddecision process

Hot potato/BGP decision process

Routing information baseRoutes

Prototype implementation

IRSCPserver

RC App Rankings

R1

IRSCP server

R2

IRSCPserver

Import policy

Export policy

I1 I2

Outline

• IRSCP architecture• Routing consistency• Implementation• Evaluation

– Can IRSCP handle routing load in a real ISP?– Both explicitly ranked and hot potato

decision process

Methodology

• Emulation of realistic large ISP topology and routing load

• Connect IRSCP implementation to emulation of ISP

• Measure if our implementation handles the emulated load

Customer 1

ISP 2

Deployment Scenario

IRSCP240 external routers per POP

15 ISP routers per POP

40 POPs

POP

1 IRSCP server per POP

IRSCP session

BGP session

Throughput of IRSCP server depends on how many of each kind of session it has

Finding maximum throughput

IRSCP

Input update rate

Routeupdatereceiver

Mix of BGP and IRSCP sessionsbased on ISP scenario

IRSCP implementation:– 3.6-GHz Xeon– 4 GB memory

Search for maximum sustainable input rate:– Gradually increase input rate, sustaining for twenty minutes– Compute expected output rate given input rate– Once measured output rate falls behind, we’ve reached

maximum throughput

Output update rate

Routeupdate

generator

Multiplier

Maximum throughput

3600

Output rate

Input rateHot potato

Explicitly ranked2400

41000

27000

600

95% in real ISPAverage in real ISP

220updates/s

updates/s

• Flexibility of rankings has a cost• But IRSCP handles Tier-1 ISP routing load, and more

• Flexibility of rankings has a cost• But IRSCP handles Tier-1 ISP routing load, and more

Conclusion

• IRSCP route control platform:– Feedback of network conditions into route selection– Scalable, robust against failures, backward compatible– Powerful, yet safe ranking abstraction

• Enables new class of “route control application”:– Security– Traffic engineering– Customer-oriented services

Trials of IRSCP for several applications taking place in AT&T!

Alex

Aman

Questions?

Kobus

Tom

Dan

Patrick

BGP sessions

• As we saw, IRSCP speaks iBGP to ISP routers

• For full control, IRSCP must also speak eBGP to external routers

IRSCP

Multihop eBGPsession

iBGP session

I

E1

E2eBGPsession

Select

routeSelect

route for I

Example consistency constraint

If e1 outranks e2 at R1 then must also do so at router(e1 )=R3

• Two simple consistency constraints• Checked by RC application before sending to IRSCP• Example:

123

123

R1

R3

e1e2

e1

e2

R1R2

R3 R4e1 e2

R2

R4

ThroughputAchieved output rate

Estimated 95 perc. required input rate

Achieved input rate

Out of 255 routers per POP