BGP trends of an AS

Looking under the hood and diagnosing the noise.

Stephan MilletNetwork Engineer

Telstra

Intro• Informational Presentation• 16 month data collection of the BGP activity of AS1221

– Sept 2004 to Jan 2006

• Analysis of the BGP behavior within the AS

Background on AS1221• Only run IP (no other protocol mixes)• Topology hasn’t changes in 3 years• Geographically confined to Australia

– Only one transit provider

• All Cisco (consistency in the BGP algorithm(?))• Core acts as RR for all access routers

– All eBGP is conducted on access– No dampening on transit paths

n-1 BGP topologySydney

Melbourne

Brisbane

Adelaide

Canberra

Perth

Where all routers have an iBGP relationship with all other routers

Add a collector to the n-1 BGP topologySydney

Melbourne

Brisbane

Adelaide

Canberra

Perth

+

Turn on logging and wait 18 months

+

Current configuration:!hostname bgp-logger!log file /data/bgp-mesh/mesh-loglog trap informationallog record-priority!debug bgp updates!router bgp 1221<snip>

BGP: 192.168.1.1 rcvd UPDATE w/ attr: nexthop 203.62.252.59, origin ?,localpref 100, community 1221:500, originator 10.10.1.1, clusterlist 0.0.0.150BGP: 192.168.1.1 rcvd 203.36.198.80/29BGP: 192.168.1.1 rcvd 165.228.83.145/32BGP: 172.16.1.5 rcvd UPDATE w/ attr: nexthop 0.0.0.0, origin i, localpref 0BGP: 172.16.1.5 rcvd UPDATE about 58.163.130.105/32 -- withdrawnBGP: 172.16.1.5 rcvd UPDATE about 144.135.252.150/32 --withdrawnBGP: 172.16.1.5 rcvd UPDATE about 149.135.62.217/32 -- withdrawn

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

1 19 37 55 73 91 109 127 145 163 181 199 217 235 253 271 289 307 325 343 361 379 397 415 433

Time (days)

Up

da

tes

in M

illio

ns

BGP Updates

Prefix Updates

Flaps

Prefixes that updated

iBGP

eBGP

BGP Table

Linear (Prefix Updates)

Every 24hrs process the data and add an extra data point

What the initial data shows

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

05/09/04 05/11/04 05/01/05 05/03/05 05/05/05 05/07/05 05/09/05 05/11/05

Da

ily

up

da

te e

ve

nts

in

millio

ns

BGP Updates

Prefix Updates

Flaps

Prefixes that updated

iBGP

eBGP

BGP Table

Linear (Prefix Updates)

Analysis of initial data set.or what happened in 16 months

• 30% increase in AS1221’s BGP table– 176k to 228k prefixes

• A doubling or greater in all other attributes– Prefix updates from 600k to 1.2M per day– BGP updates from 200k to 550k per day

• A small amount of prefixes are creating a high portion noise– See ‘noisy 100’ later in presentation– ~10% - 15% of the prefixes in the BGP table will generate an update on

a daily basis.• eBGP prefixes are noisier that iBGP prefixes

– Though not for much longer• In 2004 eBGP to iBGP ratio was 4:1• Now eBGP to iBGP ratio is passing 2:1

• The really big spikes.– Operational work on one or more cores e.g. IOS upgrade– The rebooting of the mesh-logger

Looking at the trends.

Raw daily data views

Prefix updates are the new black

pink

Updates and Prefixes

0

100000

200000

300000

400000

500000

600000

700000

800000

900000

01/12/2004 01/02/2005 01/04/2005 01/06/2005 01/08/2005 01/10/2005 01/12/2005

Da

ily B

GP

me

ss

ag

es

0

1

2

3

4

5

6

7

8

9

Pro

po

rtio

n o

f p

refi

xe

s p

er

BG

P m

es

sa

ge

Daily BGP Messages

Avg Prefixes per BGP message

Linear (Daily BGP Messages)

Linear (Avg Prefixes per BGP message)

More going than coming ?

0

0.5

1

1.5

2

2.5

3

3.5

4

01/12/04 01/02/05 01/04/05 01/06/05 01/08/05 01/10/05 01/12/05

Da

ily p

refi

x u

pd

ate

s in

mill

ion

s

Prefix additions

Prefix withdrawls

Linear (Prefix withdrawls)

‘table size’ to ‘prefix updates ratio’

0

50000

100000

150000

200000

250000

300000

350000

05/0

9/0

4

05/1

1/0

4

05/0

1/0

5

05/0

3/0

5

05/0

5/0

5

05/0

7/0

5

05/0

9/0

5

05/1

1/0

5

05/0

1/0

6

05/0

3/0

6

05/0

5/0

6

05/0

7/0

6

05/0

9/0

6

05/1

1/0

6

05/0

1/0

7

05/0

3/0

7

05/0

5/0

7

05/0

7/0

7

05/0

9/0

7

05/1

1/0

7

05/0

1/0

8

Siz

e o

f B

GP

tab

le

0

3

6

9

12

15

18

21

Rati

o o

f d

aily p

refi

x u

pd

ate

s a

s a

pro

po

rtio

n o

f th

e B

GP

tab

le

AS1221 BGP Table sizeRatio of prefix updates to tableLinear (AS1221 BGP Table size)Linear (Ratio of prefix updates to table)

• By Jan 2006 the table to prefix noise ratio increased from 3:1 to 5:1– Today for every prefix in the BGP table expect

5 prefix updates– Heading towards 10:1 by 2008

• With continued BGP table growth, expect 3.0M prefix updates per day.

– How much of this is an artifact of the statistical technique (least squares best fit) and how much is a basic BGP artifact ?

‘table size’ to ‘prefix updates’ ratio continued..

Forecast: number of prefix updates

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

05

/09

/04

05

/11

/04

05

/01

/05

05

/03

/05

05

/05

/05

05

/07

/05

05

/09

/05

05

/11

/05

05

/01

/06

05

/03

/06

05

/05

/06

05

/07

/06

05

/09

/06

05

/11

/06

05

/01

/07

05

/03

/07

05

/05

/07

05

/07

/07

05

/09

/07

05

/11

/07

05

/01

/08

Da

ily p

refi

x u

pd

ate

s in

mill

ion

s

prefix updates

Poly. (prefix updates)

Linear (prefix updates)

Forecast prefix updates continued...

• Does it really matter ?– Hasn’t been a problem to date.

• Traffic is low.. ~1.5kbps

• What about the CPU and Memory on the RP ?

What’s the CPU doing ?

0

10

20

30

40

50

60

70

80

90

100

CP

U U

tili

sa

tio

n

0

50000

100000

150000

200000

250000

Tab

le s

ize

5 Min CPU

1 Min CPU

Routing table

Nov 2003 Nov 2004 Mar 2006 Mar 2008April 2004

GRP-A/256MbGRP-B/512Mb PRP-2/1Gb CRS-1/4Gb ??

What the memory is doing ?

GRP-B to PRP2 upgrade

0

50

100

150

200

250

300

350

04/08/04 04/10/04 04/12/04 04/02/05 04/04/05 04/06/05 04/08/05 04/10/05

Mb

Me

mo

ry u

se

d

0

50000

100000

150000

200000

250000

300000

350000

BG

P t

ab

le s

ize

Memory Usage

BGP Table

Are we OK ?

• Growth, growth and more growth

• 2009 onwards may be the end of a PRP2.– Will probably run at 100% (1Min average)– What happens when the CPU receives

updates faster than it can process them ?– AS’s flapping due to CPU issues will

exacerbate the issue.

Who are the culprits?

Who’s been naughty and who’s been nice.

Noisiest 100 Origin AS’s**

** Includes AS1221 as origin

0

50000

100000

150000

200000

250000

300000

350000

400000

01/12/04 01/02/05 01/04/05 01/06/05 01/08/05 01/10/05 01/12/05

To

p 1

00

Ori

gin

AS

's d

aily

up

da

tes

0.00

12.00

24.00

36.00

48.00

60.00

72.00

84.00

96.00

To

p 1

00

AS

's a

s a

% o

f to

tal d

aily

BG

P u

pd

ate

s

Daily top 100 origin AS updates

% of total daily BGP updates

One AS to rule them all

0

2000

4000

6000

8000

10000

12000

14000

01/12/04 01/02/05 01/04/05 01/06/05 01/08/05 01/10/05 01/12/05

Da

ily B

GP

up

da

tes

0

1

2

3

4

5

6

7

Pe

rce

nta

ge

of

Da

ily u

pd

ate

s

Daily AS9121 Updates

% of daily BGP updates

Info on AS9121

• Turk Telekom• Originate ~160 prefixes

– Snapshot on Jan 30 2006• Varying number of prefixes have an ‘origin’ tag of EGP

– Using really old software or munging routing policy ?– These prefixes seem to oscillate at will

#show ip route 85.100.117.0Routing entry for 85.100.117.0/24 Known via "bgp 1221", distance 200, metric 0 Tag 4637, type internal* 203.50.13.38, from 203.62.252.26, 00:00:56 ago

show ip bgp 85.100.117.0BGP routing table entry for 85.100.117.0/24, version 114434062Paths: (0 available, no best path)

Noisiest 100 Prefixes**

0

100000

200000

300000

400000

500000

600000

01/12/04 01/02/05 01/04/05 01/06/05 01/08/05 01/10/05 01/12/05

Dai

ly t

op

100

pre

fix

up

dat

es

0.00

6.00

12.00

18.00

24.00

30.00

36.00

To

p 1

00 P

refi

xes

as a

% o

f to

tal d

aily

pre

fix

up

dat

es

Top 100 daily prefixes

Daily % of total prefixes

** Includes AS1221 as origin

One prefix to rule them all

0

2000

4000

6000

8000

10000

12000

14000

12/12/04 12/02/05 12/04/05 12/06/05 12/08/05 12/10/05 12/12/05

Da

ily P

refi

x U

pd

ate

s

0

0.5

1

1.5

2

2.5

3

3.5

Pe

rce

nta

ge

of

Da

ily U

pd

ate

s

Daily 61.4.0.0/19 updates

% of daily updates

What can we do ?

• Not run DFZ’s

• Bigger processors, good for those that can afford it.– However come 2009, those that can’t need

alternate steps or issue gets worse for everyone.

• Limit updates ?– Turn on Flap Dampening ?

Questions ?