”Operational stability of the Internet ” SANOG1 - Kathmandu, Nepal 2003-01-25 Kurtis Lindqvist

”Operational stability of the Internet”

SANOG1 - Kathmandu, Nepal 2003-01-25

Kurtis Lindqvist<[email protected]>

2003-01-25Sida 2

© 2003 - Netnod AB http://www.netnod.se/

Slides available at http://www.kurtis.pp.se/presentations

2003-01-25Sida 3


Who is Netnod?

• Continuation of D-GIX• Established in 1997• Operates four exchanges in Sweden

– Stockholm, Gothenburg, Malmö, Sundsvall

• Have been the first (only) exchange point to use DPT/SRP/RPR/802.17– Switching to Gigabit Ethernet

2003-01-25Sida 4


The Internet has changed

2003-01-25Sida 5


Different needs

• Medical applications on STM-1 based IP-VPNs

• On-line banking applications• Production dependent IP-VPNs• VoIP

• Down-time is starting to have an effect

2003-01-25Sida 6


The Internet of today won’t do it

IGP convergence

• Will impact your service offering – IP-VPNs– VoIP services– Banking?

Not really new……and relatively easy to fix

2003-01-25Sida 7



IGP convergence

– Can be improved with IGP timers for both ISIS and OSPF

2003-01-25Sida 8



IGP convergence

• Suggestions to fix the protocol also exists– See Cengiz Alaettinoglu presentation

at the ISIS-WG in San Diego Dec 2000• But also be done at Layer 2…

2003-01-25Sida 9



BGP Convergence

• Convergence is poor as documented by Abha Ahuja and Craig Labovitz

• Will also impact your service offering– How do I do intra-provider VoIP?

2003-01-25Sida 10



BGP convergence

• Harder to fix– Problem with the protocol

• A number of suggestions are out there– Dampening might harm you, see Randy

Bush et al at RIPE in Rhodes and last Nanog

• For a long term fix we need to find a better way of doing this…

2003-01-25Sida 11


..but we still need to do Interconnects…

2003-01-25Sida 12


Interconnections

• Transit connections

• Direct peerings

• Exchange points

2003-01-25Sida 13


Transit connections

• ”Where the rest goes”– Your way to the most of the Internet

• Often protected so that you won’t get isolated in case of failures

• Critical as otherwise you will be an isolated island of the Internet

2003-01-25Sida 14


Direct peerings

• Direct links between two ISPs• Normally redundant

– For economical ”fairness”

• Not really crucial as you can send traffic through the upstream– But might be crucial depending on

your traffic ratio

2003-01-25Sida 15


Exchange points

• Many ISPs connected to common and shared media

• Normally limited protection• Limited effect if lost

– But can still force large volume of traffic to be shifted and leading to higher cost

2003-01-25Sida 16


What if we loose an exchange point?

Tested in full scale at Netnod

Migration of mountain cave

2003-01-25Sida 17


Netnod is running a number of exchangepoint

medias• Cisco DPT / SRP

– 622 Mbps– 2.5Gbps

• FDDI– Legacy

• Gigabit Ethernet – Bridged to FDDI

2003-01-25Sida 18


Netnod setup - GigE 1 Customer router

Extreme switches

FDDI Switches

A B

2003-01-25Sida 19


Netnod setup - GigE 2

Customer routers

Extreme switches

FDDI Switches

A B

2003-01-25Sida 20


Netnod setup - DPT

2003-01-25Sida 21


…and then we migrated ”A”…

…so what happened?

Nothing really…

2003-01-25Sida 22


Government project

• Study of the overall stability of the Internet in Sweden– DNS– Exchange points

• Gathered a number of data– BGP logs of 10 ISPs…almost– BGP listening on US East/west coast– Load graphs

2003-01-25Sida 23


Traffic shifted as planned

2003-01-25Sida 24


No real BGP activity - withdrawals

2003-01-25Sida 25


No real BGP activity - Advertisement

2003-01-25Sida 26


BGP Activity - details

2003-01-25Sida 27


What conclusions can we make?

• Exchange points are not really crucial– Some ISPs actually shifted the traffic

beforehand• Interconnects can be made stable

– Layer 2 convergence will help– But GigE can also be made to work…– …but you need to be careful

2003-01-25Sida 28


So what do we think happened?

• Well, the only thing that really changed was the NEXT_HOP

• Backup paths was known• So BGP recalculation was very

limited, if at all

• But this needs planning and luck!– And not necessarily in that order…

2003-01-25Sida 29


Operational stability

• All this is examples of how to improve operational stability

• But why do I needed it?– The answer is obvious

• Or is it?– How dependent is your country on the

Internet?

2003-01-25Sida 30



• Netnod was to some extent created out of the needs for stable Internet in Sweden– The physical exchange is located in caves

owned by the government– But they have no control over it

• Gov and we where concerned about physical security– As far as I know unique - which have proved

to be useful

2003-01-25Sida 31



• Physical security is the easy part– Todays attacks are logical

• The Swedish government is moving into CERT like functions and is trying to find their role– The idea is good– But they add little value to the solution of

the real problem– Still they are needed and can be useful– Early start is goood

2003-01-25Sida 32



• The challenge in the future lies in moving forward on two fronts– Physical– Logical

• Both are well understood…• …but poorly implemented

2003-01-25Sida 33


Conclusions• Operational stability is becoming more

and more crucial– For revenue and infrastructure

• Work with your goverment before they work with you

• There is a lot you can do (should have done) to increase the stability of your network– And the rest of the Internet

• Conferences like these are good!

2003-01-25Sida 34


Go do it!

Before you wake up and is forced to!

2003-01-25Sida 35


?

2003-01-25Sida 36


ContactNetnod Internet Exchange i Sverige AB

Bellmansgatan 30I SE-118 47 Stockholm

Sweden

Office address: Bellmansgatan 30I

Telephone: +46-8-615 85 70

Telefax: +46-8-442 09 67 E-mail: [email protected]: http//www.netnod.se/

Documents

”Operational stability of the Internet ” SANOG1 - Kathmandu, Nepal 2003-01-25 Kurtis Lindqvist