Charles Shen and Henning Schulzrinne Department of Computer Science Columbia University IRT Group 06/10/04 NSIS – an overview and its interaction with

IRT Group 06/10/04IRT Group 06/10/04

Charles Shen and Henning Schulzrinne Charles Shen and Henning Schulzrinne Department of Computer Science

Columbia University

NSIS – an overview and its NSIS – an overview and its interaction with Route interaction with Route

ChangeChange


Why signaling?Why signaling?• NSIS stands for Next Step in SignalingNSIS stands for Next Step in Signaling• Signaling is about manipulating state in Signaling is about manipulating state in

network nodesnetwork nodes• State Management examples: State Management examples:

– QoS resource allocationsQoS resource allocations– NAT and firewall configurationNAT and firewall configuration– state used in active networkingstate used in active networking

• State Monitoring examples:State Monitoring examples:– instantaneous path property discovery (available instantaneous path property discovery (available

bandwidth, queuing delay etc.)bandwidth, queuing delay etc.)• Need for a general purpose signaling Need for a general purpose signaling

protocol protocol


How about RSVP?How about RSVP?• Original Resource reSerVation Protocol: Original Resource reSerVation Protocol:

– Standardized in 1997Standardized in 1997– designed for en-to-end resource reservations designed for en-to-end resource reservations

based on the Integrated Service architecture.based on the Integrated Service architecture.

• RSVP extensions: RSVP extensions: – RSVP refresh reduction extensionsRSVP refresh reduction extensions– RSVP operation over IP tunnelsRSVP operation over IP tunnels– RSVP-Traffic Engineering over MPLS networksRSVP-Traffic Engineering over MPLS networks– RSVP-AggregationRSVP-Aggregation– RSVP for NAT and Firewall traversalRSVP for NAT and Firewall traversal– RSVP mobility RSVP mobility


Why not RSVP?Why not RSVP?

• Not designed to be a generic signaling Not designed to be a generic signaling protocol from the first placeprotocol from the first place

• Unclear whether independently Unclear whether independently designed RSVP extensions would designed RSVP extensions would collaborate in a single implementationcollaborate in a single implementation

• There are also criticisms for using There are also criticisms for using RSVP in QoS - the built-in multicast RSVP in QoS - the built-in multicast support adds considerable overhead, support adds considerable overhead, but very often may be unnecessarybut very often may be unnecessary


Here Comes NSIS …Here Comes NSIS …

• Working group chartered in Nov. 2001 Working group chartered in Nov. 2001 to investigate the architecture and to investigate the architecture and protocol design of the next (generic) protocol design of the next (generic) signaling protocol for the Internet based signaling protocol for the Internet based on existing experienceson existing experiences

• Creating requirements, frameworks, and Creating requirements, frameworks, and specifications for a signaling protocol specifications for a signaling protocol envisioned to support various signaling envisioned to support various signaling applications that need to install and/or applications that need to install and/or manipulate state in the networkmanipulate state in the network


Current NSIS Assumptions - Current NSIS Assumptions - Path-coupled and Unicast Path-coupled and Unicast

• Signaling messages are routed, flow Signaling messages are routed, flow state is installed and maintained, only state is installed and maintained, only through NSIS Entities (NEs) that are in through NSIS Entities (NEs) that are in the data paththe data path

• Not every node along the path has to be Not every node along the path has to be NEs. There could be proxies distinct NEs. There could be proxies distinct from the sender and receiver, or from the sender and receiver, or intermediate signaling-unaware nodesintermediate signaling-unaware nodes

• Only unicast flows are consideredOnly unicast flows are considered


NSIS Two-layered ModelNSIS Two-layered Model

• Several aspects of protocol support to Several aspects of protocol support to exchange messages along the data path are exchange messages along the data path are common to all or a large number of common to all or a large number of applications, and hence should be developed applications, and hence should be developed as a common protocol as a common protocol – a 'signaling transport' layer moving signaling a 'signaling transport' layer moving signaling

messages aroundmessages around– independent of any particular signaling applicationindependent of any particular signaling application– NSIS Transport Layer Protocol (NTLP)NSIS Transport Layer Protocol (NTLP)

• a 'signaling application' layer a set of state a 'signaling application' layer a set of state management rulesmanagement rules– message formats and sequences, specific to a message formats and sequences, specific to a

particular signaling application. particular signaling application. – NSIS Signaling Layer Protocol (NSLP)NSIS Signaling Layer Protocol (NSLP)


More about NTLP - GIMPS More about NTLP - GIMPS

• Based on existing transport and security Based on existing transport and security protocols under a common messaging layer, protocols under a common messaging layer, the General Internet Messaging Protocol for the General Internet Messaging Protocol for Signaling (GIMPS)Signaling (GIMPS)

• Different signaling applications make use of Different signaling applications make use of different GIMPS services, but GIMPS does not different GIMPS services, but GIMPS does not keep state specific to signaling applications.keep state specific to signaling applications.

• GIMPS manages its own internal state and GIMPS manages its own internal state and configures underlying transport and security configures underlying transport and security protocols to deliver signaling messages on protocols to deliver signaling messages on behalf of signaling applications in both behalf of signaling applications in both directions along the flow pathdirections along the flow path


GIMPS OperationsGIMPS Operations

• ““Routing” - Determine how to reach the Routing” - Determine how to reach the adjacent GIMPS peer along the data adjacent GIMPS peer along the data pathpath– Downstream signaling: either use explicit Downstream signaling: either use explicit

local state information to determine the local state information to determine the GIMPS peer, or just send the signaling GIMPS peer, or just send the signaling towards the flow destination address and towards the flow destination address and rely on the peer to intercept it.rely on the peer to intercept it.

– Upstream signaling: has to rely on explicit Upstream signaling: has to rely on explicit local state information. local state information.

• ““Transport” - Deliver signaling Transport” - Deliver signaling information to the Peerinformation to the Peer– Datagram mode and Connection ModeDatagram mode and Connection Mode


GIMPS Datagram ModeGIMPS Datagram Mode

• A mode of sending GIMPS messages A mode of sending GIMPS messages between nodes without using any between nodes without using any transport layer state or security protectiontransport layer state or security protection

• For small, infrequent messages with For small, infrequent messages with modest delay constraintsmodest delay constraints

• UDP as the initial choiceUDP as the initial choice• Upstream messages: UDP encapsulated Upstream messages: UDP encapsulated

and sent directly to the signaling and sent directly to the signaling destinationdestination

• Downstream messages: set up router alert Downstream messages: set up router alert option and sent towards the flow receiveroption and sent towards the flow receiver


GIMPS Connection ModeGIMPS Connection Mode

• A mode of sending GIMPS messages directly A mode of sending GIMPS messages directly between nodes using point to point between nodes using point to point "messaging associations“ i.e. transport "messaging associations“ i.e. transport protocols and security associationsprotocols and security associations

• For larger data objects or where fast setup in For larger data objects or where fast setup in the face of packet loss is desirable, or where the face of packet loss is desirable, or where channel security is requiredchannel security is required

• May use any stream or message-oriented May use any stream or message-oriented transport protocoltransport protocol

• May employ specific network layer security May employ specific network layer security associations (e.g. IPsec), or an internal associations (e.g. IPsec), or an internal transport layer security association (e.g. transport layer security association (e.g. TLS)TLS)


Route Change ProblemRoute Change Problem

• In a connectionless network each packet is In a connectionless network each packet is routed independentlyrouted independently– packet route may change in the middle of a session.packet route may change in the middle of a session.

• Route change can happen due to various Route change can happen due to various reasons: reasons: – Load sharing or load balancingLoad sharing or load balancing– Policy based routingPolicy based routing– Network reconfigurationNetwork reconfiguration– Routing protocol adaptationRouting protocol adaptation

• Route changes cause a divergence between Route changes cause a divergence between the data path and the path on which signaling the data path and the path on which signaling state has been installedstate has been installed


NSIS and Route ChangeNSIS and Route Change

• GIMPS (NTLP) and signaling application (NSLP) GIMPS (NTLP) and signaling application (NSLP) state needs to be updatedstate needs to be updated– Local repair if possibleLocal repair if possible– Not simply moving state from the old to the new Not simply moving state from the old to the new

path. Application dependent. E.g. QoS path path. Application dependent. E.g. QoS path characteristics. characteristics.

• GIMPS is responsible for:GIMPS is responsible for:– Detecting the route changeDetecting the route change– Update its own routing stateUpdate its own routing state– Inform relevant signaling applications at affected Inform relevant signaling applications at affected

nodesnodes

• A core issue is Route Change DetectionA core issue is Route Change Detection– NTLP and NSLP soft-state mechanism is not NTLP and NSLP soft-state mechanism is not

sufficient especially when NSLP refresh times are sufficient especially when NSLP refresh times are extended to reduce signaling loadextended to reduce signaling load


Route Change Detection - Route Change Detection - Summary of MethodsSummary of Methods• Routing Monitoring: Routing Monitoring:

– Local Trigger delivered by the local routing table Local Trigger delivered by the local routing table – Extended Trigger by checking a link-state routing Extended Trigger by checking a link-state routing

table (OSPF, BGP with AS_PATH)table (OSPF, BGP with AS_PATH)• Packet Monitoring:Packet Monitoring:

– GIMPS C-mode Monitoring: TTL/Interface changeGIMPS C-mode Monitoring: TTL/Interface change– Data Plane Monitoring: TTL/Interface change or Data Plane Monitoring: TTL/Interface change or

loss of flow all-togetherloss of flow all-together• GIMPS D-mode ProbingGIMPS D-mode Probing

– Each GIMPS node periodically repeats the Each GIMPS node periodically repeats the discovery operationdiscovery operation

– Depending on the likely stability of routes, the Depending on the likely stability of routes, the discovery period can be set to a large valuediscovery period can be set to a large value


Route Change Detectability Route Change Detectability Upstream and Downstream Upstream and Downstream Path Path • For a single node, route change may happen For a single node, route change may happen

in its upstream path or downstream pathin its upstream path or downstream path• A change in downstream path at a node A change in downstream path at a node

means a change in upstream path for its means a change in upstream path for its downstream peer.downstream peer.

• Data Packet Monitoring may only be used to Data Packet Monitoring may only be used to detect route change in the upstream pathdetect route change in the upstream path

• Routing Monitoring is usually used to detect Routing Monitoring is usually used to detect route change in the downstream pathroute change in the downstream path

• C-mode signaling monitoring can be used to C-mode signaling monitoring can be used to detect route change in both directionsdetect route change in both directions


Route Change DetectabilityRoute Change DetectabilityCase StudyCase Study

Old Path

New Path

C

F

G

B E

D

OSPF Area

A B

G

C D

E F

Square: NE circle: none – NE

B: Divergence Node E: Convergence Node

Upstream: F (data packet TTL monitoring) D (Data absence)

Downstream: A (extended trigger)

Data flow direction


Route Change DetectabilityRoute Change Detectability- Some Recommendations- Some Recommendations

• By similarly studying other OSPF Intra-By similarly studying other OSPF Intra-area/Inter-area, RIP Intra-AS, BGP Inter-area/Inter-area, RIP Intra-AS, BGP Inter-AS route change cases, we found that:AS route change cases, we found that:

• To increase the chance of Route Change To increase the chance of Route Change Detection, it is recommended to Detection, it is recommended to implement NE at least in all “special” implement NE at least in all “special” routers, e.g., AS border routers, OSPF routers, e.g., AS border routers, OSPF area border routers, OSPF backbone area border routers, OSPF backbone area routersarea routers


Route Change in Real World Route Change in Real World ??

• We want to know – statistically,We want to know – statistically,• How frequently does a route How frequently does a route

change occur? change occur? • How long does it last?How long does it last?• How many hops does it affect? How many hops does it affect? • How is it associated with TTL How is it associated with TTL

change? Or AS change?change? Or AS change?• ……


Network Measurement Network Measurement with Traceroutewith Traceroute• Traceroute to record the path between two Traceroute to record the path between two

sitessites• Traceroute creates ICMP 'probes' datagrams Traceroute creates ICMP 'probes' datagrams

with incrementing Time To Live (TTL) values. with incrementing Time To Live (TTL) values. It continues to send out probes until it It continues to send out probes until it reaches the final destination.reaches the final destination.

• Internet routes may change between two Internet routes may change between two successive probe packets, there is no successive probe packets, there is no guarantee that probes of different hops take guarantee that probes of different hops take the same route as pervious hopsthe same route as pervious hops

• If self-consistent and shows no multiple If self-consistent and shows no multiple routing for any of its hops, treat as a valid routing for any of its hops, treat as a valid measurement.measurement.


Traceroute ExampleTraceroute Example

traceroute to lava.net (64.65.64.17), 30 hops max, 40 byte packets 1 fe0-0r0.ffm1.de.carpe.net (212.96.130.129) 0.907 ms 0.846 ms 0.775 ms 2 w1-0r0.ffm0.de.carpe.net (212.96.129.5) 2.938 ms 7.893 ms 6.443 ms 3 ge-3-1-0-4.fra20.ip.tiscali.net (213.200.64.37) 3.038 ms 7.679 ms 5.33 ms 4 so-3-0-0.was21.ip.tiscali.net (213.200.81.50) 99.954 ms 96.689 ms 94.49 ms 5 so-4-2-0.edge2.Washington1.Level3.net (4.68.127.61) 93.562 ms 97.801 ms 101.415 ms 6 so-1-1-0.bbr2.Washington1.Level3.net (64.159.3.65) 93.641 ms 93.743 ms 94.06 ms 7 ge-0-0-0.mpls1.Honolulu2.Level3.net (4.68.128.13) 221.634 ms 217.193 ms 221.168 ms 8 so-7-0.hsa1.Honolulu2.Level3.net (4.68.112.90) 218.791 ms 224.496 ms 224.18 ms 9 s1.lavanet.bbnplanet.net (4.24.134.18) 216.187 ms 216.262 ms 222.657 ms10 malasada.lava.net (64.65.64.17) 221.802 ms 224.026 ms 220.563 ms

Location: www.debug.net --to-- lava.net Status: ok Time: Fri Apr 09 07:12:23 EDT 2004 --to-- Fri Apr 09 07:12:29 EDT 2004


Measurement Measurement MethodologyMethodology• Obtained permission from 39 public Obtained permission from 39 public

traceroute servers located in US, Canada, traceroute servers located in US, Canada, Iceland, Netherlands, Finland, Austria, Iceland, Netherlands, Finland, Austria, France, UK, Germany, Switzerland, Ireland, France, UK, Germany, Switzerland, Ireland, Bulgaria, Sweden, South Africa, New Bulgaria, Sweden, South Africa, New Zealand, Taiwan, Australia, Japan, ThailandZealand, Taiwan, Australia, Japan, Thailand

• Data collection program written in Tcl. Data collection program written in Tcl. • At an exponentially distributed interval, the At an exponentially distributed interval, the

program:program:– Randomly selects two sites as source and Randomly selects two sites as source and

destination and spawns a thread for each of themdestination and spawns a thread for each of them– Each thread sends a request to the selected Each thread sends a request to the selected

source or destination and invokes a traceroute to source or destination and invokes a traceroute to the other partythe other party


Two Data SetsTwo Data Sets

• Data Set I is among 12 of these 39 Data Set I is among 12 of these 39 sites with an measurement interval of sites with an measurement interval of 15 min, collected from Apr 09 – Apr. 15 min, collected from Apr 09 – Apr. 24. Contains about 15,000 traceroute 24. Contains about 15,000 traceroute recordsrecords

• Data Set II is among all 39 sites but Data Set II is among all 39 sites but with a relatively long interval of 2 with a relatively long interval of 2 hours. It was started on Apr. 22 – May. hours. It was started on Apr. 22 – May. 18.18.


More on Data Set IMore on Data Set I

• 12 sites that allowed the 15 minutes interval12 sites that allowed the 15 minutes interval– www.valkaryn.netwww.valkaryn.net (LA, CA)(LA, CA)– www.slac.stanford.eduwww.slac.stanford.edu (Stanford, CA)(Stanford, CA)– lava.netlava.net (Honolulu, HI)(Honolulu, HI)– www.fh-friedberg.dewww.fh-friedberg.de (Frieberg, Germany)(Frieberg, Germany)– www.lf.netwww.lf.net (Germany)(Germany)– swiCE2.switch.chswiCE2.switch.ch (Geneva, (Geneva,

Switzerland)Switzerland)– Backbone.acad.bgBackbone.acad.bg (Sofia, Bulgaria)(Sofia, Bulgaria)– Stockholm1.sunet.seStockholm1.sunet.se (Stockholm, Sweden)(Stockholm, Sweden)– Traceroute.teragen.com.auTraceroute.teragen.com.au (Melbourne, Australia)(Melbourne, Australia)– www.hafey.orgwww.hafey.org (Sydney, Australia)(Sydney, Australia)– www.megamirror.comwww.megamirror.com (Sydney, Australia)(Sydney, Australia)– www.debug.netwww.debug.net (Frankfurt, (Frankfurt,

Germany)Germany)

http://www.valkaryn.net/

http://www.slac.stanford.edu/

http://www.fh-friedberg.de/

http://www.lf.net/

http://www.hafey.org/

http://www.megamirror.com/

http://www.debug.net/


AS Mapping on Data Set I AS Mapping on Data Set I

• We mapped each IP to its We mapped each IP to its corresponding AS number by looking corresponding AS number by looking up RADB (Routing Assets Database) up RADB (Routing Assets Database) and its mirrored databasesand its mirrored databases

• For IPs that does not have a mapping For IPs that does not have a mapping entry: we checkentry: we check– If it has a hostname which identifies it as If it has a hostname which identifies it as

part of a site with a known ASpart of a site with a known AS– If no hostname found, we looked at the IP If no hostname found, we looked at the IP

range and hops before and after in the range and hops before and after in the same record, and make a best guess. same record, and make a best guess.


Preliminary Processing Preliminary Processing on Data Set Ion Data Set I

• We abstracted from Data Set I We abstracted from Data Set I the following records:the following records:

• One IP address appears in One IP address appears in multiple hops in the same recordmultiple hops in the same record

• A single hop in a record contains A single hop in a record contains multiple different IP addressmultiple different IP address


Route Change Dynamics -Route Change Dynamics -One IP appears in multiple One IP appears in multiple hops hops • Captured the middle of a routing changeCaptured the middle of a routing change

– E.g., based on the preceding and subsequent E.g., based on the preceding and subsequent records, we can tell that one specific IP appears in records, we can tell that one specific IP appears in two successive hops because one extra router has two successive hops because one extra router has just been added upstreamjust been added upstream

– Others more complicated, but usually a route Others more complicated, but usually a route change can be found. The duration of the dynamics change can be found. The duration of the dynamics varies from less than half an hour to several hoursvaries from less than half an hour to several hours

• Lost of connectivity to the repeating router’s Lost of connectivity to the repeating router’s following hop. (Last several hours?)following hop. (Last several hours?)

• Skipping: packets originated from Switch.ch Skipping: packets originated from Switch.ch (Swiss Education and Research Network) (Swiss Education and Research Network) seem to occasionally skip the first outgoing seem to occasionally skip the first outgoing router within this site. (reason unknown)router within this site. (reason unknown)


Route Change Dynamics -Route Change Dynamics -Multiple IPs in a single hopMultiple IPs in a single hop• 1194 instances1194 instances• 1 unique triple – Caused by route 1 unique triple – Caused by route

changechange• 20 unique router pairs, among them:20 unique router pairs, among them:

– 3 pairs (19 instances) due to Switch.ch 3 pairs (19 instances) due to Switch.ch skippingskipping

– 7 pairs (7 instances) during the middle of 7 pairs (7 instances) during the middle of route changesroute changes

– 2 pairs (2 instances) during a temporary 2 pairs (2 instances) during a temporary network outagenetwork outage

• The rest 8 pairs share the remaining The rest 8 pairs share the remaining 1165 instances1165 instances


Route Change Dynamics - Route Change Dynamics - Fluttering BehaviorFluttering Behavior• The rapidly-variable routing exhibited by a The rapidly-variable routing exhibited by a

few routers are called “Fluttering” by Vern few routers are called “Fluttering” by Vern Paxson in his 1997 thesisPaxson in his 1997 thesis

• This occurs on a very small time scale, This occurs on a very small time scale, essentially the time between successive essentially the time between successive traceroute probestraceroute probes

• One possible mechanism that could cause One possible mechanism that could cause this: A router alternates between multiple this: A router alternates between multiple next-hop routers in order to split load among next-hop routers in order to split load among the links to those routersthe links to those routers

• As pointed out by Vern, such behavior is As pointed out by Vern, such behavior is explicitly allowed in RFC 1812 explicitly allowed in RFC 1812 “Requirements for IP Version 4 Routers” “Requirements for IP Version 4 Routers” p.79. as load splitting, though the same p.79. as load splitting, though the same document cautions that there are situations document cautions that there are situations for which it is inappropriate. So it should at for which it is inappropriate. So it should at most be a configurable option for a routermost be a configurable option for a router


Ongoing WorkOngoing Work

• Statistical analysis ofStatistical analysis of– More routing pathologiesMore routing pathologies– Types of route changesTypes of route changes– Route change durationsRoute change durations– AS changesAS changes– TTL changesTTL changes– Route symmetryRoute symmetry

• Route change detection Route change detection algorithmsalgorithms


Main References:Main References:

• http://www.ietf.org/html.charters/nsis-chartehttp://www.ietf.org/html.charters/nsis-charter.htmlr.html

• H. Schulzrinne, R. Hancock, H. Schulzrinne, R. Hancock, GIMPS: General Internet Messaging Protocol fGIMPS: General Internet Messaging Protocol for Signaling or Signaling IETF Internet Draft, 2004IETF Internet Draft, 2004

• R. Hancock et. al., R. Hancock et. al., Next Steps in Signaling: Framework Next Steps in Signaling: Framework IETF IETF Internet Draft, 2003Internet Draft, 2003

• H. Schulzrinne et. al., Design of CASP – a Technology Independent Lightweight Signaling Protocol, 2003

• V. Paxson, Measurement and Analysis of V. Paxson, Measurement and Analysis of End-to-end Internet Dynamics, Ph.D. thesis, End-to-end Internet Dynamics, Ph.D. thesis, UC Berkeley, 1997UC Berkeley, 1997

http://www.ietf.org/html.charters/nsis-charter.html

http://www.ietf.org/html.charters/nsis-charter.html

http://www.ietf.org/internet-drafts/draft-ietf-nsis-ntlp-01.txt

http://www.ietf.org/internet-drafts/draft-ietf-nsis-ntlp-01.txt

http://www.ietf.org/internet-drafts/draft-ietf-nsis-fw-05.txt


The EndThe End

Documents

Charles Shen and Henning Schulzrinne Department of Computer Science Columbia University IRT Group 06/10/04 NSIS – an overview and its interaction with