Network Architectures

Network Architectures

OARTech

Paul Schopis

October 13, 2005

Topics

• TFN/OARnet background

• General MPLS Description

• What problem are we trying to solve anyway?

• Early Experiments at ITEC

• TFN implementation

OARnet Background

• Founded in 1987 as part of the Ohio Supercomputing Center

• 90+ higher ed member institutions

• Board of Regents funding

• OSTEER advisory council

• Internet2 GigaPOP

Third Frontier Network

• Phase 1: replace backbone with dark fiber

• Phase 2: connect 17 universities to network with dark fiber or gig circuits

• Phase 3: connect other universities and colleges

• Phase 4: connect other partners

Dark Fiber Acquisition

• RFP issued during Summer of 2002

• Dark fiber was strongly preferred, but leased services considered

• Vendors who bid dark fiber were required to offer a minimum of a single pair of fiber over their network


• Determined that leased lambdas were too expensive and not widely available

• Selected a bid from Spectrum Networks for single pair of fibers– American Electric Power (AEP)– Williams Communications (Wiltel)– American Fiber Systems

Spectrum

• We had various responses • The providers in the Spectrum offer bid

individually• No price increase for using Spectrum as

integrator • SBC and others reported no bid bit desired to

bid on future last mile – inter-lata issue


• $4.6 M for 20 year IRUs

• $342K/yr for maintenance

• 1600+ route miles

• Truewave, SMF-28, LEAF or Terra Light Fiber

• Aerial and buried

TFN Financing

• $21M investment• Financing from Ohio State University

– Loan for fiber ($7M)– Short-term financing ($2M)

• Financing from state capital budget ($8.5M) – Equipment– Last mile to 17 institutions

Community

• We desired to make this a true community owned network

• Committees with schools participating in decisions and recommendations

Equipment

• Cisco 15454 integrated solution (DWDM)– all of the amps, mux/demux etc. integrated

• Multi Service Transport Platform (MSTP) – ITU G.709 compliant

• Cisco routers (GSR 12000) and switches• Juniper M7i routers

Last Mile

• RFP issued in Dec 2003 for last-mile connectivity to all higher education and K-12 sites

• OC3, gig circuits and10 gig circuits

• We did make contact with local fiber providers on backbone bid ex. Buckeye Telesys

General MPLS Description

General MPLS DescriptionPacket have a 20 bit label that routes it along a “Label Switched Path”. Values range from 0 to 1,048,575.

0 through 15 are reserved for special uses.

Some label ranges have special meanings for specific vendors.

General MPLS Description•0 IPv4 Explicit Null Label - No label stacking, must POP label

•1 Router Alert Label - delivered to local router for local processing

•2 IPv6 Explicit Null Label - Same rule as IPv4 except forwarded to IPv6 routing instance.

•3 Implicit Null Label - Control protocol (LDP or RSVP) request for down stream router to POP Label


Choosing the next hop can be thought of as the composition of two functions. The first function partitions the entire set of possible packets into a set of"Forwarding Equivalence Classes (FECs)". The second maps each FEC to a next hop.

In many ways an IP prefix is a FEC

IP routing protocols are the mechanisms to map IP FECs to a next hop.


What are the advantages of MPLS?


MPLS forwarding can be done by switches which are capable of doing label lookup and replacement, but are either not capable of analyzing the network layer headers, or are not capable of analyzing the network layer headers at adequate speed.


Since a packet is assigned to a FEC when it enters the network,the ingress router may use, in determining the assignment, any information it has about the packet, even if that information cannot be gleaned from the network layer header. For example,packets arriving on different ports may be assigned to different FECs. Conventional forwarding, on the other hand,can only consider information which travels with the packet in the packet header.


A packet that enters the network at a particular router can be labeled differently than the same packet entering the network at a different router, and as a result forwarding decisions that depend on the ingress router can be easily made. This cannot be done with conventional forwarding, since the identity of a packet's ingress router does not travel with the packet.

General MPLS Description Sometimes it is desirable to force a packet to follow a particular route which is explicitly chosen at or before the time the packet enters the network, rather than being chosen by the normal dynamic routing algorithm as the packet travels through the network. This may be done as a matter of policy,or to support traffic engineering. In conventional forwarding,this requires the packet to carry an encoding of its route along with it ("source routing"). In MPLS, a label can be used to represent the route, so that the identity of the explicit route need not be carried with the packet.

General MPLS Description Some routers analyze a packet's network layer header not merely to choose the packet's next hop, but also to determine a packet's"precedence" or "class of service". They may then apply different discard thresholds or scheduling disciplines to different packets.MPLS allows (but does not require) the precedence or class of service to be fully or partially inferred from the label. In this case, one may say that the label represents the combination of a FEC and a precedence or class of service.

What problem are we trying to solve anyway?

The Problem

• Goal create an Abilene Premium Service• Need to create “Virtual Wire” ( Smells a lot

like a light path)• Need predictable bandwidth • Need to meet DiffServ EF requirements• Need to be able to signal request for

resources• Needed admission control

The Solution

• DiffServ Code Point• Queuing mechanisms High Priority• Policy on edge to mark and forward via

high priority queue• Admission control for LSP (MPLS

Tunnels) via marked packets that conformed to requirements

The Solution

• LSPs anchored to WRED Queues on WAN side

• All CPE side used High Priority • Tested across multiple BGP Domains • Tested QPPB for discovery of QoS

resources

The Solution

• Used RSVP to signal request for “sub-pool” reservation, e.g. guaranteed BW

• Resulted in primitives being incorporated into DSTE-MPLS

• Results used to write RFC 3270

AS 1

AS 3

AS 2

AS 3

AS 4

AS 1

AS 3

AS 2

AS 3

AS 4

AS 1

AS 3

AS 2

AS 3

AS 4

AS 1

AS 3

AS 2

AS 3

AS 4

TFN implementation

TFN implementation

• Needed to migrate to new network

• Needed to provide services such as multicast and IPv6

• Needed to solve fish problem

• Executed test plan based on Abilene test plan

Legacy POP Design

I1

I2

ATMI2

I1

ATM

I1&I2ATM

POP

Campus

Legacy POP Design

I1

I2

ATMI2

I1

ATM

I1&I2ATM

POP

Campus

BGP for Route diff

New OARnet Design Goals

• Reduce Costs

• Reduce Complexity

• Reduce Maintenance Fees

• Deliver Services

MPLS Requirements

• CPE device • PE Provider Edge • P Provider Core LSP Switching Router• We can collapse P and PE to one device• Need CPE for Label to IP binding• I1 will be standard routing• I2 will be Label Switched with BGP multihop to find correct

path• Must deliver advanced services to I2 community

– IPv6, Multicast, Jumbo Frames etc.

New Architecture

PE/P

CPE

GigE Aggregator

CPE

GigE

POP Campuses

New Architecture

PE/P

CPE

GigE Aggregator

CPE

GigE

Campuses

BGP MultihopLDP Exchange with CoreBGP Multihop

LDP Exchange with Core

MPLS for I2 Routes LFIBIP for I1 Routes FIB

Red = LDP tagged

AS3112

AS600

New Architecture

PE/P

LR 1

GigE Aggregator

CPE

GigE

CampusesAS3112

AS600

LR 2

Rate Cap Architecture

PE/P

CPE

GigE Aggregator

CPE

GigE

Campuses

Red = I2 CapBlue = Commodity CapGreen = Intra State Cap

AS3112

AS600

Some Implementation Issues

• Had to come up with more robust naming convention– Old ALP1, SWALP1

• Required DNS overhaul– Pseudo CILLY code

• CLMBN-R0, CLMBN-E0, CLMBN-O1, CLMBN-OT1

Questions?

Documents

Network Architectures