Integrated Control Plane Developments: CHEETAH, HOPI, OSCARS, USN

Nagi RaoOak Ridge National Laboratory, raons@ornl.gov

Malathi VeeraraghavanUniversity of Virginia, mv@ece.virgina.edu

Tom LehmanInformation Sciences Institute East, tlehman@isi.edu

Nasir GhaniTennessee Technological University, nghani@tntech.edu

Chin Guok ESnet, chin@es.net

ESCC/Internet2 Joint Techs WorkshopJul 16-19, 2006

Madison, Wisconsin

OutlineOutline

• Introduction to Control Planes• USN Update• CHEETAH, HOPI-DRAGON, OSCARS

• Multi Domain Resource Provisioning• A Framework• Integrated Path Computation• VLAN-Based Channel Alignment

Disparate Control Plane Technologies

We area at a critical point in control-plane technologies for advanced networks:– MPLS/GMPLS being enabled and/or deployed by vendors– Several control planes are being developed under different

paradigms:• ESnet-OSCARS, CHEETAH, HOPI-DRAGON, UCLP, USN,

GEANT, DANTE, and others– Underlying challenges and solutions are slowly being

understood: Standards are being discussed• security of control plane access• bandwidth optimization• data and control plane alignment• synchronization of schedulers and signaling• multi-domain authorization• others

• Solutions appear to be diverging, at least on the surface but they address different application domains

Control Plane Paradigms

Two main paradigms• On-Demand Provisioning:

– TL1,CLI: earlier host-level mechanisms primarily for manual configuration

– MPLS for layer 3, GMPLS for lower layers meant for automated configuration at network-level

• In-Advance Provisioning: Using a front-end for– Path computation and/or– Bandwidth optimizationfollowed by mostly in-time signaling

These paradigms address different application needs:– in-advance provisioned bandwidth to coordinate with reservations

on other facilities such as supercomputers and experimental facilities.

– on-demand provisioning to provide dedicated bandwidth

Some Control Plane Technologies

• HOPI-DRAGON – on-demand– GMPLS front end with CLI/TL1 for lower layers– Ethernet switches and routers

• OSCARS-ESnet – in-advance– Path computation + MPLS at layer 3– Juniper and Cisco routers

• USN – in-advance– Bandwidth optimization + CLI/TL1– Ciena CDCI + Force10 E300

• CHEETAH – on-demand– On-demand provisioning using GMPLS– Sycamore SN16000 switches

USN Data-Plane: Node ConfigurationIn the Core:

– Two OC192 switched by Ciena CDCIs

At the Edge– 10/1 GigE provisioning

using Force10 E300s

Data Plane User Connections:Direct connections to:

core switches –SONET &1GigEMSPP – Ethernet channels

Utilize UltraScience Net hosts

Node Configuration

CDCIe300

Linux host

10GigEGigE

OC192to Seattle10GigE

WAN PHY

Connections toCalTech and ESnet

USN Secure Control-PlaneVPN-based authentication,

encryption and firewall• NetScreen ns-50 at ORNL

NetScreen ns-5 at each node

• Centralized server at ORNL– Bandwidth scheduling– Signaling

ns-5

CDCIe300

linuxhostVPN tunnel

CDCIe300

linuxhost

Controlserver

ns-50

USN Control Plane• Phase I

– Centralized path computation for bandwidth optimization– TL1/CLI-based communication with CoreDirectors and E300s– User access via centralized web-based scheduler

• Phase II (currently being tested)– GMPLS wrappers for TL1/CLI– Inter-domain “secured” GMPLS-based interface– Webservices interface– X509 authentication for web server and service

Data Plane Data Plane OC192OC192

CHEETAH

CHEETAH Data and Control PlanesCHEETAH: Sycamore SN16000 data plane – OC192

GMPLS control-plane over IP tunnelsAdditional 1GigE L2VLANS between ORNL and Atlanta

25 U

25 U

SN16K

SN16K

SN16K

GMPLS supported onGMPLS supported onSecure control planeSecure control planeNsNs--5 terminated IP tunnels5 terminated IP tunnels

OSCARS: Guaranteed Bandwidth VC Service For SC Science

• ESnet On-demand Secured Circuits and Advanced Reservation System (OSCARS)

• In its current phase this effort is being funded as a research project by the Office of Science, Mathematical, Information, and Computational Sciences (MICS) Network R&D Program

• A prototype service has been deployed as a proof of concept– To date more then 20 accounts have been created for beta users,

collaborators, and developers– More then 100 reservation requests have been processed

Functional View of OSCARS• Support user/application VC reservation requests

Users enter reservations via a web-pageApplications uses API to send signed SOAP messages

• Manage allocations of scarce, shared resourcesCentralized resource managementAuthentication is done using X509 certificates

– Authorization TBD• Provide circuit setup and teardown mechanisms

OSPE-TE for routingBGP for reachabilityRSVP-TE for signalingMPLS for switching

• Enable the claiming of reservationsPolicy based filtering for traffic destined for VC

• Enforce usage limitsPer VC admission controlSeparate router queue for VCs

2005 2006 2007 2008

• Dedicated virtual circuits• Dynamic virtual circuit allocation

ESnet Virtual Circuit Service Roadmap

• Generalized MPLS (GMPLS)

• Dynamic provisioning of Multi-Protocol Label Switching (MPLS) circuits (Layer 3)

• Interoperability between VLANs and MPLS circuits(Layer 2 & 3)

• Interoperability between GMPLS circuits, VLANs, and MPLS circuits (Layer 1-3)

Initial production service

Full production service

DRAGONProject Features and Objectives

• Develop “Hybrid Network” Architecture/Infrastructure– One infrastructure which provides basic IP routed service as well services at lower layer

(i.e., lambdas, next generation SONET, ethernet)– Services could be point to point circuits or application specific layer2 multipoint

broadcast domains– Lower layer services geared toward special purposes such as high performance

application support or for traffic engineering of the IP network– Leverage emerging optical, next generation SONET, and ethernet technologies to provide

services a multiple layers• GMPLS based control plane (dynamic provisioning)

– Multi-domain, multi-layer– Network Aware Resource Broker (NARB) for interdomain routing and path computation– Include features for Authentication, Authorization, Accounting (AAA) and Scheduling as

part of network provisioning– Interface with higher level “Web Service” based “call control” systems

• Application Specific Topology Builder – Formalized means to describe and request an application topology (which may consist of

multiple individual provisioning actions)• All-Optical metro area network

– Eliminate OEO in the core, allow alien Waves, multi-degree ROADMs• Integration with real applications:

– E-VLBI (electronic Very Long Baseline Interferometry)– HD-CVAN (High Definition Video based services)

The DRAGON TestbedWashington, DC metro region

CLPK

ARLG

DCGW

MCLN

MIT Haystack Observatory(HAYS)U. S. Naval Observatory

(pending)

UMBC Goddard Space Flight Center(GSFC)

National Computational Science Alliance (NCSA)

ACCESS

DCNE

MAX

GIG-EF

HOPI

NREN

Abilene

CaveWave

Northrop Grumman

Venter

ISI East

DRAGON Control Plane• Network Aware Resource Broker – NARB

– Intradomain listener, Path Computation, Interdomain Routing– Hooks for incorporation of AAA and scheduling into path computation

via a “3 Dimensional Resource Computation Engine (3D RCE)”• Virtual Label Swapping Router – VLSR

– Open source protocols running on PC act as GMPLS network element(OSPF-TE, RSVP-TE)

– Control PCs participate in protocol exchanges and provisions covered switch according to protocol events (PATH setup, PATH tear down,state query, etc)

• End System Agent – ESA– End system or client software for signaling into network (UNI or peer

mode)• Application Specific Topology Builder – ASTB

– User Interface and processing which build topologies on behalf of users– Topologies are a user specific configuration of multiple LSPs

• Software and Documentation available:– http://dragon.east.isi.edu

DRAGON Control Plane

Web pageXML

Interface ASTB

CLI Interface One NARB per Domain

• GMPLS Adaptations to allow Ethernet VLAN based provisioning:– OSPF-TE advertises

(and updates) available VLAN Tag IDs

– Path Computation includes a constraint for available VLAN Tag

– RSVP-TE includes VLAN Tag information in signaling messages

The HOPI Node Architecture

DRAGON Control PlaneHOPI-DRAGON Interdomain

• VLSR (open source GMPLS protocols running on PC) turn ethernet switches into Label Switching Routers (LSRs)• NARB handles interdomain routing and path computation• Ethernet VLAN based “circuits” provisioned across domains

A General Control-Plane Architecture:Single Domain

Network Device Node #1

state and scheduling signaling

Resourcedatabase

Network Device Node #N

state signalstate signal

Centralized or

Distributed

Some modules might be trivial in some control planes

UserInterface

AAA

Securechannels

Integrated Multi-Domain Resource Provisioning – A Perspective

• Front-End website/webservice– Integrates all data bases; SOAP interface

• Meta-Scheduler– Path Composition and Alignment

• Computes multi-domain path; Queries domain schedulers

– May include host scheduling• Coordinated Signaling

– Sends path requests to domain signaling modules• Multi-Domain Authentication, Authorization and

AccountingA proposal is under consideration with DOE High-Performance Networking program on

this topic

Multi-Domain Resource Provisioning Architecture

Federatedstate signalingmeta

scheduler

Resourcedatabase

Reservedatabase

Network Domain #M

state signalschedulestate signalschedule

CentralizedO

r Distributed

AAA and secure channels are domain-specificUser authorization and credentials are domain-specific

Network Domain #1

Different paths may be computed:(i) A specified bandwidth in a specified time slot,

(ii) Earliest available time with a specified bandwidth and duration,

(iii) Highest available bandwidth in a specified time slot,

(iv) All available time slots with a specified bandwidth and duration.

All are computed by extending the shortest path algorithms using a closed semi-ring structure defined on sequences of real intervals(i)-(iii): Variation of Dijkstra’s shortest path algorithm

(iv): Variation of Bellman-Ford algorithm;

- previously solved using transitive-closure algorithm

USN Path Computation – Bandwidth Optimization

( ), , , 0, 1S ⊕ ⊗

Sequence of disjoint real intervals

Point-wise union

Point-wise intersection [ ]{ }1 1, , , ,p pl h l h⎡ ⎤⎣ ⎦

{ }R+

{ }R+

Algorithm ALL-SLOTS 1. ( ) { }sτ ← ℜ ; 2. ( ) { }vτ ← ∅ for all v s≠ ; 3. for 1,2, , 1k n= −… do 4. for each edge ( , )e v w= do 5. ( ) ( ) { ( ) }ew w v Lτ τ τ← ⊕ ⊗ ; 6. return ( ( ))dτ .

Given network with band width allocations on all links

ALL-SLOTS returns all possible starting times for a connection with bandwidth b duration t between source node s and destination node d

Modified Bell-Ford algorithm: Time-complexity:

More efficient than transitive-closure algorithm:

All-Slots Algorithm

( )O mn

( )3O n

VLAN VLAN –– Unifying DataUnifying Data--Plane TechnologyPlane Technology

• IP networks– VLANs Implemented in MPLS tunnels

(L2VPN)• Circuit switched networks

– VLANs Implemented on top of Ethernet channels

• Align IP and circuit connections at VLAN level

Circuit SwitchedIP network

MPLS tunnel Layer-2 connection

Alignment of VLANs

VLAN overMPLS

VLANOver Ehthernet

UltraScienceNet

CHEETAH

USN–CHEETAH VLAN AlignmentUltraScienceUltraScience Net: LayerNet: Layer--2 2

VLAN: E300 VLAN: E300 –– CDCI CDCI -- … … -- CDCI CDCI –– E300E300CHEETAH: layerCHEETAH: layer--3 + layer 23 + layer 2

VLAN: T640VLAN: T640--T640 T640 –– SN1600SN1600 –– Cisco 3750Cisco 3750

USN L2 VLANUSN L2 VLAN

CHEETAH L2MPLS+L2 VLANCHEETAH L2MPLS+L2 VLAN

Measurements: VLANs over L1-L3 connectionsTwo types of measurements:

Ping (standard version), and TCP roundtrip time of 1K packetsThree channels:1. USN: host-SONET-host – 1000 miles; OC-21

2. CHEETAH/ORNL: host-MPLS-host – 300 miles; 1Gig VLAN

3. USN-CHEETAH: host-SONET-MPLS-host; 1300 miles 1Gig VLAN

Comparison Method for jitter and delay dynamics:normalization of wide-area measurements based on mean delay

E300switch

hosthostE300switch

CDCIswitch

CDCIswitch

1000 miles1000 miles

T640router

T640T640routerrouter

T640router

T640T640routerrouter

hosthost hosthost

ORNL Atlanta-SOX1Gig MPLS tunnel

300 miles

Normalized Comparison of VLANs:SONET - SONET-MPLS composed – L2MPLS

Ping measurements normalized for comparison:

L2MPLSL2MPLSSONETSONET--MPLSMPLScompositecomposite

mean time=9.384557msstd_dev (%)= 3.281692

mean time=35.981812msstd_dev (%)= 0.151493

SONET channelshave smaller jitter levels

SONETSONET

mean time=26.845877msstd_dev (%)= 0.187035

Components of Multi-Domain Control-Plane

• Security: AAA + Encryption System– User/Client authentication over multiple domains

• Path Optimization: Meta-Scheduler– Composite graph

• Nodes: peering points• Edge: in-domain connections or resources

– Path computation and composition• Issue individual domain path requests• Agglomerate domain paths

• Integrated Signaling System– Issue reservation commands– Exception handling

Domain Technical Tasks• USN-CHEETAH

– Webservice interface– Integrated VLAN specification

• OSCARS– VLANs over MPLS tunnels– Provisioning to Chicago and Sunnyvale routers

• HOPI-DRAGON– Webservice interface– Advanced Scheduling– Peering in Chicago and Seattle

An Integrated Scheduler for Hybrid NetworksAn Integrated Scheduler for Hybrid Networks• MPLS tunnel over IP network• Layers 1-2 over switched networksApproach

– Higher level scheduler composes end-to-end VLAN consisting of portions of MPLS tunnels and layer 1-2 channels

– Extensions of Dijkstra and Bellman-Ford Algorithms

– Coordinated daemons will be launched to setup the needed portions of path

Layer 3connection

Layer 3connection

Layer 2/1circuit

On-demandGMPLS MPLSMPLSOSCARS USN+(TL1,CLI) OSCARS In-advance

Conclusions

• Coming Together of Control-Plane Technologies– Different application domains are optimally supported by

different control-planes– Multiple-domain paths are needed for several applications– Several new technologies exist but more are needed, and

they all must be integrated• Hybrid Integrated Architecture

– Networks that combine best of shared IP and dedicated channels are feasible but need further development

– Data-planes must be aligned and peered in addition to control planes

• Peering Multiple Control-Planes is Complex– Synchronization of schedulers– Multi-domain path composition– Coordinated signaling– Multi-Domain AAA