Deploying SDN on ASR 9000d2zmdbbm9feqrf.cloudfront.net/2015/eur/pdf/BRKSPG-2722.pdf192.4/31 192.6/31 128.2/31 128.0/31 192.0/31 AG04-4 10.99.8.8/32 AG04-2 10.99.2.2/32 AG06-3 10.99.5.5

Deploying SDN on ASR 9000

BRKSPG-2722

Joel Roberts, Technical Leader, Advanced Services [email protected]

© 2014 Cisco and/or its affiliates. All rights reserved.BRKSPG-2722 Cisco Public

Agenda

• SDN Drivers, Concepts, & Definitions

• ASR9000 SDN Enabled Platform

– BGP-LS

– Stateful PCEP

– OpenFlow

– NETCONF/YANG

• Summary

3

Drivers, Concepts, & Definitions


There Are Many Reasons to Change

Industry Realignment

Significant Traffic Growth

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

2012 2013 2014 2015 2016 2017

Ex

ab

yte

s p

er

Mo

nth

Web/Data (24.2%, 18.9%)

File Sharing (15.7%, 8.1%)

Managed IP Video (21.8%, 21.0%)

Internet Video (38.3%, 52.0%)

50 Billion Connected Things

by 2020

Connected Things Growing 5X Faster than

Mobile Devices

More than 22% of all

networked events will be

Machine Driven by 2017

Emergence of Internet of Everything

Acquires 45%

of Verizon

Wireless from

+

+

+

+

+

+

5

Evolution & Intersections of Technology

Video

CloudMobility

M2M


There Are Many Key Challenges

Traffic GrowthComplexity

Competition

Total Cost Ownership Speed of Innovation

6


Video

M2M

Cloud

MobilityEPN

Evolved Programmable Network (EPN)

NCS NCS

APIs

APIs

EDGECORE

UA

VM VM

Edge

Core

VM

Orchestration

VM / Storage Control

Service CatalogService Orchestration Apps

VM

Application

CDN

AGILITY

OPTIMIZE

MONETIZE ¥£€$

Always “ON”

On-DemandServices Anywhere

Dynamic Scale

ApplicationInteraction

SeamlessExperience

Policy

Real-Time Analytics

Services Orchestration

Fully Virtualized

IntelligentConvergence

Automated

Open and Programmable

Unified Access

7


SDN: Means Many Things to Many People

Practical Attributes:

• Centralized platform with global view of the network (Definitely NOT NEW)

• Offers northbound APIs for Application interaction with the network

• Employs many different methods to collect and program information from/to network

8

Optical

IP/MPLS

NB API

Centralized Platform with

Global View

Apps


Today’s Infrastructure: Many Silos, Complex…

Data

Center

Customers

9

Wide Area Network Cloud

Optical

IP/MPLS

Software Engineering (Applications)

System Engineering

(Compute, Storage, Virtual

Machines)

Network Engineering

(WAN Services, NfV)Enterprise

Virtual


Evolved Programmable Network

Data

Center

Customers

10


Optical

IP/MPLS


System Engineering


Machines)

Network Engineering


Virtual

Physical and Virtual

API’s

Orchestration

API’s

Applications


SDN: Place In Network

Data

Center

Access/Customers

11


Optical

IP/MPLS

DC/Cloud

SDN

Enterprise

SDN

Applications DC-WAN X-Domain OrchEnt-SP WAN X-Domain Orch

SP SDN

(e.g WAN Orchestration)


Scope

12

Applications

Orchestration

API’s + Protocols


Service Provider “Hybrid” Control Plane

• Distributed Components –Functions tightly coupled to data plane

• Centralized Components –Functions where a holistic/abstracted view is required

• Existing distributed control plane ->Augmented by centralized control plane function

Traditional Control Plane

Architecture

(Distributed)

SDN Control Plane Architecture

(Centralized)

Collaborative Control Plane

Architecture

Application

Distributed Control Plane

Data Plane

Centralized Control Plane

APIs

13


Choice of Programmable Layer

Network

Orchestration Management

Device

Device Plug-Ins

Controller

Service ManagementBusiness

ApplicationsIT Software Infra

Infrastructure S/WAnalytics

BI Collaboration ERM

Policy & Compliance

abstract

detail

14


There are Many Options for Programmability

• PCEP

• BGP-LS

• OpenFlow

• Netconf

• Yang

• BGP-FlowSpec

• ReST

• Puppet

• Chef

• SNMP

• NetFlow

• CLI

• Syslog

• Others...

15


Programmability Interfaces - Segmentation

Configure

Operate

Device

Extension

DevOps

Integration

NETCONF

YANG

BGP-LS

PCEPOpFlex

Cisco

Python API

BGP

Flowspec

Cisco

Python API

16


Examples of Use-Cases“Leveraging APIs and logically centralized control plane components”

Automation of

Network Control

and Configuration(Fulfillment and Assurance)

Virtual & Physical

Custom Routing (based on business logic)

Online Traffic Engineering

Consistent Policy for Network,

Security, & Threat Mitigation

Custom Traffic Processing

(Analytics, Security)

Virtualization and Customer Domain Isolation

(Device/Appliance/Network)

17


OrchestrationRESTful APIs

ProgrammingCollection

Use-Case: Bandwidth Scheduling (On-Demand)

WAN

R1

R2

R3Data Center #1 Data Center #2

5

1

2 3

4

Congested!!

Challenge/Opportunity:

Customer has “on demand”

need for a DC backup

Network conditions reported

to collector (continuous)1

Customer requests DC1 to

DC2 bandwidth NOW2

Demand admission request:

<R1-R3, B/W, NOW>3

Orchestration returns option

and customer confirms4

R1-R3 LSP tunnel

programmed 5




Use-Case: Bandwidth Calendaring

WAN

R1

R2


5

1

2 3

4

Congested!!

Customer uses self-service portal to request and schedule bandwidth between DC’s


to collector (continuous)1


DC2 bandwidth at future date2

Demand admission request:

<R1-R3, B/W, Future Date>3

Orchestration returns

confirmation of date/time4

R1-R3 LSP tunnel

programmed 5

ASR 9000 SDN Enabled Platform


ASR 9001 ASR9904 ASR 9006 ASR 9010 ASR9912 ASR 9922

Size 2RU 6RU 10RU 21RU 30RU 44RU

# of I/O Slots 2 MPAs 2 4 8 10 20

Air Flow Side to side Side to side Side to back Front to back Front to back Front to back

Fabric BW/slot N/A 770G/385G 440G/220G 440G/220G 770G/660G 770G/660G

Chassis

scalability/slot120Gbps 2Tbps 880Gbps 880Gbps 2Tbps 2Tbps

Total Chassis BW 120Gbps 8Gbps 3.5 Tbps 7.0 Tbps 20Tbps 40Tbps

Cisco ASR 9000 Portfolio EvolutionNew

New

ASR 9001-S

New

21


ASR9K SDN Enabled Platform & Strategy

Controller

OpenFlow

OpenFlow Agent

Applications

Policy Servers OSS/BSS User AppAnalytics

Orchestration

Control PlaneManagement

Plane

Controller

Controllers3

Program Policies for Optimized Experience

BGP-LSAPIs and Protocols

OpenFlow Agent ,

BGP-LS, PCEP,

NETCONF

2

SDN Enabled Data

Plane

1

E-PBR Data Plane

Policy Fwd Flows

ASR 9000

SDN Platforms

Orchestration,

Analytics, Services, etc

4

VSM

Harvest

Network

Intelligence

22

Controller

PCEP

Applications5


Various Tools for Testing & Evaluation

OpenDaylight (ODL)

https://wiki.opendaylight.org/view/Getting_started

http://www.opendaylight.org/software/downloads

REST Client:

Chrome: Postman REST Client

Firefox: REST Client

Wireshark: https://www.wireshark.org/

IOS XR 5.2.2 with physical routers and

Virtual Internet Routing Lab (VIRL): virl.cisco.com

23

For YourReference

https://wiki.opendaylight.org/view/Getting_started

ASR9000 BGP Link-State (BGP-LS)


Network Discovery & Data Collection

• Traditional

– Methods: SNMP, SNMPCollect (polling), Netflow, CLI Parsing

– Platforms: Network Management Systems (NMS)

– Are and will continue to be used…

• Some Challenges:

– Delay in topology changes to NMS • E.g. delay based on polling intervals

– Unreliable transport• e.g. missed SNMP traps

• Must Augment with (near) real-time information about the state of the network


Overview BGP Link-State (BGP-LS)

• Use BGP to advertise LSDB and TED of a network

– New link-state address family

• Support for OSPF and ISIS LSDB

• Advantages

– Single upstream topology feed (BGP)

– IGP isolated from external entities

– Leverage well-known BGP security,transport and policy knobs

– Enables operator control

• Support introduced in IOS XR 5.1.1

• Implementations: WAN Automation Engine (WAE), ODL

26

Domain 1 Domain 2

Domain 0

BGP-LS

BGP-LS BGP-LS

RR

PCE

TED

LSP DB


Reference Diagram: BGP-LS

27

OpenDaylight

192.168.96.0/29

.1

.2

192.168.x.x/3164.4/31

64.0/31

192.4/31 192.6/31

128.2/31

128.0/31

192.0/31

AG04-4

10.99.8.8/32

AG04-2

10.99.2.2/32

AG06-3

10.99.5.5

AG06-1

10.99.3.3

AG06-2

10.99.4.4

AG06-4

10.99.6.6

AG04-1

10.99.1.1

172.18.152.3/24

For YourReference

BGP-LS

REST


router ospf 1distribute bgp-ls instance-id 1router-id 10.99.8.8area 0

router bgp 65000bgp router-id 10.99.8.8address-family link-state link-state!neighbor 192.168.96.2remote-as 65000update-source Loopback0address-family link-state link-state!!

ASR9000 BGP Link State Configuration

Distribute link state

database into BGP-LS

Specify BGP-LS peer

28

Enable address-family

link-state


BGP Link State Prefixes

• BGP-LS prefix string has the following general format

[NLRI-Type][Area][Protocol-ID][Local node descriptor][Remote node descriptor][Attributes]/prefix-length

• Node descriptors and attributes consists of potentially multiple TLVs

• Node descriptors and attributes are shown as

[X[TLV1][TLV2]…]

– Where X identifies object (e.g. local node, remote node, link, etc.)

• TLVs are shown in the format

[yVALUE]

– Where y identifies field type (e.g. AS number, interface address, etc.)

29

For YourReference


RP/0/RSP0/CPU0:ASR9K-4#sh bgp link-state link-state Tue May 6 00:09:45.523 UTCBGP router identifier 10.99.8.8, local AS number 100Status codes: s suppressed, d damped, h history, * valid, > best

i - internal, r RIB-failure, S stale, N Nexthop-discardOrigin codes: i - IGP, e - EGP, ? - incompletePrefix codes: E link, V node, T IP reacheable route, u/U unknown

I Identifier, N local node, R remote node, L link, P prefixL1/L2 ISIS level-1/level-2, O OSPF, D direct, S statica area-ID, l link-ID, t topology-ID, s ISO-ID,c confed-ID/ASN, b bgp-identifier, r router-ID,i if-address, n nbr-address, o OSPF Route-type, p IP-prefixd designated router address

Network Next Hop Metric LocPrf Weight Path*> [V][O][I0x1][N[c100][b10.99.8.8][a0.0.0.0][r10.99.1.1]]/376

0.0.0.0 0 i *> [E][O][I0x1][N[c100][b10.99.8.8][a0.0.0.0][r10.99.1.1]][R[c100][b10.99.8.8][a0.0.0.0][r10.99.2.2]][L[i192.168.192.2][n192.168.192.3]]/792

0.0.0.0 0 i

ASR9000 BGP Link State Verification

Prefix codes

Node

Link

30


BGP-LS *OpenDaylight Sample Configuration

<type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:bgp:rib:impl">prefix:rib-impl</type>

<name>example-bgp-rib</name>

<rib-id>example-bgp-rib</rib-id>

<local-as>65000</local-as>

<bgp-id>192.168.96.2</bgp-id>

<bgp>

<type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:bgp:listener">prefix:listener</type>

<name>example-bgp-peer</name>

<host>10.99.8.8</host>

</bgp>

31

* Approximately Lines 55 to 67 of <install-path>/etc/opendaylight/karaf/41-bgp-sample.xml

BGP-LS peer (ASR9000)

OpenDaylight Controller


RESTCONF* Sample OpenDaylight BGP LS Topology

32

{ "topology": [{ "link": [

{ "link-id": "bgpls://Ospf:1/type=link&local-as=100&local-domain=174262280&local-area=0&local-router=174261509&remote-as=100&remote-domain=174262280&remote-area=0&remote-router=174261766&ipv4-iface=192.168.128.0&ipv4-neigh=192.168.128.1",

"l3-unicast-igp-topology:igp-link-attributes": {"ospf-topology:ospf-link-attributes": {

"ospf-topology:ted": {} },"l3-unicast-igp-topology:metric": 10 },

"destination": {"dest-node":

"bgpls://Ospf:1/type=node&as=100&domain=174262280&area=0&router=174261766","dest-tp": "bgpls://Ospf:1/type=tp&ipv4=192.168.128.1”},

"source": {"source-tp": "bgpls://Ospf:1/type=tp&ipv4=192.168.128.0","source-node":

"bgpls://Ospf:1/type=node&as=100&domain=174262280&area=0&router=174261509”}

* http://localhost:8080/restconf/operational/network-topology:network-topology/topology/example-linkstate-topology


BGP-LS Sample* Topology Visualization

33

*Visit DevNet for Sample BGP-LS Application

ASR9000 Path Computation Element Protocol


Path Computational Element (PCE)

• Path computation in large, multi-domain, and multi-layer networks

• Can reside inside router or on an external server platform

• Setup intra-, inter-area (or domain) and inter-layer paths

• Computed Path might be:

– Explicit route identifying a contiguous set of strict hops between the source and destination

– Combination of strict/loose hops between the source and destination

Basic function is to compute paths (e.g. MPLS TE LSPs)

in MPLS/GMPLS Networks

35


PCE Definitions

• Traffic Engineering Database (TED)

– Contains topology and resource information

– Inputs from IGP LSDB

• PCE Server (PCE)

• Path Computation Client (PCC)

– Agent on router(s) that interact withPCE Server

• PCE Protocol (PCEP)

– Protocol that runs between PCC on router and PCE server

36


Stateless and Stateful PCE

• Stateless

– PCE has no knowledge of previously established paths

• Stateful

– Synchronization between PCCs and PCEs

– PCC maintains state synchronization with PCE

– PCC may delegate LSP control to PCE

• Either PCE or PCC can initiate LSP setup

• PCC always owns LSP state

PCEP

Stateful PCE

TED

LSP DB

PCC

PCE-initiated

LSP

37


Stateful PCE

• LSP Database

– Contains info/status on active LSPscommunicated by PCCs in LSP state report messages

• Active Stateful PCE

– References LSP DB for path computations

– Programs LSP state in network

• PCC reports LSP status to PCE

• PCE may reprogram an existing LSP

• LSP Delegation

– PCC delegates LSP control responsibility to PCE

38

LSP=Label Switch Path


ASR9000 PCE-Initiated Tunnels

• Treated as dynamically created tunnels (auto-tunnel)

• Router does NOT verify or compute path that PCE provides

– treated as verbatim path

• PCE responsible for LSP re-optimization

• PCE sends an PCEP Update when a better path exists

• Tunnels may be inter-area

• *PCE Initiated tunnel does not appear in configuration

– “show mpls traffic-eng tunnels”

39

PCEP

Initiate /

Create

Stateful PCE

TED

LSP DB

PCC


Reference Diagram: PCE Initiated LSP

40

OpenDaylight

192.168.96.0/29

.1

.2

192.168.x.x/3164.4/31

64.0/31

192.4/31 192.6/31

128.2/31

128.0/31

192.0/31

AG04-4

10.99.8.8/32

AG04-2

10.99.2.2/32

AG06-3

10.99.5.5

AG06-1

10.99.3.3

AG06-2

10.99.4.4

AG06-4

10.99.6.6

AG04-1

10.99.1.1

172.18.152.3/24

For YourReference

PCEP

REST


ASR9000 PCC Configuration

hostname ASR9K-AG04-4!ipv4 unnumbered mpls traffic-eng Loopback0!mpls traffic-engpcepeer ipv4 192.168.96.2stateful-clientinstantiation

!auto-tunnel pcctunnel-id min 7000 max 9999

!

Allow PCE-

initiated LSP

User defined

tunnel number

range.

Required for

auto-tunnel

PCE server

41


ASR9000 Verification PCE Peer

RP/0/RSP0/CPU0:ASR9K-4#show mpls traffic-eng pce peer Tue May 6 03:21:17.216 UTC

Address State Learned From--------------- ------------ --------------------

192.168.96.2 Up Static config

42


PCE Add-LSP (OpenDaylight) Sample

43

• REST URL: http://localhost:8080/restconf/operations/network-topology-pcep:add-lsp

• Method: POST Content-Type:application/xml

<input>

<node>pcc://10.99.8.8</node>

<name>test-tunnel</name>

<arguments>

<endpoints-obj><ipv4>

<source-ipv4-address>10.99.8.8</source-ipv4-address>

<destination-ipv4-address>10.99.6.6</destination-ipv4-address>

</ipv4></endpoints-obj>

</arguments>

<network-topology-ref xmlns:topo="urn:TBD:params:xml:ns:yang:network-topology">/topo:network-topology/topo:topology[topo:topology-id="pcep-topology"]</network-topology-ref>

</input>

PCC

Source

Destination

http://localhost:8080/restconf/operations/network-topology-pcep:add-lsp


ASR9000 Verification PCE Tunnel

RP/0/RSP0/CPU0:ASR9K-AG04-4#show mpls traffic-eng pce tunnels Mon May 19 17:36:20.237 UTCTunnel : tunnel-te7004

Destination : 10.99.6.6State : down

44


PCE Update LSP (OpenDaylight) Sample

45

• URL: http://localhost:8080/restconf/operations/network-topology-pcep:update-lsp

• Method: POST Content-Type:application/xml<input>

<node>pcc://10.99.8.8</node> <name>test-tunnel</name>


<arguments>

<operational xmlns:stateful02="urn:opendaylight:params:xml:ns:yang:pcep:crabbe:stateful:02">true</operational>

<ero>

<subobject><loose>false</loose><ip-prefix><ip-prefix>192.168.64.4/32</ip-prefix></ip-prefix></subobject>




</ero>

</arguments>

</input> Explicit-Route Object (ERO)



RP/0/RSP0/CPU0:ASR9K-AG04-4#show mpls traffic-eng pce tunnels Mon May 19 17:39:38.550 UTCTunnel : tunnel-te7004

Destination : 10.99.6.6State : upCurrent path option : 10

46


PCE Remove LSP (OpenDaylight) Sample

47

• URL: http://localhost:8080/restconf/operations/network-topology-pcep:remove-lsp

• Method: POST Content-Type:application/xml

<input>

<node>pcc://10.99.8.8</node>

<name>test-tunnel</name>


</input>

http://localhost:8080/restconf/operations/network-topology-pcep:remove-lsp



RP/0/RSP0/CPU0:ASR9K-AG04-4#sh mpls traffic-eng pce tunnels Mon May 19 17:43:35.610 UTCNo PCE tunnels.

48


ASR9000 Show Commands

• New commands

– show mpls traffic-eng auto-tunnel pcc […]

• Modified Commands

– show mpls traffic-eng pce peer [ all|ipv4 <addr>|node-id <id>|stateful|stateless]

• Other useful commands

– show mpls traffic-eng pce tunnels <id>

– show mpls traffic-eng pce trace […]

– show mpls traffic-eng tunnels tabular

49


Traffic Steering into PCE-Initiated Tunnels

• Example approaches:

• Autoroute announce

• Policy-based tunnel selection (forwarding class id)

50


Policy Based Tunnel Selection

• Local mechanism at head-end

• PBR policy sets forwarding class for incoming traffic

• Traffic switched to tunnel with matching forwarding class

• Seven forwarding classes supported (1-7)

• One forwarding class reserved as default (0)

51

PE1

PE2

Tunnel-te1000

Forwarding class 1

Tunnel-te2000

Forwarding class 0

(default)

Input policy matches

traffic based on ACL

and sets forwarding

class


Traffic Steering PCE Initiated Tunnel

Tunnel-te1000

Forwarding

class 1

Tunnel-te2000

Forwarding

class 0 (default)

52

Tunnel-te7000

PCE Initiated LSP

Forwarding class 2

ASR9K-1

CETenGigE0/1/0/1

10.201.7.43

ASR9K-2

ASR9K-3

Set Forward ClassID 2



PCEPProgramming

Use-Case: PCE Initiated LSP

WAN

R1

R2


4

3

Service Request1

Analytic to determine LSP2

PCCreate LSP3

Traffic steering onto LSP6

10.201.7.43

2

6

TE LSP Signaling4

Delegate & LSP State Report5

5

ASR9000: OpenFlow


OpenFlow (OF)

• Open Network Foundation (ONF)

• Layer 2 Communication Protocol:

– Access to forwarding plane

– Switch or Router

• OF Controller: Control Plane

• OF Agent (device): Forwarding Plane

55


OpenFlow Agent/Plugin

• Implements standard OpenFlowswitch model.

• Speaks ‘standard’ OpenFlow protocol

• Native dedicated CLI for troubleshooting

56

Availability: Universal

User Experience: Consistent

Deployment:

End-to-end

• Available* IOS-XR, IOS and IOS-XE, NX-OS

• Common code, base features and CLI across platforms

• Across SP, Data Center, Campus…

* Please check roadmap for details on supported platforms & timelines


OpenFlow

Sends instructions…

OpenFlow Controller

57

Table

Match Action

Table:

– Consists of a set of flows (each row is a flow)

– Each flow contains a set of matches and actions.

– Applied only in ingress direction


OpenFlow Table

Match:

• Ingress port

• Priority: Highest priority flow entry that matches packet gets selected

– Lowest priority is zero and highest is 32768

Action:

• Packet forwarding actions

• Packet modification actions

• Actions are immediately applied to the packet

58


OpenFlow: ASR9000

• Support starting IOS XR 5.1.2

– OpenFlow Switch Version 1.3

• IOS XR Images

– asr9k-mini-px.vm

– asr9k-mpls-px.pie (required for L3VPN, L2VPN)

– asr9k-k9sec-px.pie (required for TLS)

• Default port TCP 6653

• Any physical interface for OF controller

59

OpenFlow Agent

OpenFlow Controller

Control PlaneManagement

Plane

E-PBR Data Plane

Policy Fwd Flows

ASR9K


ASR9000: OpenFlow Hybrid Mode

• Recommended

• OpenFlow and XR forwarding

• OpenFlow as a policy based routing (PBR) type of feature.• Can provide ‘fine-tuning’ on incoming traffic based on OpenFlow rule matching and rule actions

• Ability to take advantages of both XR features and OpenFlow flexibility

60


ASR9000 OpenFlow Layer 2 Table Options


OpenFlow Controller

61

L2 Only Table

Match- L2 Header Action: L2 Only

L2_L3 Table

Match: L2 & L3 Hdr Action: L2 Only

Supported L2 Matches & Actions are detailed in

ASR9000 System Management Configuration Guide, Release 5.2.x


ASR9000 OpenFlow Layer 3 Table Options


OpenFlow Controller

62

L3_V4 Table

Match: L3 IPv4 Hdr Action: L3 Only

L3_DS Table

Match:L3 v4 & v6 Action: L3 Only

Supported L3 Matches & Actions are detailed in



ASR9000 OpenFlow Tables

63

Layer Table Type Pipeline Supported

Interfaces

Description

Layer 2

L2 129 Bridge-domain, Gigabit ethernet, Bundle,

Bundle-subinterfaces, PWHE-

subinterfaces

• Supports L2 header matches

• Supports L2 actions.

• Can be applied to ingress L2 interface

L2_L3 130 L2Bridge-domain, Gigabit ethernet,

Bundle, Bundle-subinterfaces, PWHE-

subinterfaces

• Supports L2 & L3 (IPv4/IPv6) header

matches

• Supports L2 actions

• Can be applied ingress L2 interface

Layer 3

L3_V4 131 VRF and global interfaces, BVI (IPv4

only), Bridge-domain, Gigabit ethernet,

Bundle, Bundle-subinterfaces

• Supports L3 (IPv4) header matches

• Supports L3 (IPv4) actions

• Can be applied to ingress L3

interfaces

L3_DS 132 VRF and global interfaces, BVI (IPv4

only), Bridge-domain, Gigabit ethernet,

Bundle, Bundle-subinterfaces

• Supports L2 & L3 (IPv4/IPv6) header

matches

• Supports L3 (IPv4/IPv6) actions

• Can be applied to ingress L3

interfaces

For YourReference


OpenFlow Cisco Extension Actions

• Additional OpenFlow features on ASR9000

64

Cisco Added Actions

OpenFlow Switch Types

supported on ASR9000

Applied L2 Bridge

Domain

Applied L3 or L3 VRF

Interface

Action Description L2

Only

L2_L3 L3_V4 L3_DS

Set IPv4 Next Hop Set IPv4 Nexthop

Address

No No Yes Yes

Set IPv6 Next Hop Set IPv6 Nexthop

Address

No No No Yes

Set Forward Class ID Set Forward Class ID No No Yes Yes

Netflow Enable/Disable Enable/Disable Netflow No No Yes Yes


Openflow for Traffic Steering

• Use Openflow to program classifiers/actions on WAN Edge

• Flow entries would be something like:

– MATCH/Set_IPv4_Next_Hop

– MATCH/Set_IPv6_Next_Hop

– MATCH/Set_FCID (tunnel identifier)

• Useful for services and applications requiring steering of specific flows into a programmed WAN resource

65

Head-End

PCEP

Create LSP Tunnel

WANEdge Routers

Flow Entries

<MATCH/”Forward to

Tunnel”>

Tunnel

Traffic Steering

App

SDN WAN Orchestration

OF PCEP

OF

Flows



ProgrammingOpenFlow

Use-Case: OpenFlow for Traffic Steering

WAN

R1

R2


3

3

Service Request1 Analytic to determine LSP

2

Create LSP Tunnel with

FCID 23

Use OpenFlow to program

match/action on WAN edge4

L3_DS Table

Match: SRC 10.201.7.43 Action: Set FCID=2

10.201.7.43

2

4


interface TenGigE0/0/0/3l2transport!l2vpnbridge group SDN-1bridge-domain OF-1interface TenGigE0/0/0/3

!openflowswitch 7 pipeline 129tls trust-point local openfCA remote openfCAbridge-group SDN-1 bridge-domain OF-1controller ipv4 172.16.1.45 port 6653 security tls!!

ASR9000 OpenFlow L2 Switch Configuration

Openflow Switch attached to

bridge-domain

OpenFlow Controller

67

L2 Only Switch


interface TenGigE0/0/0/3l2transport!l2vpnbridge group SDN-2bridge-domain OF-2interface TenGigE0/0/0/4

!openflowswitch 9 pipeline 130tls trust-point local openfCA remote openfCAbridge-group SDN-1 bridge-domain OF-1controller ipv4 172.16.1.45 port 6653 security tls!!

ASR9000 OpenFlow L2 + L3 Switch Configuration

Openflow Switch attached to

bridge-domain

OpenFlow Controller

L2 + L3 Switch

68


openflow switch 11 pipeline 131interface Bundle-Ether2.1interface GigabitEthernet0/1/0/7interface GigabitEthernet0/0/0/0/4.1301controller ipv4 172.16.1.45 port 6653 security tls

openflowswitch 10 pipeline 131vrf ONEcontroller ipv4 172.16.1.45 port 6653 security tls

ASR9000 OpenFlow L3_V4 Switch Configuration

All interfaces in VRF become

part of OpenFlow Switch

Specify L3

Interfaces

69

L3_V4 Switch

L3_V4 Switch


openflow switch 100 netflowflow monitor mmap sample smapinterface Bundle-Ether3interface GigabitEthernet0/0/0/0/4.1201controller ipv4 172.16.1.45 port 6653 security tls

ASR9000 OpenFlow Netflow Configuration

• Cisco extension to OpenFlow

• Used to enable/disable NETFLOW on Layer 3 Interfaces

70


Sample ASR9000 OpenFlow Show Commands

71

RP/0/RSP0/CPU0:ASR9K-AG06-4#show openflow switch 102 controllerLogical Switch Id: 102

Controller: 1 Address : 192.168.96.3:6653Protocol : tcpVRF : defaultLocal Trustpoint: : Not availableRemote Trustpoint: : Not availableConnected : YesRole : Masterlast_error : Invalid argumentstate : ACTIVEsec_since_connect : 431579

RP/0/RSP0/CPU0:ASR9K-AG06-4#show tcp briefPCB VRF-ID Recv-Q Send-Q Local Address Foreign Address State

0x501e3690 0x60000000 0 0 10.99.6.6:58633 192.168.96.3:6653 ESTAB0x501c8720 0x60000000 0 0 10.99.6.6:38212 10.99.8.8:179 ESTAB


Sample ASR9000 OpenFlow Show Commands

72

RP/0/RSP0/CPU0:ASR9K-AG06-4#show openflow switch 102 flows Logical Switch Id: 102Total flows: 1Flow: 1

Match: ipv6,ipv6_dst=192:1::2 Actions: output_nh(ipv6=2001:20::2)Priority: 0Table: 0Cookie: 0x1Duration: 196.353sNumber of packets: 0Number of bytes: 0

RP/0/RSP0/CPU0:ASR9K-AG06-4#show openflow switch 102 portsTue Jan 27 14:07:21.025 UTCLogical Switch Id: 102Port Interface Name Config-State Link-State Features

1 Te0/1/0/1 PORT_UP LINK_UP 0


OpenDaylight Controller

73

Visit DevNet for OpenDaylight and Cisco Open SDN Controller


OpenFlow Show Commands

• show openflow switch <>

• show openflow switch <> controllers | stats

• show openflow switch <> ports

• show openflow switch stats

• show openflow switch flows | brief/summary

• show openflow interface switch <>

• show openflow hardware capabilities pipeline <>

• show table-cap table-type <>

74

For YourReference


OpenFlow Show Commands

• Show policy-map commands:

• show policy-map transient list type pbr

• show policy-map transient type pbr pmap-name <>

• show policy-map transient targets summary

• PBR platform show commands

• show pbr-pal ipolicy [<policy_name> | all ] location <loc>

• show pbr-pal ipolicy <policy_name> iclass [<iclass_handle> | all ] vmr-info location <loc>

• show pbr-pal ipolicy <policy_name> iclass [<iclass_handle> | all] stats [clear-on-read] location <loc>

• show prm server tcam entries <table> vmr-id <> 100 np0 loc <>

• show prm server tcam summary <table> PBR all loc <>

75

For YourReference


OpenFlow Debug

• Debug for OpenFlow Agent:

• debug openflow switch ovs module ofproto level debug

• debug openflow switch ovs module ofproto-plif level debug

• debug openflow switch ovs module plif-onep level debug

• debug openflow switch ovs module plif-onep-util level debug

• debug openflow switch ovs module plif-onep-wt level debug

• Debug for Policy Manager

• debug policymgr all

• debug policymgr trace

• debug policymgr lib all

• debug policymgr lib trace

• Debug for PBR: debug pbr-pal all loc

76

For YourReference


ASR9000 OpenFlow References

ASR9000 and OpenFlow Whitepaper:

https://supportforums.cisco.com/document/12402506/asr-9000-sdn-openflow-whitepaper


http://www.cisco.com/c/en/us/support/routers/asr-9000-series-aggregation-services-routers/products-installation-and-configuration-guides-list.html

77

For YourReference

https://supportforums.cisco.com/document/12402506/asr-9000-sdn-openflow-whitepaper

http://www.cisco.com/c/en/us/support/routers/asr-9000-series-aggregation-services-routers/products-installation-and-configuration-guides-list.html

ASR9000 NETCONF/YANG


Traditional: Command Line Interface (CLI)

• Generally a majority of today’s configuration

• Device Specific

• Human Friendly

• Manual (challenging to automate)

• Developer un-friendly

– Such parsing CLI screen scrape

• No Common Data Model

• No Built-In Error Reporting

79

[joerober@rtp-odl ~]$ ssh [email protected]@10.99.1.1's password: RP/0/RSP0/CPU0:ASR9K-AG04-1#configure terminal Tue Jan 6 17:18:58.350 ESTRP/0/RSP0/CPU0:ASR9K-AG04-1(config)#


NETCONF

• Network Configuration Protocol (NETCONF)

– XML based interface between network device and NMS

– Mechanism to manage, configure, and monitor network device

• Published RFC 4741 (Dec 2006)

• Updated RFC 6241 (Jun 2011)

• Overcome SNMP Limitations

– 2003 IAB Network Management Workshop (RFC3535)

80


Comparing SNMP and NETCONF

81

SNMP NETCONF

Data Models Defined in MIBs Defined in YANG modules (or

XML schema documents)

Data Modeling Language Structure of Management

Information (SMI)

YANG (and XML schema)

Management Operations SNMP NETCONF

RPC Encapsulation Basic Encoding Rules (BER) XML

Transport Protocol UDP TCP (reliable transport)


NETCONF Layers

• Content

– Configuration and Operational Data

• Operations

– Defines base operations

– What clients do to servers

• Messages/Remote Procedure Call (RPC)

– Transport-independent mechanism for encoding messages

– RPC, RPC-Reply

• Transport

– Reliable communication between client and server

82

Content

Operations

Messages

Transport


ASR9000 NETCONF Transport & Messages

• TTY NETCONF session:

– Logon through telnet and then enter netconf command

• SSH NETCONF session:

– Logon through SSH and then enter netconf command

– *IANA-assigned TCP port 830 for NETCONF SSH support starts in 5.3.0

83

Transport

RPC

Operations

Content

Client Server

<rpc>

<rpc-reply>


ASR9000 NETCONF Configuration (XR 5.2.2)

• NETCONF TTY:

netconf agent tty

• NETCONF SSH:

ssh server v2

netconf agent tty

• Session verification:

show netconf session

84

! IOS XR 5.2.2

hostname ASR9K-R1

domain name cisco.com

!

netconf agent tty

!

ssh client vrf default

ssh server v2

ssh server vrf default


ASR9000 NETCONF Data Stores

• Target of NETCONF Operations

• Data stores are named containers that may hold an entire copy of the configuration

• IOS XR Supported Datastores:

– <running>

– <candidate>

– *XR two stage commit (no startup datastore)

Running Candidate

85

Transport

RPC

Operations

Content


<edit-config>

<PBR> <ServicePolicy>

<Input>PBTS</Input>

</ServicePolicy>

</PBR>


Programming

Use-Case:NETCONF Detailed Policy/Traffic Steering

WAN

R1

R2


3

3

Service Request1Analytic to determine LSP

2

Create LSP Tunnel FCID 23

NETCONF to program:

-detailed policy*

-apply policy inbound

4

10.201.7.43

2

4

*example policy in reference slides


Starting SSH NETCONF Session nms$ ssh –l lab 10.99.8.8 netconf echo format

[email protected]'s password:

Tue Apr 1 03:00:24.199 UTC

<?xml version="1.0" encoding="UTF-8"?>

<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">

<capabilities>

<capability>urn:ietf:params:netconf:base:1.0</capability>

<capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>

<capability>urn:ietf:params:netconf:capability:notification:1.0</capability>

</capabilities>

<session-id>285212672</session-id>

</hello>

87


NETCONF Operation <hello>

• Used to exchange capabilities

• Initiated by the NETCONF Server

• Must be acknowledged by client before client can send any other messages

IOS XR (5.2.2):


<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">

<capabilities>

<capability>urn:ietf:params:netconf:base:1.0</capability>

<capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>

<capability>urn:ietf:params:netconf:capability:notification:1.0</capability>

</capabilities>

<session-id>285212672</session-id>

</hello>

88

Transport

RPC

Operations

Content


NETCONF Operations - <edit-config>

• Modify configuration of particular data store

• Can only be used on writable data store

• Support on IOS XR:

– Candidate Configuration Capability

urn:ietf:params:netconf:capability:candidate:1.0

89

Transport

RPC

Operations

Content


Example <edit-config><?xml version="1.0" encoding="UTF-8" ?>

<rpc message-id="106" xmlns="urn:ietf:params:netconf:capability:candidate:1.0">

<edit-config>

<target><candidate/></target>

<config>

<Configuration><InterfaceConfigurationTable><InterfaceConfiguration><Naming>

<Active>act</Active>

<InterfaceName>TenGigE0/3/0/0</InterfaceName>

</Naming>

<Description>NETCONF CONFIG</Description>

<IPV4Network><Addresses><Primary>

<Address>10.223.1.1</Address>

<Netmask>255.255.255.0</Netmask>

</Primary></Addresses></IPV4Network>

</InterfaceConfiguration></InterfaceConfigurationTable>

</Configuration></config>

</edit-config> </rpc> ]]>]]>

Datastore

Interface

Add interface

description

IPv4 Address

Netmask

interface TenGigE0/3/0/0

description NETCONF CONFIG

ipv4 address 10.223.1.1 255.255.255.0

90


Example <edit-config> (cont)<?xml version="1.0" encoding="UTF-8"?>

<rpc-reply message-id="106" xmlns="urn:ietf:params:netconf:capability:candidate:1.0">

<ok/>

</rpc-reply>

]]>]]>

<?xml version="1.0" encoding="UTF-8" ?>


<commit/>

</rpc>

]]>]]>



<ok/>

</rpc-reply>

]]>]]>

rpc-reply

Commit

Configuration

rpc-reply

91


NETCONF Operations - <get-config>

• Used to retrieve all or portions of configuration

• Subtree filtering support:

– Attribute Match Expression• Can only be specified in Table classes (e.g. <InterfaceName Match=“TenGig.*”/> )

– Containment Nodes: • Filtering is by specifying container classes (e.g. <InterfaceConfigurationTable/> )

– Selection Node: • Filtering by specific selection (e.g. <InterfaceName>Loopback0</InterfaceName>

92

Transport

RPC

Operations

Content


Example <get-config>(Attribute match)<?xml version="1.0" encoding="UTF-8" ?>


<get-config>

<source><running/></source>

<filter>

<Configuration>

<InterfaceConfigurationTable>

<InterfaceConfiguration>

<Naming>


<InterfaceName Match=“TenGigE.*”/>

</Naming>

</InterfaceConfiguration>

</InterfaceConfigurationTable>

</Configuration>

</filter>

</get-config>

</rpc> ]]>]]>

Attribute match: all TenGigE interfaces

Datastore

93


Example <get-config>(Containment node)<?xml version="1.0" encoding="UTF-8" ?>


<get-config>


<filter>

<Configuration>

<InterfaceConfigurationTable/>

</Configuration>

</filter>

</get-config>

</rpc> ]]>]]>

Containment: all interfaces

Datastore

94


Example <get-config> (selection node)<?xml version="1.0" encoding="UTF-8" ?>


<get-config>


<filter>

<Configuration>

<InterfaceConfigurationTable>

<InterfaceConfiguration>

<Naming>


<InterfaceName>Loopback0</InterfaceName>

</Naming>

</InterfaceConfiguration>

</InterfaceConfigurationTable>

</Configuration>

</filter>

</get-config>

</rpc> ]]>]]>

Datastore

Selection: Loopback0 only…

95


Example <get-config> ASR9000 <rpc-reply><?x<?xml version="1.0" encoding="UTF-8"?>


<data><Configuration>

<InterfaceConfigurationTable MajorVersion="5" MinorVersion="3">

<InterfaceConfiguration> <Naming><Active>act</Active>

<InterfaceName>Loopback0</InterfaceName>

</Naming>

<InterfaceVirtual>true</InterfaceVirtual>

<IPV4Network MajorVersion="6" MinorVersion="3">

<Addresses><Primary>

<Address>10.99.8.8</Address>

<Netmask>255.255.255.255</Netmask>

</Primary></Addresses></IPV4Network>


</Configuration></data>

</rpc-reply> ]]>]]>

Tagged Configuration

96


Closing SSH NETCONF Session <?xml version="1.0" encoding="UTF-8" ?>

<rpc message-id="106" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">

<close-session>

</close-session>

</rpc> ]]>]]>


<rpc-reply message-id="106" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">

<ok/>

</rpc-reply>

]]>]]>

Connection to 10.99.8.8 closed by remote host.

odl-1$

RPC close-session

RPC-Reply

97

Connection Closed


YANG

• Modeling language defined in RFC 6020

• Used by NETCONF to define objects and data in requests and replies

• Models configuration, operational, and RPC data

• Provides semantics to better define NETCONF data

• Provides common data model:

– In order for NETCONF to be useful as network-wide protocol

– To consume NETCONF data from any network device

• YANG modules are for NETCONF what MIBs are for SNMP

98


YANG XR / ASR9000 Support

• YANG data modules still being developed with standards drafts being proposed and under review

• IOS XR Support for YANG being taken in phased approach based on YANG modules being developed by Cisco and industry standardization

• First phase for general release being targeted for XR 5.3.1*

• First phase targeting management, core, access, and services based functions

• YANG models available as SMU for ASR9K in controlled availability. Contact your account team for further details.

99

* DISCLAIMER: Many of the products and features described herein remain in varying stages of development and will be offered on a when-and-if-available basis. This roadmap is subject to

change at the sole discretion of Cisco, and Cisco will have no liability for delay in the delivery or failure to deliver any of the products or features set forth in this document.

Summary


Traditional Infrastructure

Data

Center

Customers

101


Optical

IP/MPLS


System Engineering


Machines)

Network Engineering


Virtual


Evolved Programmable Network

Data

Center

Customers

102


Optical

IP/MPLS


System Engineering


Machines)

Network Engineering


Virtual

Physical and Virtual

API’s

Orchestration

API’s

Applications





WAN

R1

R2


5

1

2

3

4Congested!!





to collector1


DC2 bandwidth NOW2

Analytic and confirmation to

determine LSP3

R1-R3 LSP Tunnel

programmed4

Traffic Steering onto

R1-R3 LSP5





WAN

R1

R2


5

1

2

3

4Congested!!




Augment Collection: BGP-LS1

Service Request2

Analytic, Orchestration

e.g. WAE3

LSP: PCEP, NETCONF, etc

Controller-ODL, Tail-f, etc4

Traffic Steering:

OpenFlow, NETCONF, etc5


Summary-ASR9000 APIs & Protocols

Key Function Protocol/API Comments

IGP Topology BGP Link-State Wraps up LSDB in BGP transport and pushes to BGP speaker

on SDN WAN Orch Platform

Create, Modify and Delete TE

or SR Tunnels

Stateful Extensions to

PCEP

Introduced as part of Stateful PCE effort

Classification and Action Openflow Extensions Leveraging per-flow MATCH/Action semantics

Read/Write of Persistent

Configuration Data on

Network Devices

Netconf/Yang Finally gaining traction with vendor implementations and now on

OpenDaylight Platform

105


Call to Action

• Visit the World of Solutions for

– Service Provider Section

• ASR9000

• WAN Automation Engine

• Tail-f

• DevNet Zone

– Related labs and sessions

– Cisco Open SDN Controller

– WAN Automation Engine (WAE)-Hands-On Labs

• Cisco Live/DevNet SDN Sessions: http://cs.co/CLEU15-ACI

106


DevNet: Getting Started with WAE

• DevNet – WAE Tutorials and API Documentation https://developer.cisco.com/site/wae/index.gsp

• Learning Labs – Hands on with the WAE APIs over in the DevNet zone https://learninglabs.cisco.com/#/home

• dCloud – demos of WAEhttp://dcloud.cisco.com/

• WAE home page on cisco.com: http://www.cisco.com/go/wae

WAE Learning labs

For YourReference

https://developer.cisco.com/site/wae/index.gsp

https://learninglabs.cisco.com/#/home

http://dcloud.cisco.com/

http://www.cisco.com/go/wae


World of Solutions: Tail-f NCS Overview

• Logically centralized network services

• Data models for data structures

• Structure representations of:

– Service instances

– Network configuration and state

• Mapping service operations to network configuration changes

• Transactional integrity

• Multiprotocol support

• Multivendor support

108

Device Models Network Element Drivers

Device Manager

Service Manager

Tail-f NCS Service Models

Networkwide CLI and Web UI REST, NETCONF, Java, etc.

Network Engineer

Management Applications

NETCONF, CLI, SNMP, REST, etc.

• Execution management systems (EMSs)

• Applications • Controllers

For YourReference


DevNet: Open SDN Controller (OSC)

• Cisco Distribution of OpenDaylight platform

• Enhancements

– Simplified install

– Admin/Monitoring tools

– Embedded Apps

– Plug-in Clustering

– Technical Support

• developer.cisco.com/site/openSDN/

109

Community Support Cisco Supported

“HELIUM”

Open SDN Controller

Common Content Incremental

Cisco

Value

Precluded

OpenDaylight

Content

SDNi

Defense4all

PacketCable

PCMM

VTN Project

OpenContrail

Plugin

SNMP4SDN

AAA

BGP-LS

Controller

DLUX

Group Policy

L2 Switch

LISP Flow

Mapping

Openflow

Plugin

OVSDB

Secure Network

Bootstrap Infra

Service Function

Chaining

Yang Tools

MD-SAL

AD-SAL

Log

Aggregation

Metrics

Aggregation

Monitoring

Basic

Clustering

Central Admin

OVA Distribution

Sample Apps

One Click Install PCEP

Plug-in Clustering

To be contributed back

to the “open community”

For YourReference


• Devloper Network: http://developer.cisco.com

• OpenDaylight: http://www.opendaylight.org/

• SDLC - https://developer.cisco.com/site/networking/one/sdlc/overview/

• EEM - https://supportforums.cisco.com/community/netpro/network-infrastructure/eem

• ONE Forums -https://developer.cisco.com/site/devnet/forums/index.gsp#L2CiscoONE

• APIC-EM - https://developer.cisco.com/site/networking/one/apic/enterprise-module/

• APIC-DC

– APIs https://developer.cisco.com/site/networking/routers-switches/nexus9000/documents/

– GitHub - https://github.com/datacenter/nexus9000

Cisco Developer Resources

110

For YourReference

http://developer.cisco.com

https://supportforums.cisco.com/community/netpro/network-infrastructure/eem

https://developer.cisco.com/site/devnet/forums/index.gsp#L2CiscoONE

https://developer.cisco.com/site/networking/one/apic/enterprise-module/

https://developer.cisco.com/site/networking/routers-switches/nexus9000/documents/

https://github.com/datacenter/nexus9000


Complete Your Online Session Evaluation

• Please complete your online sessionevaluations after each session.Complete 4 session evaluations& the Overall Conference Evaluation(available from Thursday)to receive your Cisco Live T-shirt.

• All surveys can be completed viathe Cisco Live Mobile App or theCommunication Stations

111

Reference Slides


PCE-Initiated LSP (Multiple PCEs)

• PCC synchronizes LSP state over all open stateful PCEP sessions

• When a PCE creates / initiates an LSP

– PCC will report LSP state to all stateful PCEs

– PCC will only delegate LSP to originating PCE

• LSP may be re-delegated if originating PCE disconnects or renounces delegation

• LSPs may be re-delegated to a stateful PCE sending a matching LSP creation / initiation before LSP cleanup timeout

PCEP

Stateful

PCC

Stateful PCE B

TED

LSP DB

Stateful PCE A

TED

LSP DB

PCEP

115

For YourReference


ASR9000 PBTS Configurationipv4 access-list IPV4_SRC10 permit ipv4 host 10.201.7.43 any!class-map type traffic match-any IPV4_SRCmatch access-group ipv4 IPV4_SRC end-class-map!policy-map type pbr PBTSclass type traffic IPV4_SRC set forward-class 2!end-policy-map!interface TenGigE0/1/0/1ipv4 address 172.16.0.5 255.255.255.254service-policy type pbr input PBTS!

Use PBR policy to

set forwarding class

Apply PBR policy to

set forwarding class

116

For YourReference


ASR9000 NETCONF Operation <edit-config><?xml version="1.0" encoding="UTF-8" ?>


<edit-config>

<target><candidate/></target>

<config>

<Configuration><InterfaceConfigurationTable><InterfaceConfiguration><Naming>


<InterfaceName>TenGigE0/3/0/1</InterfaceName>

</Naming>

<PBR>

<ServicePolicy>

<Input>PBTS</Input>

</ServicePolicy>

</PBR>


</Configuration></config>

</edit-config> </rpc> ]]>]]>

Datastore

Interface

Apply PBR Input Policy “PBTS”

117

For YourReference


openflow switch 11 pipeline 132interface Bundle-Ether2.1interface GigabitEthernet0/1/0/7interface GigabitEthernet0/0/0/0/4.1301controller ipv4 172.16.1.45 port 6653 security tls

openflowswitch 10 pipeline 132vrf ONEcontroller ipv4 172.16.1.45 port 6653 security tls

OpenFlow L3_DS SwitchUsing VRF:

All interfaces in VRF become part of

OpenFlow Switch

Using L3 Interfaces

118

For YourReference

Documents

Deploying SDN on ASR 9000d2zmdbbm9feqrf.cloudfront.net/2015/eur/pdf/BRKSPG-2722.pdf192.4/31 192.6/31 128.2/31 128.0/31 192.0/31 AG04-4 10.99.8.8/32 AG04-2 10.99.2.2/32 AG06-3 10.99.5.5