WAN Internet Data Center

LAB

Building Aggregation 20 – 30 Buildings

Building Aggregation 20 – 30 Buildings

Core Routing Module

Building Building Building Building

Labs Security Devices

Developer Internet

Business Partners Others

Today: L2VPN, L3VPN Future: VXLAN

Today: MPLS, OSPF, BGP Future: OSPF, BGP

0

50000

100000

150000

Trident+ Trident II NG

IPv4 FIB LPM IPv6 FIB LPM

0

100

200

300

Trident+ Trident II or + NG

Bandwidth in 10G Increment 10GE Ports

40GE Ports 100GE Ports

•  Simple Connectivity Over an IP Only Network •  Instant Connectivity From Service Source & Destination Tenant in a Data Center

with an IP Only L3 Clos Fabric. •  Cost Efficient Service Extender Over a 3rd Party IP Network •  Use Cost Efficient Ports When you Can and Expensive Ports Where you Must

•  Cheap Pseudo Wires without MPLS •  API Driven Provisioning: Simplified and Easily Consumed by Developers •  Removal of additional Protocols to Manage •  Reduction in Cost, Driven by Merchant Silicon & Competition

•  Remote Tap Termination •  Transport Wire Tap (SPAN) Aggregation Over an IP Network

•  Lab Patch Panel •  Anywhere to Anywhere Patch Panel for Labs

Use Cases

•  UDP Header: •  Hashing Works Just Like Any Other UDP Traffic

•  Supported in Wide Variety of Platforms •  Broadcom Trident 2:

•  Arista Networks, Cisco Systems, Dell, Juniper Networks •  White Label: Bring Your Own Switch (BYOS) & Purchase Operating System (Cumulus & Others

to Follow) •  Widely Deployed in Many Networks

•  Dense, Power/Space Efficient & Cheap! $60 – 100 per 10GE, Sometimes Better!

•  Supports Point to Point & Multi-Point •  Point to Point Doesn’t Require Control Plane Software •  Multi-Point Does Require Control Plane Software or Self Provided Control Plane Software

•  Wide Vendor Support & Interest

Why VXLAN NVGRE NOTE: -  Utilized in Other Parts of Microsoft -  Used in Overlay Network Creation

VXLAN for this presentation is not utilized for its intended use-case.

•  Worked •  Tagged Packets (VLAN) •  VRRP •  OPSFv2, OSPFv3 •  BGP Address Family: IPv4, IPv6, VPNv4 •  RSVP

•  Didn’t Work or In Progress •  LACP Doesn’t Pass Through (Ethernet Channel) •  LLDP Doesn’t Pass Through, but works locally •  LDP (Possibly 224.0.0.2/1 Multicast Issue & Interception) •  IS-IS – (Investigating)

What Worked & Didn’t

Continuously Investigating What Else Doesn’t Work and What Can Work or be Fixed…

NOTICE:

Use Case: Lab Patch Panel Puget Sound Campus: •  100+ Building •  Distributed Labs

Building A

Building B

Building C

Building D

Building E

Building F Today: MPLS L2VPN Future: T2 Based Patch Panel

Why VXLAN: •  IP Only is Cheap &

MPLS is Expensive •  Wide Vendor Choice •  UDP: Hashing •  Less Protocols &

Dependencies •  API Driven + Apps (Self

Service with Windows Phone & Tablet)

Use Case: VRF Extension

Any

T2

VTEP

/ SP

INE

Any

T2

VTEP

/ SP

INE

Use Case: TAP Remote Transport Reasons & Benefits: •  Limit Expensive Tools Deployment •  Redirect Captured Traffic for Personal Analysis •  No Tools in Local Location

•  SPAN to VXLAN Transport •  Yet another RSPAN, but IP based

Tap

Out

put

VXLAN Input Port Terminating to Remote VXLAN Switch

Loopback Cable

Lab Environment: Physical Connectivity

Underlay Network IP Fabric

Overlay Network

switch

N3132Q

N3132Q

.10

Underlay Network Configuration interface TenGigabitEthernet 0/0 ip address 192.168.254.0/31 ip ospf network point-to-point no shutdown ! interface TenGigabitEthernet 0/1 ip address 192.168.254.2/31 ip ospf network point-to-point no shutdown ! interface TenGigabitEthernet 0/2 ip address 192.168.254.4/31 ip ospf network point-to-point no shutdown ! interface TenGigabitEthernet 0/3 ip address 192.168.254.6/31 ip ospf network point-to-point no shutdown ! interface Loopback 0 ip address 192.168.254.100/32 no shutdown ! router ospf 1 router-id 192.168.254.100 network 192.168.254.0/24 area 0

interface TenGigabitEthernet 0/0 ip address 192.168.254.8/31 ip ospf network point-to-point no shutdown ! interface TenGigabitEthernet 0/1 ip address 192.168.254.10/31 ip ospf network point-to-point no shutdown ! interface TenGigabitEthernet 0/2 ip address 192.168.254.12/31 ip ospf network point-to-point no shutdown ! interface TenGigabitEthernet 0/3 ip address 192.168.254.14/31 ip ospf network point-to-point no shutdown ! interface Loopback 0 ip address 192.168.254.200/32 no shutdown ! router ospf 1 router-id 192.168.254.200 network 192.168.254.0/24 area 0

interface Ethernet1/50/2 no switchport ip address 192.168.254.11/31 ip ospf network point-to-point interface Ethernet1/50/3 no switchport ip address 192.168.254.13/31 ip ospf network point-to-point interface Ethernet1/50/4 no switchport ip address 192.168.254.15/31 ip ospf network point-to-point

interface Ethernet1/49/1 no switchport ip address 192.168.254.1/31 ip ospf network point-to-point interface Ethernet1/49/2 no switchport ip address 192.168.254.3/31 ip ospf network point-to-point interface Ethernet1/49/3 no switchport ip address 192.168.254.5/31 ip ospf network point-to-point interface Ethernet1/49/4 no switchport ip address 192.168.254.7/31 ip ospf network point-to-point interface Ethernet1/50/1 no switchport ip address 192.168.254.9/31 ip ospf network point-to-point

feature ospf router ospf 1 router-id 192.168.254.150 network 192.168.254.0/24 area 0.0.0.0

Overlay Network Configuration Part 1 feature vxlan

! vxlan-instance 1 gateway-ip 192.168.254.100 controller 1 10.37.33.35 port 6632 ptcp no shutdown ! interface ManagementEthernet 0/0 ip address 10.37.33.35/24 no shutdown ! management route 10.0.0.0/8 10.37.33.1 management route 172.16.0.0/12 10.37.33.1

interface TenGigabitEthernet 0/116 vxlan-instance 1 no ip address no shutdown ! interface TenGigabitEthernet 0/117 vxlan-instance 1 no ip address no shutdown ! interface TenGigabitEthernet 0/118 vxlan-instance 1 no ip address no shutdown ! interface TenGigabitEthernet 0/119 vxlan-instance 1 no ip address no shutdown

Base Client Facing feature vxlan ! vxlan-instance 1 gateway-ip 192.168.254.200 controller 1 10.37.33.35 port 6632 ptcp no shutdown ! interface ManagementEthernet 0/0 ip address 10.37.33.36/24 no shutdown ! management route 10.0.0.0/8 10.37.33.1 management route 172.16.0.0/12 10.37.33.1

interface TenGigabitEthernet 0/8 vxlan-instance 1 no ip address no shutdown ! interface TenGigabitEthernet 0/116 vxlan-instance 1 no ip address no shutdown ! interface TenGigabitEthernet 0/117 vxlan-instance 1 no ip address no shutdown ! interface TenGigabitEthernet 0/118 vxlan-instance 1 no ip address no shutdown ! interface TenGigabitEthernet 0/119 vxlan-instance 1 no ip address no shutdown

Base Client Facing

Overlay Network Configuration Part 2 Controller: Flow Provisioning – S6000-1 Controller: Flow Provisioning – S6000-2

Creation of Logical Network: vtep-ctl --db=tcp:10.37.33.35:6632 add-ls "LN1:ELINE:1001" vtep-ctl --db=tcp:10.37.33.35:6632 add-ls "LN2:ELINE:1002" vtep-ctl --db=tcp:10.37.33.35:6632 add-ls "LN3:ELINE:1003" vtep-ctl --db=tcp:10.37.33.35:6632 add-ls "LN4:ELINE:1004" vtep-ctl --db=tcp:10.37.33.35:6632 add-ls "LN7:ELINE:1007“ Bind Port to the Logical Network: vtep-ctl --db=tcp:10.37.33.35:6632 bind-ls "10.37.33.35" "Te 0/116" 0 LN1:ELINE:1001 vtep-ctl --db=tcp:10.37.33.35:6632 bind-ls "10.37.33.35" "Te 0/117" 0 LN2:ELINE:1002 vtep-ctl --db=tcp:10.37.33.35:6632 bind-ls "10.37.33.35" "Te 0/118" 100 LN3:ELINE:1003 vtep-ctl --db=tcp:10.37.33.35:6632 bind-ls "10.37.33.35" "Te 0/118" 200 LN4:ELINE:1004 vtep-ctl --db=tcp:10.37.33.35:6632 bind-ls "10.37.33.35" "Te 0/119" 0 LN7:ELINE:1007 Tunnel Termination to the Remote Node: vtep-ctl --db=tcp:10.37.33.35:6632 add-ucast-remote "LN1:ELINE:1001" "00:11:11:11:11:11" "192.168.254.200" vtep-ctl --db=tcp:10.37.33.35:6632 add-ucast-remote "LN2:ELINE:1002" "00:33:33:33:33:33" "192.168.254.200" vtep-ctl --db=tcp:10.37.33.35:6632 add-ucast-remote "LN3:ELINE:1003" "00:55:55:55:55:55" "192.168.254.200" vtep-ctl --db=tcp:10.37.33.35:6632 add-ucast-remote "LN4:ELINE:1004" "00:77:77:77:77:77" "192.168.254.200" vtep-ctl --db=tcp:10.37.33.35:6632 add-ucast-remote "LN7:ELINE:1007" "00:14:14:14:14:14" "192.168.254.200"

Creation of Logical Network: vtep-ctl --db=tcp:10.37.33.36:6632 add-ls "LN1:ELINE:1001" vtep-ctl --db=tcp:10.37.33.36:6632 add-ls "LN2:ELINE:1002" vtep-ctl --db=tcp:10.37.33.36:6632 add-ls "LN3:ELINE:1003" vtep-ctl --db=tcp:10.37.33.36:6632 add-ls "LN4:ELINE:1004" vtep-ctl --db=tcp:10.37.33.36:6632 add-ls "LN7:ELINE:1007“ Bind Port to the Logical Network: vtep-ctl --db=tcp:10.37.33.36:6632 bind-ls "10.37.33.36" "Te 0/116" 0 LN1:ELINE:1001 vtep-ctl --db=tcp:10.37.33.36:6632 bind-ls "10.37.33.36" "Te 0/117" 0 LN2:ELINE:1002 vtep-ctl --db=tcp:10.37.33.36:6632 bind-ls "10.37.33.36" "Te 0/118" 100 LN3:ELINE:1003 vtep-ctl --db=tcp:10.37.33.36:6632 bind-ls "10.37.33.36" "Te 0/119" 200 LN4:ELINE:1004 vtep-ctl --db=tcp:10.37.33.36:6632 bind-ls "10.37.33.36" "Te 0/8" 0 LN7:ELINE:1007 Tunnel Termination to the Remote Node: vtep-ctl --db=tcp:10.37.33.36:6632 add-ucast-remote "LN1:ELINE:1001" "00:22:22:22:22:22" "192.168.254.100" vtep-ctl --db=tcp:10.37.33.36:6632 add-ucast-remote "LN2:ELINE:1002" "00:44:44:44:44:44" "192.168.254.100" vtep-ctl --db=tcp:10.37.33.36:6632 add-ucast-remote "LN3:ELINE:1003" "00:66:66:66:66:66" "192.168.254.100" vtep-ctl --db=tcp:10.37.33.36:6632 add-ucast-remote "LN4:ELINE:1004" "00:88:88:88:88:88" "192.168.254.100" vtep-ctl --db=tcp:10.37.33.36:6632 add-ucast-remote "LN7:ELINE:1007" "00:15:15:15:15:15" "192.168.254.100"

Overlay Network Configuration Part 3

RTR1 { interfaces { xe-0/0/0 { unit 0 { family inet { address 192.168.1.2/24 { vrrp-group 1 { virtual-address 192.168.1.1; accept-data; } } } } } } }

feature interface-vlan interface Ethernet1/2/1 switchport access vlan 300 spanning-tree port type edge speed 10000 interface Ethernet1/2/2 switchport access vlan 300 spanning-tree port type edge interface Vlan300 no shutdown vrf member VRF1 ip address 192.168.1.10/24

switch

root@MX80# run show vrrp logical-system RTR1 Interface State Group VR state VR Mode Timer Type Address xe-0/0/0.0 up 1 master Active A 0.588 lcl 192.168.1.2 vip 192.168.1.1

root@MX80# run show vrrp logical-system RTR2 Interface State Group VR state VR Mode Timer Type Address xe-0/0/1.0 up 1 backup Active D 3.130 lcl 192.168.1.3 vip 192.168.1.1 mas 192.168.1.2

RTR2 { interfaces { xe-0/0/1 { unit 0 { family inet { address 192.168.1.3/24 { vrrp-group 1 { virtual-address 192.168.1.1; priority 90; accept-data; } } } } } } }

N3132Q-2# ping 192.168.1.1 vrf VRF1 packet-size 1422 count 3 df-bit PING 192.168.1.1 (192.168.1.1): 1422 data bytes 1430 bytes from 192.168.1.1: icmp_seq=0 ttl=63 time=2.233 ms 1430 bytes from 192.168.1.1: icmp_seq=1 ttl=63 time=1.922 ms 1430 bytes from 192.168.1.1: icmp_seq=2 ttl=63 time=2.079 ms --- 192.168.1.1 ping statistics --- 3 packets transmitted, 3 packets received, 0.00% packet loss round-trip min/avg/max = 1.922/2.077/2.233 ms

Verification Part 1

S6000-1#sho vxlan vxlan-instance 1 logical-network Instance : 1 Total LN count : 7 Name VNID LN1:ELINE:1001 4656 LN2:ELINE:1002 4657 LN3:ELINE:1003 4658 LN4:ELINE:1004 4659 LN7:ELINE:1007 4660 LN5:ELINE:1005 4661 LN6:ELINE:1006 4662

S6000-1#sho vxlan vxlan-instance 1 logical-network name LN1:ELINE:1001 Name : LN1:ELINE:1001 Description : Tunnel Key : 4656 VFI : 28673 Port Vlan Bindings: Te 0/116: VLAN: 0 (0x80000001), S6000-1#show vxlan vxlan-instance 1 physical-locator Instance : 1 Tunnel : count 1 192.168.254.200 : vxlan_over_ipv4 (up)

S6000-1#sho vxlan vxlan-instance 1 Instance : 1 Admin State : enabled Management IP : 10.37.33.35 Gateway IP : 192.168.254.100 MAX Backoff : 30000 Controller 1 : 10.37.33.35:6632 ptcp (connected) Fail Mode : non secure Port List : Te 0/8 Te 0/9 Te 0/10 Te 0/11 Te 0/116 Te 0/117 Te 0/118 Te 0/119 Te 0/120 Te 0/121 Te 0/124

S6000-1#sho vxlan vxlan-instance 1 statistics interface te 0/116 0 Port : Te 0/116 Vlan : 0 Rx Packets : 326100 Rx Bytes : 23360185 Tx Packets : 91579 Tx Bytes : 8487292

S6000-1#sho vxlan vxlan-instance 1 unicast-mac-remote Total Local Mac Count: 7 VNI MAC TUNNEL 4656 00:11:11:11:11:11 192.168.254.200 4657 00:33:33:33:33:33 192.168.254.200 4658 00:55:55:55:55:55 192.168.254.200 4659 00:77:77:77:77:77 192.168.254.200 4660 00:14:14:14:14:14 192.168.254.200 4661 00:99:99:99:99:99 192.168.254.200 4662 00:12:12:12:12:12 192.168.254.200

Verification Part 2

•  Operationally Simple Ethernet Line Services •  Eliminate Protocol Dependency & Complexity •  Expand Vendor Choice

•  Practical & Economical Service Deployments •  Deploy Services Over Legacy Networks or IP Only Networks •  Deploy Services Over Other Providers Networks

•  Future Use Cases in the Works: •  ELINE Health Status Details •  Encapsulation / De-encapsulation Gateway – Bridging of Overlay Networks •  Hub and Spoke ELINE Services •  More Mobile Management Applications to deliver Self Service •  Pass Through Encryption Module

Summary

-  IPv4 & IPv6 VRRP -  IPv4: TAG + OSPF + BGP -  IPv6: TAG + OSPFv3 + BGP -  Ethernet Bundle: TAG + IPv4 -  MPLS: RSVP+ Remote PE -  OPEN:

MX80

N3132Q-1

S6000-1 N3064 S6000-2 QFX5100-48s-6q PE

N3132Q-2: VRF1

N3132Q-2

N3132Q-2: VRF2

N3132Q-2: VRF3

MX80: LSYS1

MX80: LSYS2

MX80: LSYS3

MX80: LSYS4