MIGRATION TO A FUTURE PROOF ARCHITECTURE - … · 2012-10-05 · RVI, routing and advance service (MPLS/VPLS) configuration on MX RVI, routing + Zone/Policy configuration on SRX L2/VLAN

MIGRATION TO A FUTURE PROOF ARCHITECTURE

Antoine SIBOUT, Data Center Consultant EMEA

September 19th

2 Copyright © 2012 Juniper Networks, Inc. www.juniper.net

AGENDA

Qfabric review: key components & roles

Building a L2/L3 network with services on Qfabric

Interconnecting/migrating ‘Old’ to Qfabric

Storage networking

Conclusion


QFABRIC SYSTEM REVIEW


A NEW ARCHITECTURE

Distributed Control plane model shares processing tasks among system

components which improves scaling when compared to master/backup model

Distributed Forwarding (L2/L3)

Centralized management & monitoring

Distributed RE Model Master/Backup Model


QFABRIC COMPONENTS

Device Name High-level Function

Node device Line card component, entry and exit point for data

passing through fabric

Interconnect device Backplane component, high-speed transport device

that interconnects Node devices

Director device Routing engine component, provides control and

management services for system

Control plane

switches

Control plane link, interconnects all devices in QFabric

system on control plane network

Director

devices

Control plane

switches

Node devices

Interconnect

devices


NODE DEVICES

Node devices:

Connect to servers, storage devices, or external networks

Can serve in a standalone mode or in fabric mode

Two options available: QFX3500 and QFX3600

Node devices


DATA PLANE CONNECTIONS

QFX3500 supports 1 GbE and 10 GbE connections and 2 Gb, 4 Gb, or 8 Gb Fibre Channel ports

QFX3600 supports 10 GbE connections using QSFP+ direct attached break out cables

Both use 40G uplinks to interconnects

QFX3500 QFX3600

Direct Attached/Twinax SFP+ copper

1, 3, 5, 7, or 10 meter

SFP+ optical transceiver

USR, SR, LR

SFP transceiver

SR, LR, 1000BaseT

SFP optical transceiver

2 Gb, 4 Gb, or 8 Gb FC-SW

QSFP+ DAC BO

1 or 3 meter

Fibre Channel

capable ports


INTERCONNECT DEVICES

Interconnect devices:

Connect all Node devices and direct traffic between them

A/A multipathing (no loop, no idle link)

Interconnect

devices

Two options available:

QFX3008-I and QFX3600-I


DIRECTOR CLUSTER

Highly scalable & robust infrastructure

Collection of virtual machines, new software abstracts, and processes associated

with the Director group software

A distributed control plane between system components

CentOS

Fabric Manager

RE

Junos VM

Network Node

Group RE

Junos VM

Fabric Control

RE

Junos VM

Diagnostic

RE

Junos VM

Fabric Admin

Junos CLI

Other Services

DNS, NTP, …

KVM Hypervisor

DG 0 DG 1

Director group

Fabric Manager Provisioning and Topology Management. Network Node Group RE: Routing Engine for Network node group (runs all L2/L3 protocols)

Fabric Control:

Responsible for exchanging L2/L3 routes between Nodes via

Control Plane learning.

Diagnostic RE:

Provides advanced Debugging/analysis Fabric Admin: Provides the single logical switch abstraction (CLI, user session etc..)


QFABRIC-G VS QFABRIC-M

Same software release for both platforms

Same feature set

Same capabilities across nodes with the following specifics

FC interfaces only on QFX3500 Node (local Gateway)

40 G access interfaces only on QFX 3600 Node (used for 10G today with break out

cable as of today)

Can be mixed together as NNG nodes

Cannot be mixed inside RSNG

As a consequence, the following content is relevant to both Qfabric-G/M and

QFX3500/3600


QFABRIC NODES


NODE GROUPS

Node groups are a new software abstraction that allow you to designate

the role of each Node device:

Server Node group (SNG)

Redundant server Node group (RSNG)

Network Node group (NNG)

Assigning a role to a node dictates it capabilities / behavior

SNG/RSNG construct allow to scale the # of states

All Node devices initially belong to individual server Node groups, can be

changed through explicit configuration


SERVER NODE GROUP

Server Node groups:

Consist of a single Node device

Run only host-facing protocols such as LACP,

LLDP, ARP, and DCBX

Typically connect to servers and storage devices

not requiring cross-Node resiliency

Node device performs RE and PFE functions LAGs

SERVER NODE GROUP

RE PFE

Default group assignment for

all Node devices


REDUNDANT SERVER NODE GROUP

Redundant server Node groups:

Consist of two Node devices

Run only host-facing protocols such as LACP,

LLDP, ARP, and DCBX

Connects to servers, networking and storage

devices requiring cross-Node resiliency

LAGs

RE (Backup) PFE

RE (Active) PFE

Redundant Server Node Group

One Node device performs RE functions and both

Node devices perform PFE functions

Paired through explicit configuration


NETWORK NODE GROUP

The network Node group:

Consists of one or more Node devices (up to eight)

Can run host and network-facing Layer 2 and Layer

3 protocols (xSTP, OSPF, BGP)

Superset of SNG/RSNG

Typically connects to routers, switches and firewalls

but can connect to all network devices

LAGs

PFE

Network Node Group (PFEs)

PFE

PFE

DIRECTOR

DIRECTOR

Network Node Group REs Backup Active

PFE functions only.

RE functions on network Node group

REs on the Director group.

The network Node group exists by default and

uses the name NW-NG-0

OSPF


NODE GROUPS SUPPORT MATRIX

Node Groups Max # of members per node-group

Max # of node-groups

Qfabric-G/(Qfabric-M)

Same member LAG

Number of LAG

Qfabric-G/(Qfabric-M)

Cross member LAG

Host facing protocols

Network facing protocols

SNG 1 128 (16) ✔

3072 (384) ✔

RSNG 2 64 (8) ✔

3072 (384) ✔ ✔

NNG 8 1 ✔ (128) ✔ ✔ ✔


DEFAULTS

Initially, Node devices belong to server Node groups with names based on serial numbers of Node

devices

root@qfabric> show fabric administration inventory node-groups

Item Identifier Connection Configuration

Node group

NW-NG-0 Connected Configured

P1406-C Connected Configured

P1406-C Connected


P1481-C Connected


P2009-C Connected

…

[edit fabric resources]

root@qfabric# set node-group ?

Possible completions:

<name> Node group identifier

NW-NG-0 Node group

P1406-C Node group

P1481-C Node group

P2009-C Node group

…

You can change group names, create new groups, and

associate Node devices with the network Node group at the [edit fabric resources] hierarchy.


CONFIGURING ROLES & ALIASES

You can define customized aliases, and referenced them in the configuration (SNG, RSNG or NNG)

[edit fabric aliases]

root@qfabric# show

node-device P1406-C {

LC0;

}


LC1;

}


LC2;

}


LC3;

}


LC4;

}

[edit fabric resources]

root@qfabric# show

node-group sng1-f1-r2 {

node-device LC0;

}

node-group rsng1-f2-r3 {

node-device LC1;

node-device LC2;

}

node-group NW-NG-0 {

network-domain;

node-device LC3;

node-device LC4;

}

Note: Changing Node group assignments will cause the affected Node devices to reboot!

Aliases defined

Aliases referenced


QFABRIC ARCHITECTURE (L2, L2/L3)


QFABRIC - LAYER 2 ONLY

SRX Series L4/7 services

MX Series

Servers Storage

Internet MPLS/ VPN

L2

L3

Fin Mkt

RVI RVI

Configuration: RVI, routing and advance service (MPLS/VPLS) configuration on MX

RVI, routing + Zone/Policy configuration on SRX

L2/VLAN configuration on QFabric

Eng

RVI

SRX Service

NO SRX Service


QFABRIC - LAYER 2 ONLY

Qfabric is a large flat powerful & flexible ‘location agnostic’ L2 network which

connects/stitches services/routing devices to compute and storage

• Only traffic that needs services hits the SRX (traffic steering using default

gateways

• QFabric will be used for switching all Intra-VLAN traffic

• MX routes Inter VLAN traffic that doesn’t need SRX service

• MX routes ‘outside’ and provides advance service like MPLS and VPLS.


VRF_A VRF_B

QFABRIC – LAYER 2/3 WITH VRF

SRX Series

MX Series

Storage

Internet MPLS/ VPN

L3

Fin Mkt

RVI RVI

Eng

RVI

Zone_A

RVI

HR

Configuration: Routing and advance service (MPLS/VPLS) configuration on MX

RVI, VRs, routing + Zone/Policy configuration on SRX

Routing, VRF, RVI, and VLAN configuration on QFabric.

SRX Service

VRF_A


QFABRIC – LAYER 2/3 WITH VRF

Good Platform Adaptation & Appropriate Traffic Steering

• Only traffic that needs services hits the SRX

• QFabric – will be used for switching and routing

• QFabric routes a significantly higher volume of traffic that don’t need services,

only WAN and Inter-DC traffic go to MX

More Complex Configuration

• User has to configure corresponding VRFs on QFabric and stitch zones and

VRFs to SRX

NNG Nodes required to connect SRX/MX


MIGRATION 'FROM 'OLD' TO QFABRIC


SSL VPN Firewall IPSec VPN

IPS

L2 Switch

L2/L3 Switch

L2/L3 Switch

L2/L3 Switch

LEGACY DATA CENTER NETWORK INFRASTRUCTURE

WAN Edge Router

WAN Edge Router

Servers + Storage

L2/L3 Switch

Hard to manage

STP in a flat L2

access network

Security Sprawl

WAN Edge

Core Tier

Aggregation

Tier

Access

Tier


MODERN DATA CENTER NETWORK (PRE FABRIC)

Centralized

Virtualized

services

Extend Virtual Private LANs

with MPLS

WAN Edge

Collapsed

Core/Aggregation Layer

Access

Layer

GbE servers

Spanning tree removed

L2 access with LAGs

Or Routed Access

STP


KEY CHALLENGES WITH LEGACY DATACENTER

- Spanning Tree used to break loops with different flavors of it:

Standard: STP/RSTP/MSTP

Proprietary: PVST+/RPVST+

- Local routing and Services deployed on different aggregation layers (FHR ?)

- Multiple PODs routed by Core layer (where will routing be at the end of the migration?)

- VLAN space overlap ?

- Routed access ?

Need to interconnect multiple L2 domains, and avoid loops !

Need to deal with routing



IPS

L2 Switch

L2/L3 Switch

CONNECTION TO THE DISTRIBUTION LAYERS

Servers + Storage

Aggregation

Tier

Access

Tier

Node-0 Node-1 Node-2 Node-3

NW-NG-0 RSNG-1

Start with1 distribution layer, connect it to NNG

L2 loop => xSTP must run on NNG


CONNECTION TO THE DISTRIBUTION LAYERS

Dist 1

Dist 2

Qfabric

(NNG)

Enable Spanning tree on Qfabric (disabled by default) – New with JUNOS 12.2 !

Choose the appropriate flavors: RSTP/MST and/or VSTP in proprietary environment

Qfabric is One switch, 1 bridge-id

No specific Qfabric requirements regarding root placement

Most xSTP code on NNG is inherited from EX, proven/robust implementation

Interoperability reports and configuration cookbook (EX)

RSTP (Juniper) / PVST+ (Cisco)

MSTP (Juniper and Cisco, using the 802.1s specification)

VSTP (Juniper) / PVST+ (Cisco)

MSTP (Juniper) / PVST+ (Cisco)

MSTP (Juniper) / Rapid-PVST+ (Cisco)

http://networktest.com/jnpriop/jnpriop.pdf

http://networktest.com/jnpriop/cookbook.pdf

http://networktest.com/jnpriop/jnpriop.pdf

http://networktest.com/jnpriop/cookbook.pdf


VERIFYING STP OPERATIONS

root@qfabric> show spanning-tree interface

Spanning tree interface parameters for instance 0

Interface Port ID Designated Designated Port State Role

port ID bridge ID Cost

NW-NG-0:ae0.0 128:1 128:1 4096.78fe3d5c5d76 1000 FWD ROOT

root@qfabric> show spanning-tree bridge

STP bridge parameters

Context ID : 0

Enabled protocol : RSTP

Root ID : 4096.78:fe:3d:5c:5e:3a

Root cost : 1000

Root port : NW-NG-0:ae0.0

Hello time : 2 seconds

Maximum age : 20 seconds

Forward delay : 15 seconds

Message age : 1

Number of topology changes : 2

Time since last topology change : 161 seconds

Topology change initiator : NW-NG-0:ae0.0

Topology change last recvd. from : 78:fe:3d:5c:5e:3a

Local parameters

Bridge ID : 61440.f8:c0:01:f1:f0:02

Extended system ID : 0

Internal instance ID : 0

Node-0 Node-1

NW-NG-0

xe

-0/0

/24

xe

-0/0

/24

NW-NG-0:ae0.0


NEXT, CORE/WAN LAYER….

NW-NG-0

WAN Edge Router

WAN Edge Router

WAN Edge

or Core

Routed interfaces to WAN Edge/Core

Setup the NNG with OSPF or BGP

Well known ‘Junos’ RP

…

protocols {

ospf {

area 0.0.0.2 {

interface vlan.1100;

interface vlan.1101;

interface NW-NG-0:ae0.0;

}

}

}


CORE/WAN LAYER USING EIGRP 1/2?

The following can be used with Qfabric:

Overlay Model

Configure OSPF on top of EIGRP

Both IGPs running at the same time (legacy side)

Admin preference/distances keep protocols separated


CORE/WAN LAYER USING EIGRP 2/2 ?

Redistribution Model

Little more complex

Redistribution EIGRP<-> OSPF

More details available here:

http://www.juniper.net/us/en/community/junos/training-certification/day-one/networking-technologies-series/migrating-

eigrp-to-ospf/

http://www.juniper.net/us/en/community/junos/training-certification/day-one/networking-technologies-series/migrating-eigrp-to-ospf/

















L2 Switch

MIGRATION (ACCESS LAYER)

Servers + Storage

Access

Tier

Node-2 Node-3

RSNG

Qfabric and legacy network already interconnected @L2 in distribution layer

Next, connect access layer switches to RSNG with LAGs

STP free network !


MIGRATION (ACCESS LAYER) – RECOMMENDATIONS

- Use LACP (protects against miswiring)

- Enable xSTP on Qfabric

Edge node will send BPDU out for loop detection

BPDU-block set on all SNG/RSNG interfaces

- Disable STP on the switch uplink

Qfabric will bring down the link if BPDU are received (default setting)

Enable err-disable auto-recovery

- Set Storm Control to protect against storm ‘behind’ the Fabric

- Enable LLDP (makes it easier to operate !)

- Same recommendation applies to Bladecenters with embedded switches!

Node-2 Node-3

RSNG


MIGRATION (SERVICES)

Services clusters can be connected to SNGs or RSNGs (if no routing within Qfabric)

Same recommendations as before applies for dual homing (LACP, xSTP etc.)

Move one member of the cluster first, check, then the other one

(Qfabric = High bandwith, Low latency => Moving devices is ‘transparent’, but check the

other side…)

SNG

Cluster

SNG RSNG

Cluster

NNG

Cluster

L3 L2


AND FINALLY, L3 FHR MIGRATION TO QFABRIC

Traditional Network

VRRP is required for gateway redundancy for host device

Increase gateway availability by eliminating a single point-of-failure

QFabric

"Single device" behavior (RVI)

Every node-group is capable of doing L3

VRRP is no longer needed in QFabric deployment

2 step process: disable VRRP/HSRP on existing network, and enable the routed

interface on Qfabric

May need some cleaning of arp entries…



IPS

L2 Switch

L2/L3 Switch

L2/L3 Switch

L2/L3 Switch

END RESULT

WAN Edge Router

WAN Edge Router

Servers + Storage

L2/L3 Switch

Hard to manage

STP in a flat L2

access network

Security Sprawl

WAN Edge

Core Tier

Aggregation

Tier

Access

Tier


STORAGE


SOLUTIONS FOR NETWORKING STORAGE

Array Controller (incl.

RAID)

HDD HDD HDD

LUNs


RAID)

HDD HDD HDD

HDD HDD HDD

SERVER

Applications & Operating System

File System

HBA

SERVER


File System

HBA

SERVER


File System

HBA

“Fabric”


RAID)

HDD HDD HDD

LUNs


RAID)

HDD HDD HDD

HDD HDD HDD

SERVER


NIC

SERVER


NIC

SERVER


NIC

“Network”

File System

STORAGE

ARRAY

Filer Server

NAS

Filer

Storage Area Networks (SAN) Networks Attached Storage (NAS)

“Blocks” “Files”

iSCSI (Internet SCSI)

Fibre Channel (FC)

Fibre Channel over Ethernet (FCoE)

Network File System (NFS)

Common Internet File System (CIFS)

IP Traffic - “Just” another flow Qfabric with specific CoS requirements


FCOE PROTOCOLS

Ethertype = 8914h

Ensures device discovery

Facilitates FLOGI process

Resolves FC Ethernet addressing requirements

Ethertype = 8906h

The data plane protocol

Provides for the simple encapsulation of FC frames

in Ethernet frames

Lossless Ethernet

FC control plane unchanged... FCoE & FC devices interoperate

FCoE is, in reality, two distinct protocols

FCoE information transfer… FCoE Initiation Protocol (FIP)…


WHY WE NEED TO ENHANCE ETHERNET

Ethernet/IP network

Is a “lossy” network – dropped frames require upper layer protocols (e.g. TCP retransmit) for recovery

Network congestion is handled by dropping frames – uses upper layer protocols (e.g. TCP windowing and slow start) to manage

Inbuilt Layer 2 mechanisms (IEEE 802.1x PAUSE) not suitable for converged networks – all traffic on the link is stopped

Storage I/O protocols (e.g. SCSI)

Requires a “lossless” connection – normal SCSI timeout and recovery is 30 seconds or more – unacceptable for business applications

Fibre Channel specifies excellent BER and mandates buffer credit scheme to eliminate frame discards due to congestion


IEEE DCB – DATA CENTER BRIDGING

Priority Flow Control (PFC) ✔

Enable multiple traffic types to share a common Ethernet with independent flow control

Allows some traffic to be given a lossless service, while other traffic uses traditional Ethernet procedures

Enhanced Transmission Selection (ETS) ✔

Enable consistent management of QoS at the connection level

Designed to facilitate multiple services sharing a common connection

Data Center Bridging Exchange Notification (DCBX) ✔

Link level Management protocol for enhanced Ethernet capabilities

Leverages 802.1AB Link Level Discovery Protocol (LLDP)

Quantative Congestion Notification (QCN)

Designed to allow upstream signaling of network congestion for policy enforcement

IEEE 802.1 Data Center Bridging (DCB) – Qfabric & QFX support


Ethernet

QFABRIC AS FCOE TRANSIT SWITCH

FC QFabric

FCoE Transit Switch FC/FCoE Switch

(3rd Party)

FIP

ACLs

Created by

“FIP Snooping”

Function

Security & Visibility

DCB

Port

VF_Port

F_Port DCB

Port N_Port

Ethernet

VN_Port

VF_Port

FCF

VN_Port

DCB

Port


OVERVIEW - FCOE/FC GATEWAY OPERATION

FIP

Allows the servers to discover the Gateway or FC/FCoE Fabric (through an L2 cloud) and

establish a virtual connection of N to F ports FCoE VLAN Discovery

Enode Discovery and maintenance

FIP FLOGI processing – VF Port instantiation

FCoE Virtual Link Initialization

FCoE Virtual Link Maintenance

NPIV Proxy

The gateway ‘pretends’ to the servers to be the FCF (FC Fabric), and ‘pretends’ to the FC fabric

to be a bunch of physical & virtual servers

(FC) FLOGI to FC fabric

(FIP) FLOGI/FDISC (FC) FDISC

Load balancing


FCOE-FC GATEWAY (QFABRIC NODE // QFX3500)

2/4/8 Gbps FC uplinks to SAN (F_Port)

Uses NPIV proxy (no administration overlap // little interop challenges // well-known model)

Multiple SAN switch hops supported

3rd party Fibre Channel switch required

Provide fabric services

Works with most existing SAN switches from Brocade, Cisco etc.

No change to exist SAN – same management tools etc.


FC

Switch NPIV

Proxy

Ethernet

JUNIPER SOLUTIONS – FCOE-FC GATEWAY

FC

QFX 3500

Or

Single QFabric Node

FCoE-FC Gateway

FC Switch

(3rd Party)

F_Port N_Port

FC

VN_Port

F_Port N_Port

VN_Port

DCB

Port VF_Port

FDISC

FDISC


JUNIPER SOLUTIONS – TRANSIT + FCOE-FC GATEWAY

Ethernet (CEE) FC

QFabric

FCoE Transit Switch FC Switch

(3rd Party)

QFX 3500

FCoE-FC Gateway

FIP

ACLs

Created by

“FIP Snooping”

Function

DCB

Port

VN_Por

t

VN_Port

DCB

Port

FC

Switch NPIV

Proxy F_Port N_Port F_Port N_Port

DCB

Port VF_Port

FDISC

FDISC


STORAGE ECO-SYSTEM & QUALIFICATION

QFX3500

FCoE Transit Switch

FCoE-FC Gateway

QFabric

FCoE Transit Switch

FCoE-FC Gateway

Operating

System Hyper

Visor Server

Platform

CNA FCoE/FC

SAN

Switches

Embedded

Switch

Virtualization

Appliance

Optics

Cables

Disk &

Tape

Check our Qfabric storage interoperability guide


EMC QUALIFICATION !

http://www.emc.com/interoperability




FC SWITCH INTEROPERABILITY


CNA INTEROPERABILITY


STORAGE ARRAYS QUALIFICATION


CONCLUSION


QFABRIC DEPLOYMENT - LESSONS LEARNT

Datacenters are extremely sensitive and complex environments

Need to carefully Plan, Design, Test & Implement

Our PS teams together with our partners no only bring technical

knowledge, but also our expertise in projects methodology

This is the best way we de-risk projects, meet deadlines and control

cost


QFABRIC– DEPLOYMENT EXAMPLE


Firm Fix Price

DC JUNIPER LED PS PACKAGE: OVERVIEW

High-Level Design Low Level Design Validation and

Testing Network Implementation

Low-Level Design Strategic Network Consulting Optional:

Network Review/Optimization

Technical Workshop

QFabric Bring-Up Service

Network Implementation

Planning

Migration Planning + Validation

(IF Migration)

First Migration support (IF

Migration)

Junos Space Integration


Firm Fix Price

DC PARTNER LED PS PACKAGE: OVERVIEW

Design Workshop Migration Plan Review First Migration Support (IF

MMigration)

QFabric Bring-Up Service Low-Level Design Review Strategic Network Consulting Optional:

Network Review/Optimization

High-Level Design Review Junos Space Integration


THEN, YOU HAVE A BEST IN CLASS NETWORK SOLUTIONS FOR YOUR DATA CENTER

Servers, Virtual Machines

Pooled Ethernet Storage iSCSI / NAS

FC Storage

SRX HA

MX Dana Center Connectivity

L3VPN

• Best of the breed platforms

• Single JUNOS

• Optimized L2, L3, L4-7 services

delivery

• Physical & Virtualized

Fabric

SRX: L4-7

Services Complex

Any port to any port L2/L3

connectivity

IP VPLS

vGW: Hypervisor level networking

Questions ?

THANK YOU !

Don’t forget: You can copy-

paste this slide into other

presentations, and move or

resize the poll.




resize the poll.




resize the poll.

Documents

MIGRATION TO A FUTURE PROOF ARCHITECTURE - … · 2012-10-05 · RVI, routing and advance service (MPLS/VPLS) configuration on MX RVI, routing + Zone/Policy configuration on SRX L2/VLAN