NetScaler SDX. © 2012 Citrix | Confidential – Do Not Distribute Agenda Why consolidation? Multi-tenant solutions SDX Overview Hardware Internals Performance

NetScaler SDX

© 2012 Citrix | Confidential – Do Not Distribute© 2012 Citrix | Confidential – Do Not Distribute

Agenda

• Why consolidation?

• Multi-tenant solutions

• SDX Overview

• Hardware Internals

• Performance

• Service VM

• Consolidation across security zones

• Licensing

Why consolidation?


A LOT of Different Applications


Applications Have Different Owners

DesktopAdmin

NetworkComms

Collaboration

Collaboration

Commerce

CommerceFinance

Finance

Manufacturing

Sales/Service AdministrationCommerceManufacturing

LoBSpecialists

LoBSpecialists

LoBSpecialists

LoBSpecialists

Commerce

Sales/Service

Sales/Service


Each Application Has it’s Own Needs

Throughput

Functionality

Policies

Service Levels


Each Application Has it’s Own Lifecycle

Maintenance windows

Infrastructure change frequency

Application change frequency

Desire for new ADC functionality


Network Itself Can Drive Further Sprawl

Segmentation driven by compliance

Hierarchical network topologies


Exte

rnal

DM

Z

Inte

rnal

DM

Z

Inte

rnal

Lab

Different Apps and Networks


SLAs after Consolidation: Two Issues/Questions

CapacityHow much performance can a single instance offer?

IsolationCan I get the needed data and management plane isolation?

Multi-tenant solutions


Multi-tenant ADC

• All tenants share a single entity

• Rate limits, RBA and ACLs partition the instance

• Partitions are NOT fully isolatedᵒ No CPU/Resource isolationᵒ No version independenceᵒ No life cycle independenceᵒ No HA independence

ADC

Par

titio

n 2

Par

titio

n 3

Par

titio

n 1

Par

titio

n 4


Multi-tenancy - Virtual ADCs

• Hypervisors

• Each tenant gets a virtual ADC

• Brick-wall partitioning between tenants

• Good isolation

• Performance doesn’t scale

Hardware

Virt

ual A

DC

Virt

ual A

DC

Virt

ual A

DC

Virt

ual A

DC

Hypervisor


Packet Flow

• RXᵒ NIC receives a packetᵒ vSwitch forwards the packet to the

destination ADC ᵒ ADC processes the packet

Virtual ADC

HypervisorvSwitch

1

23

4

• TX○ ADC transmits a packet ○ vSwitch receives the packet ○ vSwitch transmits the packet on NIC

© 2012 Citrix | Confidential – Do Not Distribute© 2012 Citrix | Confidential – Do Not Distribute#CitrixSynergy

NetScaler SDX

• NetScaler Hardwareᵒ Intel Processorsᵒ SR-IOV capable NICs

• XenServerᵒ CPU Virtualizationᵒ IO Virtualization

• ServiceVMᵒ Management console

• NetScaler VPXsNetScaler Hardware

N

etS

cale

r V

PX

N

etS

cale

r V

PX

N

etS

cale

r V

PX

XenServer

Ser

vice

VM


New NetScaler SDX Multi-tenancy Approach

• Complete instance per tenantᵒ Memory, CPU isolationᵒ Separate entity spacesᵒ Version independenceᵒ Lifecycle independence

• Completely isolated networks

• Single license per appliance provides system throughput limits and max number of instances

• Meets performance and scalability requirements

Up to 40Tenant Instances


Multiple Instances In A Single Platform

• Complete isolation

• Complete independence

• Segmentation w/in instances


IO Virtualization

• PCI SR-IOV, Intel VT-d

• Physical Function (PF) and

Virtual Function (VF)

• Assign VF to a VM

• IOMMU

• Efficient sharing of resources

• NICs support SR-IOV

Switch

RXQueue

TXQueue

MAC & VLAN Filters

RXQueue

TXQueue

MAC & VLAN Filters

VF0 VF1

NIC

© 2012 Citrix | Confidential – Do Not Distribute© 2012 Citrix | Confidential – Do Not Distribute22#CitrixSynergy

IO Virtualization - NIC

• VFᵒ RX and TX queuesᵒ MAC addressesᵒ VLAN Filters

• RXᵒ MAC filtering – phase 1ᵒ VLAN filtering – phase 2ᵒ Queue the packet if both are passed

• TXᵒ NIC fetches the packet directly from

TX queue and transmits it

• No Hypervisor Involvement

Switch

RXQueue

TXQueue

MAC & VLAN Filters

RXQueue

TXQueue

MAC & VLAN Filters

VF0 VF1

NIC


Virtualization layer

SDX: Multi-tenant NetScaler Appliance

0/1 0/2 1/1 1/71/2 1/3 1/4 1/5 1/6 1/810/1

10/2

10/3

10/4

ServiceVM

Net

Sca

ler

2

Net

Sca

ler

3

vSwitch Net

Sca

ler

1

Management plane for the entire device

Instances are separate VMsMultiple management

networks

Data plane uses SR-IOV


Resource Isolation

Dedicated/Shared ResourcesCPU & Network

Dedicated ResourcesMemory & SSL


Fine grained CPU allocation

Core 7

Core 5

Core 11

Core 1

Core 3

Core 9

Core 8

Core 6

Core 12

Core 2

Core 4

Core 10

Core 19

Core 17

Core 23

Core 13

Core 15

Core 21

Core 20

Core 18

Core 24

Core 14

Core 16

Core 22

CPU 1

CPU 2

VPX 1

VPX 2 VPX 3, 4


• RAM is a hard allocation – no sharing

• SSL offload is a hard allocation – no sharing

• Data plane CPU can be a hard allocation

Resource Isolation


• Version management is done at instance level

• HA is done at the instance level

Lifecycle Management Isolation


Instance High Availability


• Each instance is it’s own kernel, soᵒ Gets its own connection tablesᵒ Gets its own routing tablesᵒ Gets its own IP stack

• Strong isolation of data traffic on data plane

• Strong isolation of management traffic on management plane

Network Isolation


• Ability to have multiple management networksᵒ Separate network for ServiceVM and NSIPsᵒ Separate networks for different NSIPs

• Very strong data plane isolation optionsᵒ Dedicate interfaces to instancesᵒ Share interfaces with VLAN filtering

Data and Management Plane Isolation Summary


NetScaler SDX: Platform for Evolution

Throughput

Function

Density

2000 2005/06

Throughput

Density

NS-S NS-E NS-P

Density

NS-S NS-E NS-P

20 Gbps

35 Gbps

50 Gbps

NS-S NS-E NS-P

20 Gbps

35 Gbps

50 Gbps

5 20 40

2009 Early - 2011

NetScaler Editions Pay/Grow SDX


System At-A-Glance

SDX-17500Per System•Service VM OS•Up to 16 instances•20Gbps throughput•16 SSL cores•VPX uses software RSS only




System At-A-Glance

SDX-11500Per System•Service VM OS•16 instances•8Gbps throughput•VPX uses software RSS only






System At-A-Glance





Hardware Internals


NetScaler VPX I/O Limitation on XenServer

Xen

Domain0

Bridge

Drivers

NetScaler VPX

VF Driver

NetScaler VPX

VF Driver

NetScaler VPX

VF Driver

NetScaler VPX

PV Driver

• NetScaler traffic goes through dom0

• XenServer dom0 does actual RX/TX over the physical NIC

• XenServer becomes a bottleneck (<3Gbps)*

* With Release 9.2.nc


NetScaler SDX with SR-IOV(Single Root I/O Virtualization)

XenServer

Domain0

Drivers

NetScaler VPX

VF Driver

NetScaler VPX

VF Driver

NetScaler VPX

VF Driver

NetScaler VPX

VF Driver

• Virtual machines can communicate directly with virtual NICs (bypassing dom0)

• SDX can achieve near native performance

• VPX VMs tend to be network I/O bound rather than memory or CPU bound, making VPX VMs ideal candidates to take advantage of SR-IOV.


SR-IOV Advantages

• SR-IOV is a PCI device virtualization technology that allows a single PCI device to appear as multiple PCI devices on the physical PCI bus: the real physical device is called physical function (PF) while the others are called virtual functions (VF).

• The XenServer hypervisor can directly assign one or more of these VFs to a virtual machine using Intel VT-D technologyᵒ The guest can use the VF as any other directly assigned PCI device.

• Assigning one or more VFs to a virtual machine allows the virtual machine to directly exploit the hardware without any mediation by the hypervisor.

• This means better performance and scalability because it has very little or no impact on dom0.


Intel 82599 NIC Virtual Function Driver Capabilities

• Supported • Not Supported

• Link up/down status for HA environments

• Tagged VLANs

• IPv6 on VPX instances

• Manual Link Aggregation

• Speed/duplex/flow control

• LACP

• IPv6 (on Service VM)

• Hardware Request Side Steering (RSS)All instances

must be configured similarly; share a physical interface


VF Driver Provides Network Isolation w/VLANs

• Per instance network isolation• Traffic is sent to intended instance• Isolation enforced at the NIC

• Full instance isolation• Separate routing domain• Independent routing, IP stack• Independent connection table, ACLs, etc.

Supported Configurations


SDX High Availability

• Instance-level HAᵒ Stateful connection failover ᵒ HA per-instance within applianceᵒ HA per-instance b/w appliances

• Limitations are:ᵒ Identical VLAN IDs between HA pairs

• Requires –trunk option to tag the heartbeat packets with the VLAN ID

ᵒ VMAC not supportedᵒ Active/Active capability targeted for future release

Active systems can exist on both

devices


Link Aggregation Configurations

• Link aggregation will work across like speed ports (10GE)

• Link aggregation only works with ports of same type (fiber with fiber)

• Supports up to 4 channels per system

• Supports up to 8 ports per channel

• No LACP

Service VM


Service VM Overview

• Service VM is pre-provisioned FreeBSD 64-bit VM

• Service VM manages the whole appliance (XenServer is not exposed)

• Management Interfaces GUI (HTTP/S) & API (similar to NITRO in NetScaler)


Device Management

• Show System Informationᵒ Number of CPU coresᵒ Available/free memoryᵒ Version details

• Scheduling of data backup and pruningᵒ Backup of database and configuration files (last 5 versions)ᵒ Pruning of database (to keep data size in control)

• VPX Instance inventory (with system details)

• Device level stats (CPU, Memory, Stats)


Device Management (con’t)

• Port administration changes the interface speed and auto negotiation settings

• Assign management IP Address to XenServer (only SSH is allowed) for service virtual machine failure conditions

• Resources to upload files from a local system to the service virtual machine

• Event management

• Task management

• Auditing

• Tech Support for the service virtual machine and XenServer


Instance Management

• Start, stop, reboot, remove• Upgrade (single or multiple)• Running/saved config• Instance resource utilization• Audit Messages• Service VM Admin user management (without RBA)• Port Administration

ᵒ To change Interface Speedᵒ To change Auto Negotiation Settings

• Save Config on VPX • Add MIP/SNIP on VPX


Instance Provisioning

Instance loaded from XVA template repository:• Apply Memory Settings• Assign CPU cores • Apply SRIOV Virtual Functions• Assign SSL cores• Apply port/interface configuration• Install SSL Certificates• Assign NSIP, MIP, SNIP• Assign VLAN Tagging• Add pre-canned root user on VPX • Restrictions on this root user through command policy• Set Throughput and PPS settings (Rate Limiting)


Service VM Internals

• Service VM sends API calls to the VMs for management tasks.

• No CLI for Service VM.

• Memory usage is per VM and therefore per dedicated core. Other monitoring screens are system -based, aggregate usage.

• Reboot the SDX appliance (including XenServer)ᵒ When the Service virtual machine starts, it must set auto_poweron on itself through

XenServer so that when XenServer is rebooting, it can automatically start the service virtual machine.

ᵒ The service virtual machine on XenServer should contain the “Service VM” description to identify the service virtual machine in order to set auto_poweron on the virtual machine.

Consolidation across security zones


• Consolidation within a single zone

• Device admin is also admin for all instances

Simple Consolidation


NetScaler SDX-11500 Interface Topology

0/1

0/2

1/1

1/7

1/2

1/3

1/4

1/5

1/6

1/8

10/1

10/2

ManagementInterfaces

1G DataInterfaces

10G DataInterfaces

10/3

10/4


Simplest Deployment

0/1

0/2

1/1

1/7

1/2

1/3

1/4

1/5

1/6

1/8

10/1

10/2

10/3

10/4

Inst

ance

1

ServiceVM

Inst

ance

2

Inst

ance

3

Inst

ance

4

Inst

ance

5

10.1.1.x (ServiceVM and NSIPs on same network)

No sharing of data interfaces


Simplest Deployment

0/1

0/2

1/1

1/7

1/2

1/3

1/4

1/5

1/6

1/8

10/1

10/2

10/3

10/4

Inst

ance

1

ServiceVM

Inst

ance

2

Inst

ance

3

Inst

ance

4

Inst

ance

5


No sharing of data interfaces

Deployments where compliance is not a concernDeployments when all instances in the same security zone

• Instance density limited to number of physical interfaces• Data plane isolation achieved via no sharing of physical interfaces• 4096 VLANs per interface and instance


Data Plane Isolation with Shared Interfaces

0/1

0/2

1/1

1/7

1/2

1/3

1/4

1/5

1/6

1/8

10/1

10/2

10/3

10/4

Inst

ance

1

ServiceVM

Inst

ance

2

Inst

ance

3

Inst

ance

4

Inst

ance

5


Inst

ance

6

VLAN6 VLAN5

VLAN Filtering enabled on 10/4 interface


Data Plane Isolation with Shared Interfaces

0/1

0/2

1/1

1/7

1/2

1/3

1/4

1/5

1/6

1/8

10/1

10/2

10/3

10/4

Inst

ance

1

ServiceVM

Inst

ance

2

Inst

ance

3

Inst

ance

4

Inst

ance

5


Inst

ance

6

VLAN6 VLAN5


Need more instances than physical portsScenarios where conserving switch ports is important

• Instance density limited only by platform maximum• SDX will NOT forward VLAN5 traffic to Instance6• VLAN filtering can be enabled/disabled interface by interface


• Consolidation within a single zone

• Different admin for applications

Simple Consolidation with Delegated Administration


From One Management Network

0/1

0/2

1/1

1/7

1/2

1/3

1/4

1/5

1/6

1/8

10/1

10/2

10/3

10/4

Inst

ance

1

ServiceVM

Inst

ance

2

Inst

ance

3

Inst

ance

4

Inst

ance

5


Inst

ance

6

VLAN6 VLAN5


Device admin doesn’t want instance admins on same network as ServiceVM


To Separate Management Networks

0/1

0/2

1/1

1/7

1/2

1/3

1/4

1/5

1/6

1/8

10/1

10/2

10/3

10/4

Inst

ance

1

ServiceVM

Inst

ance

2

Inst

ance

3

Inst

ance

4

Inst

ance

5

10.1

.1.x

10.1

.2.x

Inst

ance

6

VLAN6 VLAN5


To Separate Management Networks

0/1

0/2

1/1

1/7

1/2

1/3

1/4

1/5

1/6

1/8

10/1

10/2

10/3

10/4

Inst

ance

1

ServiceVM

Inst

ance

2

Inst

ance

3

Inst

ance

4

Inst

ance

5

10.1

.1.x

10.1

.2.x

Inst

ance

6

VLAN6 VLAN5

device admin doesn’t want instance admins on Service VM network Deployments when all instances in the same security zone

• Data plane isolation achieved via either port(s) per instance or VLAN filtering• When ports are dedicated, each instance gets up to 4096 VLANs


• Consolidate across security zones

• Each security zone has its own management network

• Device admin wants to let others administer individual instances

Consolidation Across Security Zones

© 2012 Citrix | Confidential – Do Not Distribute© 2012 Citrix | Confidential – Do Not Distribute69#CitrixSynergy

Separate Security Zones

0/1

0/2

1/1

1/2

1/3

1/4

1/5

1/8

1/6

1/7

10/1

10/2

10/3

10/4

Net

Sca

ler

VP

X 1

Net

Sca

ler

VP

X 2

Net

Sca

ler

VP

X 3

Net

Sca

ler

VP

X 4

ServiceVM10

.0.2

.x

10.0

.1.x

Net

Sca

ler

VP

X 5

VLAN4 VLAN5

10.0

.3.x

Internal DMZ


Separate Security Zones

0/1

0/2

1/1

1/2

1/3

1/4

1/5

1/8

1/6

1/7

10/1

10/2

10/3

10/4

Net

Sca

ler

VP

X 1

Net

Sca

ler

VP

X 2

Net

Sca

ler

VP

X 3

Net

Sca

ler

VP

X 4

ServiceVM10

.0.2

.x

10.0

.1.x

Net

Sca

ler

VP

X 5

VLAN4 VLAN5

10.0

.3.x

Internal DMZ

Scenarios where compliance is an issueSpecifically when compliance stance requires separate management networks per security

zone

• Data plane isolation achieved via either port(s) per instance or VLAN filtering• When ports are dedicated, each instance gets up to 4096 VLANs

Licensing


NetScaler SDX Licensing

• Platform license – entitles base SDX appliance

○ Default 5 instances allowed on certain platforms

○ 5-Instance Add-On Pack license (Instance Pay-Grow)

○ Enables adding additional VPX instances, beyond the default 5

• Platform Upgrade license (Platform Pay-Grow)

○ Upgrade to higher throughput capacity on same hardware platform

• Platform Conversion license

○ Change MPX to SDX (not applicable for FIPS, 9500, 7500, 5500)


NetScaler SDX Licensing Example

• Purchased system = SDX 11500

• Apply platform license

○ up to 5 VPX instances, max system throughput 8 Gbps

• Add 3 x 5-pack instance licenses

○ add up to 20 VPX instances – max system throughput still 8 Gbps

• Apply SDX-11500-to-SDX-18500 platform upgrade license

○ max system throughput increases to 36 Gbps

Documents

NetScaler SDX. © 2012 Citrix | Confidential – Do Not Distribute Agenda Why consolidation? Multi-tenant solutions SDX Overview Hardware Internals Performance