1

Design and Implementation of aConsolidated Middlebox

ArchitectureVyas Sekar Sylvia Ratnasamy Michael Reiter Norbert Egi

Guangyu Shi

Need for Network Evolution

2

New devices

New applications

Evolving threats Policy

constraintsPerformance, Security, Compliance

3

Type of appliance NumberFirewalls 166NIDS 127Media gateways 110Load balancers 67Proxies 66VPN gateways 45WAN Optimizers 44Voice gateways 11Total Middleboxes 636Total routers ~900

Network Evolution today: Middleboxes!

Data from a large enterprise: >80K users across tens of sites

Just network security$10 billion

4

Specialized boxes

Narrowinterfaces

“Point”solutions!

Increases capital expenses & sprawl Increases operating expensesLimits extensibility and flexibility

Management ManagementManagement

Key “pain points”

• Motivation

• High-level idea: Consolidation

• System design

• Implementation and Evaluation5

Outline

Network-wide Controller

Key idea: Consolidation

2. Consolidate Management

1. ConsolidatePlatform

Two levels corresponding to two sources of inefficiency

6

Consolidation at Platform-Level

7

Proxy Firewall IDS/IPS AppFilterToday: Independent, specialized boxes

Decouple Hardware and Software

Commodity hardware:e.g., PacketShader, RouteBricks,ServerSwitch, SwitchBlade

Consolidation reduces capital expenses and sprawl

Consolidation reduces CapEx

8

Multiplexing benefit = Max_of_TotalUtilization / Sum_of_MaxUtilizations

Consolidation Enables Extensibility

9

Session Management

Protocol Parsers

VPN Web Mail IDS Proxy

Firewall

Contribution of reusable modules: 30 – 80 %

Management consolidation enables flexible resource allocation

10

N1 N3

N2P: N1 N3

Process (0.4 P)

Today: All processing at logical “ingress”

Overload!

Network-wide distribution reduces load imbalance

Process (0.3 P)

Process (0.3 P)Process (P)

• Motivation

• High-level idea: Consolidation

• CoMb: System design

• Implementation and Evaluation11

Outline

Network-wide Controller

CoMb System Overview

12

General-purpose hardware:e.g., PacketShader, RouteBricks, ServerSwitch,

Logically centralized e.g., NOX, 4D

Existing work: simple, homogeneous routing-like workload

Middleboxes: complex, heterogeneous, new opportunities

Network-wide Controller

Processingresponsibilities

CoMb Management Layer

Policy Constraints

ResourceRequirements

Routing, Traffic

Goal: Balance load across network.Leverage multiplexing, reuse, distribution

13

Capturing Reuse with HyperApps

14

IDS Proxy

common

Footprint onresource

HTTPUDP

HTTPNFS

2 311

Memory

CPU

HTTP = IDS & Proxy

43Memory

UDP = IDS

NFS = Proxy

13

41

CPU

CPU

Memory

Memory

CPU

HyperApp: find the union of apps to run

Policy, dependency are implicitNeed per-packet policy, reuse dependencies!

HTTP:1+2 unit of CPU1+3 units of mem

Modeling Processing Coverage

15

HTTPN1 N3

N1 N2 N3

What fraction of traffic of class HTTP from N1 N3 should each node process?

IDS < Proxy0.4

HTTP: Run IDS < Proxy

IDS < Proxy0.3

IDS < Proxy0.3

Network-wide Optimization

16

Minimize Maximum Load, Subject to

Processing coverage for each class of traffic Fraction of processed traffic adds up to 1

Load on each node sum over HyperApp responsibilities per-path

A simple, tractable linear programVery close (< 0.1%) to theoretical optimal

No explicitDependencyPolicy

Network-wide Controller

CoMb System Overview

17

General-purpose hardware:e.g., PacketShader, RouteBricks, ServerSwitch,

Logically centralized e.g., NOX, 4D

Existing work: simple, homogeneous routing-like workload

Middleboxes: complex, heterogeneous, new opportunities

CoMb Platform

18

NIC

Policy Shim (Pshim)

Core1 Core4

IDS

…

… Proxy

Traffic

Applications

Policy Enforcer

Classification: HTTP

IDS < Proxy

Challenges: PerformanceParallelizeIsolation

Challenges: LightweightParallelize

Challenges: No contentionFast classification

Parallelizing Application Instances

19

M1 M2

PShim

App-per-core

Core3

M3

PShim

Core1 Core2

HyperApp-per-core

M2 M3

PShim

M1 M2

PShim

Core1 Core2

- Inter-core communication- More work for PShim+ No in-core context switch

+ Keeps structures core-local+ Better for reuse- But incurs context-switch- Need replicas

✔

HyperApp-per-core is better or comparableContention does not seem to matter!

CoMb Platform Design

20

HyperApp1

HyperApp2

HyperApp4

HyperApp3

HyperApp3

PShim PShim PShimPShim PShim

M1 M4 M1 M4M2 M3

Q1 Q3Q2 Q4 Q5

M1 M5

Core 1 Core 3Core 2

NIC hardware

Contention-free network I/O

Core-local processing

Parallel, core-local

Workload balancing

• Motivation

• High-level idea: Consolidation

• System design: Making Consolidation Practical

• Implementation and Evaluation21

Outline

Implementation

22

Network-wide Management

Session

Protocol

Extensible apps Standalone apps

Policy Shim

using CPLEX

Kernel mode Click

Ported logic FromBro Click

Memory mappedOr

Virtual interfaces

8-core Intel Xeon with Intel 82599 NIC

Consolidation is Practical

• Low overhead for existing applications

• Controller takes < 1.6s for 52-node topology

• 5x better than VM-based consolidation

23

Benefits: Reduction in Maximum Load

24

Consolidation reduces maximum load by 2.5-25X

MaxLoadToday /MaxLoadConsolidated

Benefits: Reduction in Provisioning Cost

25

Consolidation reduces provisioning cost 1.8-2.5X

ProvisioningToday /ProvisioningConsolidated

Discussion

• Changes traditional vendor business – Already happening (e.g., “virtual appliances”)– Benefits imply someone will do it!– May already have extensible stacks internally!

• Isolation– Current: rely on process-level isolation– Get reuse-despite-isolation?

26

• Network evolution occurs via middleboxes

• Today: Narrow “point” solutions– High CapEx, OpEx, and device sprawl– Inflexible, difficult to extend

• Our proposal: Consolidated architecture– Reduces CapEx, OpEx, and device sprawl – Extensible, general-purpose

• More opportunities– Isolation– APIs (H/W—Apps, Management—Apps, App Stack)

27

Conclusions