Ally: OS-Transparent Packet Inspection Using Sequestered Cores

1

Ally: OS-Transparent Packet Inspection Using Sequestered Cores

Jen-Cheng Huang 1, Matteo Monchiero2, Yoshio Turner3, Hsien-Hsin

Lee1

1Georgia Tech 2Intel Labs 3HP Labs

2

Deep Packet Inspection (DPI)

Data Center

MiddleBoxes

Intrusion Detection

ContentInsertion

TrafficClassification

Internet

Deployment of Packet Processing Services

3

Problem

Internet

Data Center

MiddleBoxes

Local Traffic is growing in importance…

But The traffic within the data center is not inspected!

4

Approach

“Co-locate” DPI with the server

DPI appliance

Server

Leverage abundant CPU resources

Leverage existing management interfaces on servers, e.g. HP iLO

Compatible with heterogeneous architecture, e.g. on-chip accelerators

5

Requirements

• Transparency– Independent to the server’s software stack

• Efficiency– Low overhead packet interception

• Isolation– Resistant to attacks

6

ETTM: a scalable fault tolerant network manager. C. Dixon et al. NSDI ‘11

Related Work

Transparency

Hypervisor Overhead

Hypervisor Vulnerability

Virtualization Support for DPI deployment

Hypervisor

DPI VM

Guest VM

HW

SW

Virtualized Platform

Processors

7

NIC

Unprivileged Partition

Multi-core processor

core

Privileged Partition

Ally Architecture

Software Stack (OS + Applications)

Software Stack (DPI Application)

core core core core core

NIC Traffic

8

Outline

• Introduction & Motivation• Architecture

– Overview– Multicore Partitioning– Packet interception

• Evaluation• Conclusions

9

Northbridge

MMUMMU

MMUMMU

ServiceProcessor

NIC

Memory Controlle

r

Interconnect

IOMMUInterrupt

Unit

BIOS

ExternalNetwork

Main Memory

Core

Inte

rrup

tC

ontr

olle

r

MMU

Core

Inte

rrup

tC

ontr

olle

r

MMU

LastLevelCach

e

Management Network

Baseline Architecture

10

Northbridge

MMUMMU

MMUMMU

ServiceProcessor

NIC

Memory Controlle

r

Interconnect

IOMMUInterrupt

Unit

BIOS

ExternalNetwork

Main Memory

Core

Inte

rrup

tC

ontr

olle

r

MMU

Core

Inte

rrup

tC

ontr

olle

r

MMU

LastLevelCach

e

Management Network

Ally ArchitectureUnprivileged

partitionPrivileged partition

11

Outline




12

Multicore Partitioning

NIC



core





Invisible

13

Core SequestrationModify the BIOS to hide privileged core information from the OS

BSP core - the first core that boots AP cores - the other cores IPI - Inter-processor interrupts

OS retrieves cores information

BSPAP

AP

AP

Core InfoTable

Wakeup IPI Update

Ally Booting Procedure:

AP DPIEngine DPI core waits for

IN/OUT packetsInitialize

…...

14

Memory Protection

TLB

TLB Miss Handler

CR3BoundaryRegister

Page Table

Range Checking

Main Memory

Privileged partition

Unprivileged partition

MMUUnprivileged Core

Partition the memory into two physically contiguous regions

TLB Miss

TLB Fill

15

Outline




16

NIC



core


Packet Interception




NIC Traffic

17

Packet InterceptionVirtualization of the Descriptor Queues

NIC

OS memory

Descriptor queues replicated

DPI memory

Only one copy of the packet buffers

Descriptor queues

18

Packet Interception

• Virtualization of the Descriptor Queues– Device independent, software independent– No copying on packet buffers

• Processor and NIC communication– Queue manipulation uses Memory Mapped IO (MMIO)

accesses– NIC event notification uses Interrupt

19

MMIO redirection

MMU

MMU detects specific MMIO addresses

MMU redirects RW to a reserved region in DPI memory

MMU sends IPI to DPI core

DPI memory

DPI core

OS core

IPI

R/W redirection

Load/store

20

Ally Hardware Properties

• Simple extensions to existing hardware components

• No impact expected on critical timing paths

• Compatible with virtualization support (Intel VT-x/EPT, AMD SVM/NPT)

21

Outline




22

Evaluation

Full system emulationQEMU

Core sequestration

HW changes

Real machine prototype Hardware– Intel Core 2 duo 2.66 GHz with 1 Gbit Intel NICBenchmarks– Netperf– SPECwebSystems– Ally, Linux and Xen

23

System Configurations

Queue Virtualization

NIC Driver

Kernel

Netperf/Specweb

Snort

DPI core OS coreHW

SW NIC Driver

Kernel

Netperf/Specweb

Snort

DPI core OS coreHW

SW

IP queue

Ally Linux

24

System Configurations

Hypervisor

Dom0 Kernel

Netperf/Specweb

Snort

DPI core OS coreHW

SW

Xen

DomUKernel

25

Netperf CPU Usage

26

SPECweb CPU Usagecy

cles

/req

ues

t *

10

6

27

Outline




28

Conclusions

Ally: a framework for transparent deployment of packet inspection appliances

Ally uses a set of simple HW/FW extensions enable reliable multicore partitioning and efficient packet inspection

Ally is fully compatible with new virtualization technology as well as heterogeneous architecture

29

Thanks

30

Throughput

31

DPI using Network Processor

32

NIC



core

Conventional Architecture


core core cores cores cores

33

NIC



core


Transmission Path




34

NIC



core


Receive Path




35

Integrated Northbridge

DPI core

Loca

l A

PIC

MMU

Interface

DPI core

Loca

l A

PIC

MMU

Interface

OS core

Loca

l A

PIC

MMU

Interface

OS core

Loca

l A

PIC

MMU

Interface

Platform Controller Hub

NIC

Memory Controll

er

On chip interconnect

Processor

IOMMUPCIe ctrlInterrupt

Unit

BIOSNetwork

Main Memory

DMI Ctrl

OS core

Loca

l A

PIC

MMU

Interface

Unprivileged partition Privileged partition

DPI core

Loca

l A

PIC

MMU

Interface

Last Level Cache

IOAPIC

Management NIC

Service Processo

r

Management Network



36


DPI core

Loca

l A

PIC

MMU

Interface

DPI core

Loca

l A

PIC

MMU

Interface

OS core

Loca

l A

PIC

MMU

Interface

OS core

Loca

l A

PIC

MMU

Interface


NIC

Memory Controll

er


Processor


Unit

BIOSNetwork

Main Memory

DMI Ctrl

OS core

Loca

l A

PIC

MMU

Interface


DPI core

Loca

l A

PIC

MMU

Interface

Last Level Cache

IOAPIC

Management NIC

Service Processo

r

Management Network



37

MMU Modification – Memory Protection

TLB

TLB Miss Handler

CR3Special_re

g

Page Table DPI core boundary register

phys_addr >

special_reg ?

Main Memory



38

Memory Protection Procedure

TLB

TLB Miss HandlerTLB miss

Virtual Address

CR3Special_re

g


phys_addr >

special_reg ?

Main Memory



39

Memory Protection Procedure

TLB

TLB Miss HandlerTLB miss

Virtual Address

TLB fill

CR3Special_re

g


phys_addr >

special_reg ?

Main Memory



40

NIC



core


Memory Protection




Invisible

41


DPI core

Loca

l A

PIC

MMU

Interface

DPI core

Loca

l A

PIC

MMU

Interface

OS core

Loca

l A

PIC

MMU

Interface

OS core

Loca

l A

PIC

MMU

Interface


NIC

Memory Controll

er


Processor


Unit

BIOSNetwork

Main Memory

DMI Ctrl

OS core

Loca

l A

PIC

MMU

Interface


DPI core

Loca

l A

PIC

MMU

Interface

Last Level Cache

IOAPIC

Management NIC

Service Processo

r

Management Network



42


DPI core

Loca

l A

PIC

MMU

Interface

DPI core

Loca

l A

PIC

MMU

Interface

OS core

Loca

l A

PIC

MMU

Interface

OS core

Loca

l A

PIC

MMU

Interface


NIC

Memory Controll

er


Processor


Unit

BIOSNetwork

Main Memory

DMI Ctrl

OS core

Loca

l A

PIC

MMU

Interface


DPI core

Loca

l A

PIC

MMU

Interface

Last Level Cache

IOAPIC

Management NIC

Service Processo

r

Management Network



43

MMU Modification – MMIO Redirection

TLB

Redirection BitPhysical Page

TLB Miss HandlerCheck

uncacheable address map

Redirection

Table

Physical Address

Remapped Address

44

MMIO Redirection – TLB Miss

• On a TLB miss, the TLB miss handler does the page table walk

TLB


Virtual Address

TLB missTLB Miss Handler Page Table Lookup

45


• The TMH checks if the resulting physical address falls in an uncacheable page and hence potentially a MMIO page

TLB


TLB Miss Handler Physical AddressCheck


46


• If the page is uncacheable, the TMH looks up the redirection table to check if any address in this page needs to be redirected

TLB




Redirection

Table

Physical Address

Remapped Address

Physical Address

47


• If any address in the page needs to be redirected, the TMH sets the redirection bit in addition to fill the TLB

TLB




TLB fill

Redirection

Table

Physical Address

Remapped Address

48

MMIO Redirection – TLB Hit

• On a TLB hit, if the redirection bit is set, the MMU looks up the Last Level Cache (LLC) used to cache translations in Redirection Table

TLB

Redirection Bit

Physical Page

Offset

Physical Address

Virtual Address

LLC

Physical Address

Remapped Address

49


• If a translation is found, the MMU returns the translated address and sends IPI to privileged cores.

TLB

Redirection Bit

Physical Page

LLC

Physical Address

Remapped Address

Translated Address

Generate IPI

Physical Address

Hit

50


• If the LLC misses, then Redirection Table Lookup is performed

TLB

Redirection Bit

Physical Page

LLC

Physical Address

Remapped Address

Redirection Table LookupPhysical

AddressMiss

51

Interrupt Unit Modification

DPI core

OS core

Interrupt Unit

NIC

If Source == NIC, Redirect Interrupt

52

• When NIC raises an interrupt, The interrupt Unit redirects the interrupt to DPI core

Interrupt Redirection

DPI core

OS core

Interrupt Unit

NIC


Interrupt

53

• After the NIC interrupt is handled, DPI core sends an IPI to OS core mimicking NIC interrupt

Interrupt Redirection

DPI core

OS core

Interrupt Unit

NIC


IPI

54

Summary of Hardware Modifications

Unit Description Purpose

OS-core MMU

Prevent memory accesses to DPI memory from OS-core

Protection

Redirect MMIO accesses to DPI memory from OS-core and interrupt DPI core

Packet Interception

IOMMU Prevent non authorized DMA to DPI Memory Protection

IOAPIC Redirect NIC interrupts to DPI-core Packet Interception

All Units Protected configuration registers Protection

55

Functional Evaluation

Full system emulation• QEMU• Validate Hardware and Firmware Changes

56

DPI core Usage

57

SPECweb Cache Misses

58

NIC



core


Memory Protection




Invisible

How?Modified MMU

59

Challenges

- Make privileged partition protected and invisible from the unprivileged partition- Core Sequestration- Memory Protection

- Intercept packets efficiently- Packet Interception

60

Ally System

NIC

Linux

kernel

NIC Traffic

Queue Virtualization

NIC Driver

Other Apps

Snort

DPICore

Core

61

Linux System

NIC

Linux

kernel

NIC Traffic

IP queue

NIC Driver

Other Apps

Snort

Core Core

62

Xen System

NIC

Linux

VM #0

NIC Traffic

IP queue

Hypervisor

Other Apps

Snort

Core Core

VM #1

Documents

Ally: OS-Transparent Packet Inspection Using Sequestered Cores