Ally: OS-Transparent Packet Inspection Using Sequestered Cores

Jen-Cheng Huang 1, Matteo Monchiero2, Yoshio Turner3, Hsien-Hsin

1Georgia Tech 2Intel Labs 3HP Labs

Deep Packet Inspection (DPI)

Data Center

MiddleBoxes

Intrusion Detection

ContentInsertion

TrafficClassification

Internet

Deployment of Packet Processing Services

Problem

Internet

Data Center

MiddleBoxes

Local Traffic is growing in importance…

But The traffic within the data center is not inspected!

Approach

“Co-locate” DPI with the server

DPI appliance

Server

Leverage abundant CPU resources

Leverage existing management interfaces on servers, e.g. HP iLO

Compatible with heterogeneous architecture, e.g. on-chip accelerators

Requirements

• Transparency– Independent to the server’s software stack

• Efficiency– Low overhead packet interception

• Isolation– Resistant to attacks

ETTM: a scalable fault tolerant network manager. C. Dixon et al. NSDI ‘11

Related Work

Transparency

Hypervisor Overhead

Hypervisor Vulnerability

Virtualization Support for DPI deployment

Hypervisor

DPI VM

Guest VM

Virtualized Platform

Processors

Unprivileged Partition

Multi-core processor

Privileged Partition

Ally Architecture

Software Stack (OS + Applications)

Software Stack (DPI Application)

core core core core core

NIC Traffic

Outline

• Introduction & Motivation• Architecture

– Overview– Multicore Partitioning– Packet interception

• Evaluation• Conclusions

Northbridge

MMUMMU

ServiceProcessor

Memory Controlle

Interconnect

IOMMUInterrupt

ExternalNetwork

Main Memory

LastLevelCach

Management Network

Baseline Architecture

Northbridge

MMUMMU

ServiceProcessor

Memory Controlle

Interconnect

IOMMUInterrupt

ExternalNetwork

Main Memory

LastLevelCach

Management Network

Ally ArchitectureUnprivileged

partitionPrivileged partition

Outline

Multicore Partitioning

Invisible

Core SequestrationModify the BIOS to hide privileged core information from the OS

BSP core - the first core that boots AP cores - the other cores IPI - Inter-processor interrupts

OS retrieves cores information

Core InfoTable

Wakeup IPI Update

Ally Booting Procedure:

AP DPIEngine DPI core waits for

IN/OUT packetsInitialize

…...

Memory Protection

TLB Miss Handler

CR3BoundaryRegister

Page Table

Range Checking

Main Memory

Privileged partition

Unprivileged partition

MMUUnprivileged Core

Partition the memory into two physically contiguous regions

TLB Miss

TLB Fill

Outline

Packet Interception

NIC Traffic

Packet InterceptionVirtualization of the Descriptor Queues

OS memory

Descriptor queues replicated

DPI memory

Only one copy of the packet buffers

Descriptor queues

Packet Interception

• Virtualization of the Descriptor Queues– Device independent, software independent– No copying on packet buffers

• Processor and NIC communication– Queue manipulation uses Memory Mapped IO (MMIO)

accesses– NIC event notification uses Interrupt

MMIO redirection

MMU detects specific MMIO addresses

MMU redirects RW to a reserved region in DPI memory

MMU sends IPI to DPI core

DPI memory

DPI core

OS core

R/W redirection

Load/store

Ally Hardware Properties

• Simple extensions to existing hardware components

• No impact expected on critical timing paths

• Compatible with virtualization support (Intel VT-x/EPT, AMD SVM/NPT)

Outline

Evaluation

Full system emulationQEMU

Core sequestration

HW changes

Real machine prototype Hardware– Intel Core 2 duo 2.66 GHz with 1 Gbit Intel NICBenchmarks– Netperf– SPECwebSystems– Ally, Linux and Xen

System Configurations

Queue Virtualization

NIC Driver

Kernel

Netperf/Specweb

DPI core OS coreHW

SW NIC Driver

Kernel

Netperf/Specweb

DPI core OS coreHW

IP queue

Ally Linux

System Configurations

Hypervisor

Dom0 Kernel

Netperf/Specweb

DPI core OS coreHW

DomUKernel

Netperf CPU Usage

SPECweb CPU Usagecy

Outline

Conclusions

Ally: a framework for transparent deployment of packet inspection appliances

Ally uses a set of simple HW/FW extensions enable reliable multicore partitioning and efficient packet inspection

Ally is fully compatible with new virtualization technology as well as heterogeneous architecture

Thanks

Throughput

DPI using Network Processor

Conventional Architecture

core core cores cores cores

Transmission Path

Receive Path

Integrated Northbridge

DPI core

Interface

DPI core

Interface

OS core

Interface

OS core

Interface

Platform Controller Hub

Memory Controll

On chip interconnect

Processor

IOMMUPCIe ctrlInterrupt

BIOSNetwork

Main Memory

DMI Ctrl

OS core

Interface

Unprivileged partition Privileged partition

DPI core

Interface

Last Level Cache

IOAPIC

Management NIC

Service Processo

Management Network

DPI core

Interface

DPI core

Interface

OS core

Interface

OS core

Interface

Memory Controll

Processor

BIOSNetwork

Main Memory

DMI Ctrl

OS core

Interface

DPI core

Interface

Last Level Cache

IOAPIC

Management NIC

Service Processo

Management Network

MMU Modification – Memory Protection

TLB Miss Handler

CR3Special_re

Page Table DPI core boundary register

phys_addr >

special_reg ?

Main Memory

Memory Protection Procedure

TLB Miss HandlerTLB miss

Virtual Address

CR3Special_re

phys_addr >

special_reg ?

Main Memory

Memory Protection Procedure

TLB Miss HandlerTLB miss

Virtual Address

TLB fill

CR3Special_re

phys_addr >

special_reg ?

Main Memory

Memory Protection

Invisible

DPI core

Interface

DPI core

Interface

OS core

Interface

OS core

Interface

Memory Controll

Processor

BIOSNetwork

Main Memory

DMI Ctrl

OS core

Interface

DPI core

Interface

Last Level Cache

IOAPIC

Management NIC

Service Processo

Management Network

DPI core

Interface

DPI core

Interface

OS core

Interface

OS core

Interface

Memory Controll

Processor

BIOSNetwork

Main Memory

DMI Ctrl

OS core

Interface

DPI core

Interface

Last Level Cache

IOAPIC

Management NIC

Service Processo

Management Network

MMU Modification – MMIO Redirection

Redirection BitPhysical Page

TLB Miss HandlerCheck

uncacheable address map

Redirection

Physical Address

Remapped Address

MMIO Redirection – TLB Miss

• On a TLB miss, the TLB miss handler does the page table walk

Virtual Address

TLB missTLB Miss Handler Page Table Lookup

• The TMH checks if the resulting physical address falls in an uncacheable page and hence potentially a MMIO page

TLB Miss Handler Physical AddressCheck

• If the page is uncacheable, the TMH looks up the redirection table to check if any address in this page needs to be redirected

Redirection

Physical Address

Remapped Address

Physical Address

• If any address in the page needs to be redirected, the TMH sets the redirection bit in addition to fill the TLB

TLB fill

Redirection

Physical Address

Remapped Address

MMIO Redirection – TLB Hit

• On a TLB hit, if the redirection bit is set, the MMU looks up the Last Level Cache (LLC) used to cache translations in Redirection Table

Redirection Bit

Physical Page

Offset

Physical Address

Virtual Address

Physical Address

Remapped Address

• If a translation is found, the MMU returns the translated address and sends IPI to privileged cores.

Redirection Bit

Physical Page

Physical Address

Remapped Address

Translated Address

Generate IPI

Physical Address

• If the LLC misses, then Redirection Table Lookup is performed

Redirection Bit

Physical Page

Physical Address

Remapped Address

Redirection Table LookupPhysical

AddressMiss

Interrupt Unit Modification

DPI core

OS core

Interrupt Unit

If Source == NIC, Redirect Interrupt

• When NIC raises an interrupt, The interrupt Unit redirects the interrupt to DPI core

Interrupt Redirection

DPI core

OS core

Interrupt Unit

Interrupt

• After the NIC interrupt is handled, DPI core sends an IPI to OS core mimicking NIC interrupt

Interrupt Redirection

DPI core

OS core

Interrupt Unit

Summary of Hardware Modifications

Unit Description Purpose

OS-core MMU

Prevent memory accesses to DPI memory from OS-core

Protection

Redirect MMIO accesses to DPI memory from OS-core and interrupt DPI core

Packet Interception

IOMMU Prevent non authorized DMA to DPI Memory Protection

IOAPIC Redirect NIC interrupts to DPI-core Packet Interception

All Units Protected configuration registers Protection

Functional Evaluation

Full system emulation• QEMU• Validate Hardware and Firmware Changes

DPI core Usage

SPECweb Cache Misses

Memory Protection

Invisible

How?Modified MMU

Challenges

- Make privileged partition protected and invisible from the unprivileged partition- Core Sequestration- Memory Protection

- Intercept packets efficiently- Packet Interception

Ally System

kernel

NIC Traffic

Queue Virtualization

NIC Driver

Other Apps

DPICore

Linux System

kernel

NIC Traffic

IP queue

NIC Driver

Other Apps

Core Core

Xen System

NIC Traffic

IP queue

Hypervisor

Other Apps

Core Core

Ally: OS-Transparent Packet Inspection Using Sequestered Cores

Documents

The Commodification of Manually Sequestered Plastic · The Commodification of Manually Sequestered Plastic “We envision a Transition from plastic in our Homes, Communities and Companies

Our Russian Ally

Ally Krowner

Measuring the Total and Sequestered Organic Matter … SOM was measured by optimized loss-on-ignition (LOI) of dried soil samples at 650 C for 12 h. Sequestered SOM is represented

SEQUESTERED PARALYTIC SHELLFISH POISONING · 2015-03-04 · SEQUESTERED PARALYTIC SHELLFISH POISONING TOXINS MEDIATE GLAUCOUS-WINGED GULL PREDATION ON BIVALVE PREY RIKK G. KVITEK

Have You Been Sequestered?—Developing Diverse Sources of Revenue

OCC Consent Order - Ally Financial, Inc., ResCap, GMAC Mortgage, & Ally Bank

ALLY FINANCIAL INC. ALLY DEMAND NOTES $12,500,000,000 ALLY FINANCIAL INC. ALLY DEMAND NOTES The Ally Demand Notes (“Demand Notes”) are designed to …

Quarantined, Sequestered, Closed: Theorising Academic

Ally: OS-Transparent Packet Inspection using Sequestered Cores · for both private Enterprise datacenters and public cloud computing datacenters, which support a fast-changing multitude

Ally Capellino

Ally List6

Ally Code of Conduct and EthicsThe Ally Code of Conduct and Ethics (the ) is designed to help Ally employees and Code representatives understand the standards of conduct that Ally

Ally Marquez,

11.total sequestered carbon stock of mangifera indica

Identities of Sequestered Proteins in Aggregates from

Ally or Acquire

Ally Croping

RESEARCH Open Access Identification of sequestered

Evaluacion Final Proyecto Ally Mama Ally Wawa