I Know What Your Packet Did Last Hop: Using

Packet Histories to Troubleshoot Networks

Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, David Mazières, and Nick McKeown, Stanford University

NSDI 2014 Speaker:Cheng-Da Tsai

1

OutlineMotivation

Netsight architecture

Implementation

Debug tool(API + Application)

Compression

Evaluation

Scaling Netsight

Summary

2

Motivation

Provide direct evidence to diagnose network problem

Bug example:

3

Motivation

Provide direct evidence to diagnose network problem

Bug example:

4

Motivation

Provide direct evidence to diagnose network problem

Bug example:

After hours of debugging…..

5

Motivation

Provide direct evidence to diagnose network problem

Bug example:

forwarding rules were improperly update in wired switch…..QQ

6

Motivation

Use Netsight: you can only ask

“Show me all packet histories for packets to the client when the handover occurred. “

This packet go to the wrong AP.You can also check which switch flow table updated error.

7

Motivation

Packet History Definition:(3W1H)

What the packet looked like as it entered the network (headers)

Where the packet was forwarded (switches + ports)

How it was changed (header modifications)

Why it was forwarded that way (matched flow/actions + flow table).

Goal:

Complete visibility: every event that ever happened to every packet

8

Netsight architecture

9

Topology 保證正確

Implementation

Life Of a Postcard:

10

Implementation

Postcard Generation:Control Plane

Flow table state table recorder

Postcard

collector

Packet headerSwitch

id outport

version #

copy

tag

11

Implementation

Postcard Collection:Control Plane

Flow table state table recorder

Postcard

collector

All postcards for a packet to one server send by VLAN ID

12

Implementation

Postcard Collection:

Postcard

collectorhash based on flow key

(5-tuple)

13

Implementation

History Assembly:

Topo-Sort: assemble to a flow

14

Debug tool(API)

Postcard Filters: --bpf [packet description] -- dpis [switch id] --inport [port#] … EX:--bpf "ip src A" --dpid S --inport not P.

Packet History Filter: start at X: ^{{X}}

end at X: {{X}}$

go through X: {{X}}

go through X, and later Y: {{X}}.*{{Y}}

start at X, never reach Y: ^{{X}}[^{{Y}}]*$

experience a loop: (.).*(\1)

15

Debug tool(Application)

ndb:Interactive Network Debugger

netwatch:Live Invariant Monitor

netshark:Network-wide Path-Aware Packet Logger

netprof: Hierarchical Network Profiler

16

Implementation

Filter triggers:

PHFPHFPHFPHFPHFnotify

Application17

Compression

Compress in two places:

Before shuffling postcards to servers.

Before archiving assembled histories to disk.

18

Compression

Huge redundancy in packet header fields

19

Evaluation

Compression

20

Evaluation

Matching latency

21

Scaling Netsight

Basic Netsight (No Compress):

extract 31% traffic.

Netsight-SwitchAssist(Compress in Switch side):

extract 7% traffic.

Netsight-HostAssist(Compress in Host side):

extract 3% traffic.

22

Summary

Complete visibility: every event that ever happened to every packet is possible.

Exact traffic can be resolved by compression.

Speed of generating Postcard can be resolved by map-reduce method.

23