Netfilter:Making large iptables rulesets scale

Challenges and experiences with IPTVfrom a network point of viewOpen Source Days 2010d.6/3-2010

ComX Networks A/S

byJesper Dangaard Brouer Master of Computer ScienceComX Networks A/S

Who am I

Name: Jesper Dangaard BrouerEdu: Computer Science for Uni. CopenhagenFocus on Network, Dist. sys and OS

Linux user since 1996, professional since 1998Sysadm, Developer, Embedded

OpenSource projectsAuthor ofADSL-optimizer

CPAN IPTables::libiptc

Patches accepted intoLinux kernel, iproute2, iptables and Wireshark

What will you learn?

This talk is about network experiences with IPTVNot about: challenges to get multicast to work

Part1: about trouble shooting IPTV network problems

Part 2: about developing our own IPTV monitor probe

Future plans for an IPTV shaper

ComX: Physical surroundings

ComX Networks A/S: Fiber based solutionsvariety of services (TV, IPTV, VoIP, Internet)

Our primary customers are

Apartment buildings with end-user relationship

OEM customers, like Syd Energi

Apartment building have Ethernet switchesPort separation, private VLAN

CPE box in apartment performsservice separation into VLANs

What is this talk NOT about

Lot of challenges to implement multicast routingIts not straight forward, BUT its not the focus of this talk.

Shortly, pitfalls to watch out for:It can be difficult to:get different vendors multicast to work together

get even the same vendorWatch out for firmware upgrades...

E.g: New firmware (on Foundry FWSX 424) drops packets, MC in HW.

Also Remember:Apartment buildings Ethernet switches, proper config.

CPE equipment needs IGMP snooping, more Settop boxes

(BCU-26 currently have 2223 active equipment, cust:1176, iptables chains:7551 rules:32015)

(Amager POP, Internet customers:4892 with active equipment:8615.)

The focus: Scalability

What happened when more channels were addedHigh-Definition channels

More than 120 IPTV channels

Approx 600 Mbit/s multicast traffic.

What happens to a realtime serviceon a loaded ethernet switch based network?

TALK: First I'll focus on the routing performance issue I'll come back to slow rule changes later

Basics: MPEG2 TS packets

Each IP-packet contains 7 Transport Stream Packets/Frames

The Continuity Counter is very small values 0-15Thus, an exact drop counter is hard as wrap around occurs quickly

History: What was the problem?

Customers were experiencing bad quality,due to packet drops

Traffic had reached 600Mbit/sDrops were worse on loaded links.

But dedicated links also showed significant drops60 percent loaded link should not have drops

Finding the cause of packet dropsTurned out to be a harder task than expected...

History: Measure the problem

First objective: Be able to measure the problemUsed "IP-probe" from BridgeTech, to detect/see drops.Which is an IPTV analyzer applianceinternally runs Linux and has a special FPGA chip

Really nice tool, but very expensive!

Work under pressure: OEM Customer mad!!!Network Core team was getting desperateFoundry supportFound issues with their PIM implementationBad distribution over trunked links

Desperate clear PIM cache drills

What caused the drops?

Bursty traffic from some of the IPTV streamers.Some of the "cheap" (100k DKK) streamers, bursting

Worst: Linux boxes with VLC and old 2.6.18 kernel

Linux VLC machines easily fixedby newer kernel with high resolution timer support

The commercial streamers were harder to "fix"Closed source

Slow and expensive support

Kickoff IPTV probe/shaper project based on Linux

Important to enablehigh resolution timers and tickless OSto avoid VLC multicast bursts

Kernel config optionsCONFIG_HIGH_RES_TIMERS=y

CONFIG_NO_HZ=y

Watch the kernel log messages for:

Switched to high resolution mode on CPU 0orCould not switch to high resolution mode on CPU 0

Hint slide: Enable Linux Highres

How did we locate the problem?

BridgeTech probe tool, very helpful, showing the dropsbut could not show the burstiness directly

One of the Linux based streamers,Alternating between 6 Mbit/s and 15 Mbit/s streams

All channels experienced more drops during 15 Mbit/s streams

Detailed analysis of traffic dumpsWireshark was used to show burstiness

Wireshark IO-graph with resolution < 1 ms

Implemented drop detect our selves in Wireshark(the power of open source)

Wireshark IO-graph

Menu: Statistics IO Graph Tick Interval 0.001

Wireshark drop detection

MPEG2 TS drop detection in WiresharkIncluded in mainline since SVN rev. 27381

Not fast enough for realtime/continuous usage

Dumping 600Mbit/s trafficStandard tcpdump not fast enoughWARN cause fake packet drops (Func: packet_rcv)

Use tcpdump/pcap with Mem Mapped IOUse minimum libpcap version 1.0.0

Function: tpacket_rcv (See /proc/net/ptype)

Summary for 1.0.0 libpcap release, October 27, 2008.

On Linux, support new tpacket frame headers (2.6.27+)

Part 1: Conclusion

Watch out for burstsPackets bursts experiencesSurprisingly bursts do cause problems on 60% loaded links

Non-bursty streams are affected by bursty-streams

On Linux: Use Highres timers and no tick

Know you switch buffer sizesdon't place low-buffer switches in backbone

Part 2: Develop our own IPTV probe

LearnedNeed to monitor the signal

Bursts can cause drops, on all channels

Bursts can occur per network segment

Monitor each network segmentApprox measurement points: 12

Price per IPTV-probe approx 80.000 DKKTotal cost: 960.000 DKK

Develop our own IPTV-probe

Project plan: IPTV probe/shaper

Develop our own MPEG2-TS analyserMilestone 1: Measure the drops DONEFirst objective: measure the problem

Milestone 2: Measure the burstsDetect potential issues, before drops occur

Milestone 3: Smooth out the burstsSolve the burst issue permanently

Milestone 1: Measure drops

Offline analysisContribute to Wirehark, mostly prototyping

Need continuous monitoring (24x7x365)Kernel module for detection (iptables)

Collector daemon (Perl)

Database design, storing events (MySQL)

Web based frontend (PHP)

Trick: use settop boxes as probes (bash)

IPTV Probe architecture

Kernel module: Efficient!

Developed iptables/netfilter module (mp2t)For realtime/continuous drop monitoring

Only uses 2% CPU with 600Mbit/s traffic

on a low power ATOM 330 CPU (1.6Ghz)

tcpdump solution used 100% and caused false drops

Implemented with RCU lockingfor parallel processing on each CPU

(requires multi-queue NICs)

Stats via /proc/ fileCollector currently polls every 10 sec

Collector

DesignOnly report on changes

Poll every 10 secCompare with previous data

Plan: Avoid pollingkernel report activity (via netlink)

Written as Perl daemon process

Drop this slide???

Web-Frontend

Just finished the web-frontendIts not pretty, but it works!

PHP developers needed on the project!

Web-frontend: Probe overview

Web-frontend: Channel view

Trick: Use settop box as probes

See what the customer seesAs many probes as customers

But only one channel at a time

Trick: The settop box runs Linuxlocal tool support asking for sync errors

install small bash scriptperiodically poll, and submit result back central

Still need probes, in the networkneed to identify the network segment

Settox boxes as probes

Open Source Status

WiresharkDrop patches accepted

PCR-clock patches, not accepted :-(

IPTV probe tool: (Is anybody interested?)Kernel module is GPL, but not released publicContact me if you want a copy...

Collector + DB design will hopefully be GPLWant the same basis for frontends

FrontendWe are looking for PHP developers...

Future?

Milestone 2: Bursts, detect issues, before drops occurCurrently down prioritized due to other tasks...

Milestone 3: Shaper, solve issueHave an implementation idea...

Will it ever happen?

Progress through cooperation?Anybody want to cooperate?Or I am a just an one-man-army ?

The End

Goodbyeand thank you for committing your patches...

Want to try the IPTV-probe?Please contact me: jdb@comx.dk

81.161.128/0/18195.135.216.0/2287.72.0.0/1682.211.224.0/19

Switch buffer size is key!

Problem comes from: The switch buffer sizeEthernet switches, are suppose to drops packets

QoS didn't help, all data was multicast

From burst sizeDirectly calculate required switch buffer size

Fixed size packets of 1358 bytes on wire

Know you switch buffer sizes

don't place small-buffer switches in backbone

Click to edit the title text format

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level

/29

IPTV multicast from a network point of view

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level