Upload
truongcong
View
249
Download
0
Embed Size (px)
Citation preview
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
RemovingThe
Linux Routing Cache
David S. Miller
Red Hat Inc.
Columbia University, New York, 2012
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
1 Linux Maintainership
2 Networking: An Introduction
3 Linux Routing Cache
4 Removing the Routing Cache
5 The End
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
My Background
Started working on the kernel 18+ years ago.First project: helping with ELF executable supportSecond project: Porting Linux to SPARC cpusThird project: Taking over Networking maintainenceCan be overwhelming at times.
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
What exactly do you do?
Like being an editor for a major publicationGet to write my own articles from time to timeMajority of time is spent reviewing work of others...... and putting out the occaisional fire.Merging, lots and lots of merging.Strong benefits from tools such as GIT and patchwork.
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
GIT
Distributed version control systemOriginally written by Linus TorvaldsFull disconnected operation (unlike CVS/SVN)Buildup of trust networks, who I pull fromSide topic: bisectionContributors don’t have to use GIT ...... but for heavy contributors it helps a lot.GIT is used for many projects other than the Linux kernelGIT can be used on top of other source code controlsystems
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Patchwork
Web based patch tracking systemPatchwork is subscribed to various developer mailing listsIt recognizes software patch submissions and GIT pullrequestsMaintainers can manage as objects in various states“Under Review”, “Changes Requested”, “Applied”Allows maintainer as well as submitted to see a changesstate
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Computer Networking
A collection of computers“interconnected” by communication channelsAllows the sharing of resources or informationA group of devices connected to each other
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Protocols
Prearranged method for successful communicationPrecisely defined message formatsSignalingAuthenticationError detection and correctionSyntax, Semantics, Synchronization
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Networking from an OS Perspective
Drivers, Protocol Implementation, System CallsBSD “Sockets”
De-facto interface for network programmingCreated in 4.2 BSD Unixsocket()bind(), connect(), listen(), accept()send(), recv() (read(), write())close()
Drivers send and receive packets “on the wire”Most low level protocols implemented in the kernel
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Internet Protocol
Successor to the ARPANETA DARPA project funded by the US Defense DepartmentUses an addressing system of 32-bit “IP Addresses”Transport Protocols include TCP, UDP, and SCTP
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
IPV4 Routing
Routers implement “store and forward”Move packets to their final destinationDecisions are made using a “routing table”Route selection uses “longest matching prefix”Static Tables vs. Routing Protocols
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Routing Example
Two routes:172.168.82.0/24 “A”172.168.0.0/16 “B”
Lookup for 172.168.82.6 would match “A”Lookup for 172.168.2.2 would match “B”The “default” route 0.0.0.0/0
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Hierarchy of Tables
Cache is implemented as a hash tableCache is populated by lookup requestsInformation for cache entries comes from “ForwardingInformation Base” or FIBFIB is populated by administrator or routing daemonCache is therefore dynamic, whereas FIB is relatively staticCache lookup is simple and extremely low latencyFIB lookups are more complicated and thus moreexpensiveFIB tables implemented using trie data structure (LC-Trie)
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Features Largely Unique to Linux
Routing by source address (“routing realms” or “policyrouting”)Routing by Type-of-Service (“TOS based routing”)Routing by firewall marking (“MARK based routing”).Front end is called “routing rules”Initial lookup finds a matching “rule”A “rule” leads to a “routing table”“routing table” holds routes by destination address
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Source Address Validation
We receive packet from address “A” on interface “I”If I were to route a packet to “A” ...... would that route use interface “I”?Nearly useless on end-hosts.Very useful at points of egress, routers, and firewalls.Can add up to two route table lookups per route lookup.
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Routing Cache Keying
Hash table keyed on all possible route attributes:Destination Address, Source AddressType-of-ServiceFirewalling MARKSecurity ContextOutgoing or Incoming Interface
Entries therefore have a high level of granularity
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Population of Routing Cache
Created on demand by lookupsEvery packet sent causes a lookupEvery packet received causes a lookupAnyone can send us packetsAnyone can create a lookupTherefore, anyone can influence contents of our cache
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Hash Function is Important
Must distribute well.Must be non-deterministic for remote entitiesImplies some unknown component fed into hash.Routing cache uses Jenkins Hash ...... with random number salting the input.By default random input recomputed every 10 minutes
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Cache is Exploitable
This is true regardless of hash quality.Attacker can simply cycle through all values of all keysEach new packet sent modifies the lookup key in some wayEach new packet creates a new routing cache entryTriggers garbage collection when size limit is reachedCache is no longer a cache, since every lookup missesThis is more expensive than having no cache at all.
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Cache in Non-hostile Environment
Even with “well behaved” traffic, cache is undesirableGoogle sees hit rates on the order of only 10 percentOn simpler systems, cache is effectiveBut still exposed to key cycling denial of serviceThe cache has to be removed
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Not Easy
Multi-year projectIssues:
Performance retentionNo externally visible semantic changesDependencies on CacheFinding the right mix of interfaces
A lot of feeling around in the dark
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Performance Testing
Micro vs. Macro analysisComplete isolation of route lookupTop-level operation measurementsLooking at route lookups “in context”Goal: less than 10 percent cost increase for real operations
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Macro Analysis: udpflood
Built upon a design by Eric DumazetBasically: Loop sending N small UDP packetsNo external network connectivity necessaryUses special “dummy” network deviceAs well as static ARP table entriesAll packets simply dropped when they hit deviceAllows pure software analysis of packet send cost
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Micro Analysis: First Try
“route_bench”Uses routing sockets, specifically GETROUTE requests.Basically: Loop looking up N routesDoesn’t work well at all.Measures the wrong thing.Routing socket message overhead dominates test.So test isn’t really measuring route lookups.
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Micro Analysis: Second Try
Specialized kernel moduleCalls route lookup functionTakes CPU cycle counter before and afterModule parameters configure the lookup parametersLoading the module runs the benchmarkEven deeper analysis: “perf”
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Micro Analysis: Perf
Performance analysis tool (since Linux 2.6.31)Uses CPU performance monitor facilitiesCan monitor both user and kernel modeCan watch software events (page faults, context switches)Sampling records program counter at time of eventAllows easy detection of “hot spots” during a test
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Micro Analysis: Final Setup
Run module load under “perf record”Analyze run using “perf report”Provides detailed cost analysis of all route lookup codepathsAllowed quick fixing of lowest hanging fruitProfile counts with and without routing cache could becomparedAll base perf runs using CPU cycle counter as “event”Deeper analysis with “cache miss” and “data access”events
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Route Identities
Routing Cache entry was “fully specified”Contained saddr, daddr, TOS, etc.Lots of dependencies existRemove such dependenciesReplace with other methods of reconstituting values
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Routing Metrics
Path parametersTCP round trip timesPath MTUSpecific to exact path usedTherefore must be maintained with strict keyingMetrics cannot be shared amongst several destinations
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
TCP Metrics Cache
Entries created by successful TCP connectionestablishmentNot susceptible to exploitability like routing cacheTCP connection creation requires work by senderAttacker has to use his own resources and reveal hisidentityEntries keyed only on destination address
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Route Exceptions
Created for Path-MTU and RedirectsResult from responses to packets we sendCannot be triggered arbitrarily by attackersThese two cases are rare
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Neighbours
Hold link-level nexthop information (for ARP, etc.)Routing cache pre-computed neighboursRemember: One “route” can refer to several nexthopsNeed to disconnect neighbours from route entries.Solution:
Make neighbour lookups cheaper (faster hash, etc.)Compute neighbours at packet send time ..... instead of using precomputed reference via route
Most of work involved removing dependenies on old setup
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Mitigating Factors: Part I
“Generic Receive Offload” or simply GROAccumulates RX packets for the same TCP connectionRules:
Same connection IDNo special flagsConsequetive sequence numbers
Passes large accumulated packet as one object into stackDecreases “transactional overhead”Process 1 packet instead of N packetsMeans “perform 1 route lookup instead of N route lookups”
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Mitigating Factors: Part II
“TCP Pre-Demux”Lookup TCP socket on receive, before route lookupCache input route in TCP socketUse cached input route to avoid route lookupTCP hash table not susceptible to routing cache problems
“attacker” must allocate his own resourcesMust complete 3-way handshakeMust maintain lots of state
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
End Result
Routing cache removed in kernel 3.610 percent goal achieved for udpfloodInput route lookup for local TCP sockets actually fasterThanks to TCP pre-demuxDoS attacks are completely eliminatedMajor improvements are still possibleFor example: Combine local and main tablesWould half a major component of lookups
David S. Miller Removing The Linux Routing Cache
Linux MaintainershipNetworking: An Introduction
Linux Routing CacheRemoving the Routing Cache
The End
Thanks
Jason NiehEric DumazetJulian AnastasovSteffen KlassertHerbert XuTom Herbert
David S. Miller Removing The Linux Routing Cache