Embed Size (px)
Citation preview
•Surachai CHITPINITYON•Kasom KOHT-ARSA•Surasak SANGUANPONG•Anan PHONPHOEM•Pirawat WATANAPONGSE•Chalermpol CHUPAMPUN
•Office of Computer Services•Kasetsart University
•E-mail: [email protected]
Design and Implementation of Design and Implementation of Large Scale URL Filtering Large Scale URL Filtering
•APAN, Xi’an, Network Security, 29th August 2007
This work is partially supported by Commission of Higher Education (CHE), UniNET, Thailand
2Network Operation Center Kasetsart University Office of Computer Services
Agenda
Why Need URL Filtering? Filtering Techniques TCP Revisited Proposed Solution Performance Facts Current Deployment Scalability Planning for 10Gbps
3Network Operation Center Kasetsart University Office of Computer Services
Agenda
Why Need URL Filtering? Filtering TechniquesFiltering Techniques TCP RevisitedTCP Revisited Proposed SolutionProposed Solution Performance FactsPerformance Facts Current DeploymentCurrent Deployment Scalability Planning for 10GbpsScalability Planning for 10Gbps
4Network Operation Center Kasetsart University Office of Computer Services
Why Need URL Filtering?
Access Policy Enforcement Parental Control Other restricted website by Policy
Suspected Harmful Website (on-demand filtering) Spyware, Phishing Embedded Scripting Websites intend to attack
OS/Software Vulnerabilities
5Network Operation Center Kasetsart University Office of Computer Services
Agenda
Why Need URL Filtering?Why Need URL Filtering? Filtering Techniques TCP RevisitedTCP Revisited Proposed Solution Proposed Solution Performance FactsPerformance Facts Current DeploymentCurrent Deployment
6Network Operation Center Kasetsart University Office of Computer Services
Gateway
Filtering Engine
Client
Internet
Pass-Through Web Filtering
Traffics must pass through the filtering engine (Firewall, Proxy, Application Gateway)
Create a queue of processing with delay Delay is depend on traffic volume and machine performance
1
2
3
??
Allow
Block
Unknown
4
7Network Operation Center Kasetsart University Office of Computer Services
Pass-by Web Filtering
Traffics are captured and passed by without queuing Zero delay, independent from traffic volume
Ease of Installation (No Traffic Interruption)
Non Blocking Traffic Stream
No Single Point of Failure Scalable
Gateway
Filtering Engine
Client
Internet
3
??
1 2
2
8Network Operation Center Kasetsart University Office of Computer Services
Agenda
Why Need URL Filtering?Why Need URL Filtering? Filtering TechniquesFiltering Techniques TCP Revisited Proposed SolutionProposed Solution Performance FactsPerformance Facts Current DeploymentCurrent Deployment Scalability Planning for 10GbpsScalability Planning for 10Gbps
9Network Operation Center Kasetsart University Office of Computer Services
TCP Connection Establishment & Data Transfer
SYN J
SYN K , ACK J+1
ACK K+1
SYN_SENTSYN_SENT
ESTABLISEDESTABLISED
SYN_RCVDSYN_RCVDESTABLISEDESTABLISED
Data (request)
Data
(reply)
ClientClient ServerServer
10Network Operation Center Kasetsart University Office of Computer Services
TCP Connection Termination
FIN L
ACK L+1
CLOSE_WAITCLOSE_WAIT
FIN_WAIT_1FIN_WAIT_1
FIN_WAIT_2FIN_WAIT_2
ClientClient ServerServer
LAST_ACKLAST_ACK FIN M
ACK
M+1
TIME_WAITTIME_WAIT
CLOSEDCLOSED
11Network Operation Center Kasetsart University Office of Computer Services
FilteringFiltering
TCP Session Hijacking
SYN J
SYN K , ACK J+1
ACK K+1
FIN L
ClientClient ServerServer
Data (request)
Data
(reply)Packet will be ignoredPacket will be ignored
Faked FIN by Filtering EngineFaked FIN by Filtering Engine
12Network Operation Center Kasetsart University Office of Computer Services
Agenda
Why Need URL Filtering?Why Need URL Filtering? Filtering TechniquesFiltering Techniques TCP RevisitedTCP Revisited Proposed Solution Performance FactsPerformance Facts Current Deployment Current Deployment Scalability Planning for 10GbpsScalability Planning for 10Gbps
13Network Operation Center Kasetsart University Office of Computer Services
Proposed Solution
Pass by method incorporated with 2 techniques Session HijackingSession Hijacking
Fast Sequence Number InterceptionFast Sequence Number Interception Keywords Capturing in Application Request Packet Keywords Capturing in Application Request Packet
URL Processing Designed toURL Processing Designed to Handle Hundred Million of URLs listHandle Hundred Million of URLs list Very fast access to URLs repositoryVery fast access to URLs repository
14Network Operation Center Kasetsart University Office of Computer Services
Session Hijacking
FIN L
ClientClient ServerServerFilteringFiltering
Data (request)
Data
(reply)
Successful filtering
ACK L+1Faked FINFaked FIN
FIN Mignoredignored
Unsuccessful filtering
ACK M+1
FIN L
Faked FINFaked FIN
15Network Operation Center Kasetsart University Office of Computer Services
GET
3
Keyword Capturing
Gateway
Filtering Engine
Client
Internet
GET/PUT/POST
1
GET
search
??
Matching
5
FIN2
GET
4
FIN
Black Lists
2
GET
16Network Operation Center Kasetsart University Office of Computer Services
URL Management Technique
Key design URL Compression Techniques In-Memory Balanced Tree of URLs Utilize KSpider’s Core Architecture (URL
Manager Module)
Benefits 69% Averaged Compression Ratio of
URLs Length (currently supported Max 268 Millions URLs List under 8 GB RAM)
Almost Linear Access Speed (10 microseconds by averaged
17Network Operation Center Kasetsart University Office of Computer Services
URL BufferQueue
URL BufferQueue
SchedulerScheduler
URL Manager
URL Storage Manager
URL Storage Manager
On Disk On Disk
ParallelDNS
ParallelDNS
In-memoryIn-memory
Storage
KSpider’s Architecture
URL Filter URL Filter
Data StreamerData Streamer
URL Processor
URL ExtractorURL Extractor
URL BufferQueue
URL BufferQueue
SchedulerScheduler
Communicator
ClusterCommunicator
ClusterCommunicator
Data Collector
URL BufferQueue
URL BufferQueue
Storage Manager
Data CompressorData Compressor
Data Decompressor Data Decompressor
HTTP DataCollector
HTTP DataCollector
Stats CollectorStats Collector
Online indexer Other processing
To Communicator
Storage
18Network Operation Center Kasetsart University Office of Computer Services
URL Compression Technique
Prefix Balance Search Tree
http://www.lovely.com
http://www.lion.com
http://www.lovely12.com
http://www.lovely11.net
http://www.lower13.net
Webscreen List 0 http://www.lovely.com/
1 12 ion.com 2 17 12.com
3 18 1.net 4 18 3.net
19Network Operation Center Kasetsart University Office of Computer Services
Agenda
Why Need URL Filtering?Why Need URL Filtering? Filtering TechniquesFiltering Techniques TCP RevisitedTCP Revisited Core TechnologyCore Technology Performance Facts Current DeploymentCurrent Deployment Scalability Planning for 10GbpsScalability Planning for 10Gbps
20Network Operation Center Kasetsart University Office of Computer Services
Performance
0
5
10
15
20
25
30
35
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77
Compressed URL length
perc
ent f
ound
Hijack Activationunder 0.6 msec
Test Record268 Million URLs
with 8 GB
Avg. Search Time10 µsec
(350 µsec MAX with 268 Million URLs)
Memory Requirement
34M URL/GB
0
2
4
6
8
10
12
14
16
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43
Time used (millisecond)
Perc
ent f
ound
Performance collected under Dell 2900, Intel Xeon 5160(3Ghz)
69% compression ratio with average 26.5 bytes per URL
21Network Operation Center Kasetsart University Office of Computer Services
Agenda
Why Need URL Filtering?Why Need URL Filtering? Filtering TechniquesFiltering Techniques TCP RevisitedTCP Revisited Core TechnologyCore Technology Performance FactsPerformance Facts Current Deployment Scalability Planning for 10GbpsScalability Planning for 10Gbps
22Network Operation Center Kasetsart University Office of Computer Services
Reference Site
3 Gbps 2 Gbps
EtherChannel 2 Gbps
Ethernet 1 Gbps
CPU : 2xDual Core Opteron 2.4 GhzRAM : 8 GBHD : SAS 146 GB
WebScreen Agent
Multiple Links/Interfaces
Operations since December 2005
Inter. GW Inter. GW
CAT Telecom
8 gigabit links span to 8 gigabit interfaces
in 4 machine
23Network Operation Center Kasetsart University Office of Computer Services
Collected Statistics
Avg. 110 request/s Dropping rate (9.5 M per day)
Peak 250 request/s Dropping rate
4.6 Gbps aggregated traffic 1.6 M packet/s incoming packets 64 K packet/s http request packets
24Network Operation Center Kasetsart University Office of Computer Services
Agenda
Why Need URL Filtering?Why Need URL Filtering? Filtering TechniquesFiltering Techniques TCP RevisitedTCP Revisited Core TechnologyCore Technology Performance FactsPerformance Facts Current DeploymentCurrent Deployment Scalability Planning for 10Gbps
25Network Operation Center Kasetsart University Office of Computer Services
Scalability Planning for 10Gbps
Solutions for 10 Gbps Link Deploy Traffic Distribution Device
(1x10 Gbps to 10x1 Gbps) Currently on the test of GigaVUE
GigaVUE1
LANLAN
Mirror port Mirror port
THAISARNUNINET
GigaVUE2
Typical servers can handle up to 800 Mbps
bit rate per 1 Gbps interface
1G
1G10G
10G
10G 10G
10G
10G 10G1G
1G 1G
1G
10G 10G
26Network Operation Center Kasetsart University Office of Computer Services
Q&A
27Network Operation Center Kasetsart University Office of Computer Services
Thank You
