Network Anomaly Detection Based on Behavioral Traffic ... · Zurich Research Laboratory IBM | Jul 06 | Network Anomaly Detection Andreas Kind Paul Hurley Jeroen Massar Xenofontas

Zurich Research Laboratory

IBM | Jul 06 | Network Anomaly Detection www.zurich.ibm.com/aurora

Andreas KindPaul HurleyJeroen MassarXenofontas Dimitropoulos

Network Anomaly Detection Based on Behavioral Traffic Pattern Recognition

2


A Kind | Network Anomaly Detection | www.zurich.ibm.com/aurora © 2006 IBM Corporation

Network AnomaliesUnusual and significant changes in network traffic characteristics

– Data volume (octets, packets)

– Flows (number, duration, size, service type)

– Communication matrix (src/dst IP, src/dst ports)

– Packets (size, flags)

Caused by…– “Season”

– Organizational change (eg, new application, new user group, new business process)

– Flash crowd

– Vulnerability scan

– Outage, fault, misconfiguration (eg, port scanning AFS, DNS used by IDS)

– DoS attack, self-propagating attack (virus, worm)

– Research on networks

3



Network Anomalies

IPIPxIPPorts

FlowsOctets

Flow duration

PacketsOctets/packets

4



Detection Requirements

Scalable for data centersNo additional equipment (eg, splitters, taps, meter appliances)No traffic insertion (eg, active probing)No agents, no credentialsNo access to traffic payloadNo increase in monitoring trafficReal-time operationLow hardware costsNo explicit configuration of thresholds and confidence intervalsApplicable to highly varying workloads

… which is a bit of a contradiction

No automatic prevention, no prediction, but deployment in combination with flow-based network profiling system

– Which are the end-to-end flows causing the anomaly?

5



Related Work

Signature-based approaches– Too slow, payload needed, only know worms/viruses are addressed

Statistical approaches– Typically based on abrupt changes and therefore error-prone with varying

workloads in distributed environments

Rule-based approaches– Difficult to train, complex rule-sets too slow

Service spoofing– Traffic destined to unused addresses is a priori suspicious

– Most effective for worms

Pattern-based approaches– Capture traffic patterns from network characteristics and compare with

baseline pattern

– How to compose and compare traffic patterns in order to address detection requirements?

6



.

.

.

.

.

.Time

Behavioral Traffic Patterns(ie, histograms of measured network parameters during time period dx)

OK

OK

not OK (ie, traffic anomaly)

OK

Measured network parameters (via NetFlow/IPFIX only)

dx

Desired Detection System

7



Network Traffic PatternsDefined as histograms that display the frequency of flow parameter ranges during observation periodExamples: IP address range, TCP/UDP port range, flow duration

8



Network Traffic Patterns

9



Behavioral Analysis of Virus ActivityS

rc A

ddr

Mon Tue

Dst

Add

r

Wed Thu Fri

Host scan prefix mask 255.0.0.0prefix mask 0.255.0.0

10



Distance Between Traffic PatternsDefined as the number of changes in the relative order between two patterns

0 if ( w1[i]≥w1[mod(i,n)+1] ∧w2[i]≥w2[mod(i,n)+1] ) ∨

ord(w1[],w2[],i) = ( w1[i]≤w1[mod(i,n)+1] ∧w2[i]≤w2[mod(i,n)+1] )

1 otherwise

ExampleGiven w1 = (1,2,3,4), w2 = (0,7,2,1)

ord(w1,w2,1) = 0

ord(w1,w2,2) = 1

ord(w1,w2,3) = 1

ord(w1,w2,4) = 0

11



Distance Between Traffic PatternsDistance function

d(w1, w2) = 1/n∑ ord(w1, w2, i) for 0 < i ≤ n

ExampleGiven w1 = (1,2,3,4), w2 = (0,7,2,1)

d(w1, w2) = 1/4 * 2 = 0.5

12



Archetypes

13



Clustering Traffic Patterns

Tree Clustering– Joining patterns into successively larger clusters using distance function

– Results in hierarchical tree

– But: How to determine mean (most likely “dummy”) pattern for which variability in distances to other cluster members is the smallest?

k-Means Clustering– Given fixed member of k clusters

– Assign patterns to clusters so that overall variability in distances to other cluster members is minimized

14



Traffic Pattern Archetypes

Traffic pattern archetypes are computed with k-means clustering

Find w1, …, wi ∈ W so that ∑ MIN d(wi , wk) with wk ∈ W \ {w1, …, wi} is minimized

– Find the i patterns for which the sum of the minimum distances to all other patterns is minimized

– We used i up to 4

i

15



Validation

Abnormal patterns

e

Maximum distance of baseline pattern to archetypes

Patterns ordered by distance to archetypes

Dis

tanc

e

Threshold at 0.25

16



Distance is above

threshold?

. . .Training set of patterns

Select Archetypes

. . .

Archetypes

System in normal

operation

Determine distance of

current pattern from

archetypes

Mark pattern

OK

Mark pattern not OK

no

yes

Step 1 Step 2

Step 3

Determine threshold

System in current

operation

Get current pattern

Initialization

Start

Threshold

17



Future Work

http://www.zurich.ibm.com/aurora

Continue the theoretic and empirical work on this approachExperiment with different distance functions and clustering algorithmsProve the time advantage of behavioral network problem prediction

Close integration with IBM’s flow-based network profiling systemUse approach with server workloadsVisualization with force-directed graphs (ie, attractive/repulsive forces)

– ip08, duration, …

THANKS!

Time

Max

Failure & threshold events

Impact

∆t1

Behavioral abnormality

∆t2

Avalanche analogy

Documents

Network Anomaly Detection Based on Behavioral Traffic ... · Zurich Research Laboratory IBM | Jul 06 | Network Anomaly Detection Andreas Kind Paul Hurley Jeroen Massar Xenofontas