Embed Size (px)
Citation preview
Computer Networks 53 (2009) 774–789
Contents lists available at ScienceDirect
Computer Networks
journal homepage: www.elsevier .com/locate /comnet
Characterizing network traffic by means of the NETMINE framework
Daniele Apiletti *, Elena Baralis, Tania Cerquitelli, Vincenzo D’EliaPolitecnico di Torino, Dipartimento di Authomatica Informatica, Corso Duca degli Abruzzi, 24, 10129 Torino, Italy
a r t i c l e i n f o
Article history:Available online 29 December 2008
Keywords:Network traffic characterizationNetwork data analysisGeneralized association rules
1389-1286/$ - see front matter � 2008 Elsevier B.Vdoi:10.1016/j.comnet.2008.12.011
* Corresponding author. Tel.: +39 0110907084.E-mail addresses: [email protected] (D. Ap
polito.it (E. Baralis), [email protected] (T. [email protected] (V. D’Elia).
a b s t r a c t
The NETMINE framework allows the characterization of traffic data by means of data miningtechniques. NETMINE performs generalized association rule extraction to profile communi-cations, detect anomalies, and identify recurrent patterns. Association rule extraction is awidely used exploratory technique to discover hidden correlations among data. However,it is usually driven by frequency constraints on the extracted correlations. Hence, it entails(i) generating a huge number of rules which are difficult to analyze, or (ii) pruning rareitemsets even if their hidden knowledge might be relevant. To overcome these issues NET-
MINE exploits a novel algorithm to efficiently extract generalized association rules, whichprovide a high level abstraction of the network traffic and allows the discovery of unex-pected and more interesting traffic rules. The proposed technique exploits (user provided)taxonomies to drive the pruning phase of the extraction process. Extracted correlations areautomatically aggregated in more general association rules according to a frequencythreshold. Eventually, extracted rules are classified into groups according to their semanticmeaning, thus allowing a domain expert to focus on the most relevant patterns. Experi-ments performed on different network dumps showed the efficiency and effectiveness ofthe NETMINE framework to characterize traffic data.
� 2008 Elsevier B.V. All rights reserved.
1. Introduction
Due to the continuous growth in network speed, tera-bytes of data may be transferred through a network everyday. Thus, two major issues hamper network data captureand analysis: (i) a huge amount of data can be collected ina very short time (e.g., an Ethernet frame at 10 Gbps is re-ceived in less than 70 ns). (ii) It is hard to identify correla-tions and detect anomalies in real-time on such largenetwork traffic traces. New efficient techniques able todeal with huge network traffic data need to be devised. Asignificant effort has been devoted to the application ofdata mining techniques to network traffic analysis [9].The application domains include studying correlationsamong data (e.g., association rule extraction for network
. All rights reserved.
iletti), elena.baralis@erquitelli), vincenzo.
traffic characterization [5,17] or for router misconfigura-tion detection [22]), extracting information for prediction(e.g., multilevel traffic classification [19], Naive Bayes clas-sification [27]), grouping network data with similar prop-erties (e.g., clustering algorithms for intrusion detection[32], or for classification [12,13,36]), and context specificapplications (e.g., multi-level association rules in spatialdatabases [23]). Data mining techniques play an importantrole in intrusion detection systems [34,30], where associa-tion rules are successfully exploited for anomaly identifica-tion [31].
While supervised classification algorithms require pre-vious knowledge of the application domain (e.g., a labeledtraffic trace), association rule extraction does not. Hence,the latter is a widely used exploratory technique whichmay be exploited to highlight hidden knowledge in net-work flows. The extraction process is driven by enforcinga minimum frequency (i.e., support) constraint on themined correlations. However, to discover (potentially rele-vant) knowledge, a very low support constraint has to be
D. Apiletti et al. / Computer Networks 53 (2009) 774–789 775
enforced, hence generating a huge number of unmanage-able rules [5]. To address this issue, a higher level abstrac-tion than traditional association rules and a networksummarized representation for traffic data are needed.
This work presents a framework, called NETMINE, whichperforms network traffic analysis by means of data miningtechniques to characterize traffic data and detect anoma-lies. NETMINE performs (i) on-line stream analysis to aggre-gate and filter network traffic, (ii) refinement analysis todiscover relationships among captured data, and (iii) ruleclassification into different semantic groups. NETMINE allowson-line stream analysis concurrently with data capture bymeans of user-defined continuous queries. ‘‘Continuousqueries” [4] is an efficient technique to perform real-timeaggregation and filtering; thus it allows an effective reduc-tion of the amount of network data to be analyzed. The re-sult of this step is a meaningful summary of network trafficdata appropriate for pattern discovery. NETMINE performs arefinement analysis to discover traffic features from net-work data summaries by means of generalized associationrule extraction. Differently from previous approaches [15],NETMINE generalized rule extraction is support driven, i.e.,rules are generalized only when lower levels in the taxon-omy are below a frequency threshold. Thus, in NETMINE
generalization is automatically performed only whenneeded. Finally, extracted rules may be classified in severalsemantic categories, depending on the traffic features theyobserve. NETMINE ’s final output is a set of categorized andgeneralized rules [18] which are able to characterize net-work traffic and to show correlation and recurrence of pat-terns among data. Experiments performed on differentnetwork dumps showed the efficiency and effectivenessof the NETMINE framework in characterizing traffic dataand highlighting meaningful features.
The paper is organized as follows. Section 2 presents anoverview of the NETMINE framework and the main featuresof its building blocks. In Section 3 experiments to validatethe proposed framework are reported. Section 4 discussesrelated works, while Section 5 draws conclusions.
1.1. Motivating example
A traffic capture process produces a trace which holdsinformation on the stream of packets. A trace is composedof a set of records, each of which is a set of tagged items(also denoted as itemset in the following). While itemsare values captured from the net (e.g., the IP address130.192.3.17), their tags are the description of the repre-sented information (e.g., the label IP destination address).
Relevant patterns may be represented by means ofassociation rules. An association rule is represented inthe form X ) Y , where X and Y are disjoint conjunctionsof tagged items. Each rule is usually characterized by thesupport and confidence quality index [18]. The support isthe prior probability of X and Y (i.e., its observed frequencyin the data set). The confidence is the conditional probabil-ity of Y given X and characterizes the ‘‘strength” of a rule.The traditional association rule mining problem can be de-scribed as follows. Given a database of transactions (in ourcase a network trace), a minimum confidence thresholdand a minimum support threshold, find all association
rules whose confidence and support are above giventhresholds. However, the following example highlightsthe need of a more powerful abstraction of associationrules.
Consider a web server on port 80 having address130.192.5.7. To describe the activity of a client connectingto this server, a rule in the form
may be extracted, where ðs%; c%Þ are support and confi-dence values. Since a single source address is a 1-itemsetwhich is infrequent in a very large traffic network trace,extracting such rule would require enforcing a very lowsupport threshold, which makes the task infeasible. How-ever, a higher level view of the network may be providedby the following generalized association rule:
which shows a subnet generating most of the traffic. Thisgeneralized rule may provide valuable knowledge for net-work monitoring. The number of different tagged items innetwork traffic may be very large (e.g., different IP ad-dresses) and association rules on single tagged items pro-vide excessively detailed and hardly usable knowledge.Generalized association rules allow raising the abstractionlevel at which correlations are represented. Hence, they area powerful tool to address this issue.
2. The NETMINE framework
NETMINE is a framework to efficiently perform networktraffic analysis. NETMINE addresses two main issues: (i) datastream processing to dynamically reduce the amount oftraffic data and allow a more effective use, both in timeand space, of data analysis techniques, (ii) correlationextraction from traffic data to characterize network traffic,detect anomalies, and identify recurrent patterns.
Fig. 1 shows NETMINE main blocks: data stream process-ing, refinement analysis, and rule classification. Trafficpackets, captured by means of available network capturetools [35,28], are the input data of the stream processingblock, whose objectives are (i) summarizing the trafficwhile preserving structural similarities among temporallycontiguous packets and (ii) discarding irrelevant traffic toreduce traffic volume. Data stream processing is performedconcurrently with data capture by means of continuousqueries, which perform aggregation (i.e., similar recordscan be summarized by a proper summary) and filtering(i.e., data irrelevant for the current analysis is discarded)
Fig. 1. NETMINE architecture.
776 D. Apiletti et al. / Computer Networks 53 (2009) 774–789
of network traffic. The output flows (i.e., filtered and aggre-gated packet summaries) may be saved in a permanentdata store. Storage is required only when different refine-ment analysis sessions need to be performed.
The aim of the refinement analysis is to discover inter-esting correlations, recurrent patterns and anomalies intraffic data. Currently, interesting patterns are extractedin the form of generalized association rules, i.e., rules whichrepresent general correlations among network traffic data.But the framework allows different data mining techniques[18] to be easily integrated. The refinement analysis is atwo step process: (i) an optional data stream view blockselects a suitable user-defined subset of flows to focusthe following analysis on. (ii) Generalized association rulesmining is performed either on the data stream view, whichcontains the selected flows, or on all the flows in the per-manent data store. Generalized association rule mining isimplemented by means of the novel Genio algorithm [6].The Genio algorithm is more effective and efficient thanprevious approaches [15] in automatically extracting inter-esting generalized rules from structured data.
The rule classification phase organizes the extractedrules according to their semantic interpretation in the net-work domain. Hence, it allows a network operator to focuson relevant features of the captured data without priorknowledge of the desired network patterns. All the build-ing blocks of the NETMINE framework are described in moredetail in the following subsections.
2.1. Data stream processing
The data stream processing block of NETMINE reducesthe volume of traffic data by grouping similar packetsand discarding irrelevant ones. Network traffic can be con-sidered as a stream of structured data, where each cap-tured packet is a record whose attributes (i.e., tags) aredefined by network protocols. Each record is characterizedby at most one value for each tag. More formal definitionsfollows.
Definition 1. Tagged item. Let T ¼ ft1; t2; . . . ; tng be a setof tags which describe network protocol attributes. Atagged item ti ¼ itemi assigns the value itemi to thenetwork protocol attribute ti, where itemi belongs to thedomain of attribute ti.
For example, in this context, relevant tags are sourceand destination addresses, source and destination ports,the level 3 and level 4 protocols (e.g., TCP, UDP) and thesize of the packet.
Definition 2. Itemset. Let I ¼ ft1 ¼ item1; t2 ¼ item2; . . . ;
tn ¼ itemng be the set enumerating all tagged items. Anitemset X #I is a set of tagged items.
Definition 3. Network trace. Let T ¼ ft1; t2; . . . ; tng be aset of tags which describe network protocol attributesand I ¼ ft1 ¼ item1; t2 ¼ item2; . . . ; tn ¼ itemng the corre-sponding set enumerating all tagged items. A networktrace is a collection of records, where each record r is anitemset X #I. Each tag in T may occur at most once inany itemset X.
Since packets are captured as an unbounded stream,conventional query processing would never terminate. Toovercome this issue, continuous queries [4], i.e., a set ofinstructions to process a subset of the stream, areexploited. Continuous query languages have been thor-oughly addressed in [2], where CQL (Continuous QueryLanguage [2]) is proposed to express continuous queries.Several approaches have been proposed to efficiently eval-uate continuous queries over streaming data, among whichNiagaraCQ [10] for internet databases, STREAM [4] for dif-ferent streaming applications (e.g., network monitoring,telecommunications data management, publish-subscribe,web personalization), and Telegraph CACQ [24] for sensordatabases. Data stream processing in NETMINE is based onthe continuous queries presented in [4].
Continuous queries are issued once and then logicallyrun continuously over a sliding window of the originalstream. Hence, the following parameters need to be defined:(i) aggregation and filtering rules, expressed in a subset ofthe SQL language instruction set, (ii) a sliding window,whose length is expressed in seconds, which identifies thecurrent set of data on which rules are applied, (iii) the stepstep 6 length, which defines how often the window movesand the output is produced. In NETMINE a record producedas output by the continuous query is a flow, which summa-rizes a group of similar and temporally contiguous packets.
In NETMINE, continuous queries have been decoupled inaggregation queries and filtering queries. Aggregation isperformed on the incoming packet stream by the streamprocessing block. The same block also performs packet fil-tering, hence discarding uninteresting packets from theaggregation. Flow filtering is performed in the data streamview block and allows discarding undesired flows for thespecific analysis purpose. A sample continuous queryimplemented in NETMINE is the following:
Sample query. This query targets the extraction of thebiggest IP traffic flows. Once packets have been aggregated
D. Apiletti et al. / Computer Networks 53 (2009) 774–789 777
by source and destination addresses, and source and desti-nation ports, flows whose size is lower than a given thresh-old are discarded. The threshold is expressed as apercentage of the total traffic of the current window. Bothfiltering and aggregation considerably reduce the datasetsize. The query is expressed by means of CQL, whose mostrelevant clauses are introduced by examples referencingthe language constructs in the query.
Aggregate
Select source-address, destination-address, source-port, destination-port
Sum(size) as flow-size Count(*) as packetsFrom
Packets [Range by 60 s] Where level3 = ’IP’ GroupBy
source-address, destination-address, source-port, destination-portFilter
Select source-address, destination-address, source-port, destination-port, flow-size
From Aggregate Where flow-size P ratio *(Select Sum(flow-size) From Aggregate)
Select clause. It defines the names of the tags to be ex-tracted from the incoming stream (e.g., source-address,destination-address, source-port, destination-port in thesample query) and the list of aggregate operations to beperformed (e.g., Sum(size) returns the total size of selectedflows, while Count(*) the number of packets in the samplequery).
From clause. It introduces the name of the stream to bequeried (e.g., the Packets network trace in the samplequery). The stream name may be followed by the time per-iod which defines how often the window moves and theoutput is produced, introduced by the Range by keywords(e.g., the time period is 60 s in the sample query).
Where clause. It expresses the conditions which shouldbe satisfied by the incoming stream records to be selected.The conditions are expressed as a boolean combination ofpredicates of the form expression op expression, where opis one of the comparison operators f<;6;P; >;<>;¼g.An expression is a tag name, a constant, or an (arithmeticor string) expression (e.g., level3 = ‘IP’ in the sample query).
Group by clause. It defines the list of tag names with re-spect to which the aggregation operation needs to be per-formed (e.g., source-address, destination-address, source-port, and destination-port in the sample query). Recordsare grouped together when, for all the attributes in the list,they share the same value (e.g., a group might be formedby records sharing the values source-address = 130.192.1.1,destination-address = 130.192.254.2, source-port = 32,100,destination-port = 80). The aggregate functions specified inthe select clause are computed separately for each group. h
2.2. Refinement analysis
Given a network trace stored in the data store, NETMINE
extracts association rules, which provide an abstract repre-
sentation of interesting correlations and recurrent patternsin data.
Definition 4. Association rule. Let I ¼ ft1 ¼ item1; t2 ¼item2; . . . ; tn ¼ itemng be a set of tagged items in a networktrace. An association rule is represented in the form X ) Y ,where X and Y (the body and the head of the rule,respectively) are disjoint conjunctions of tagged itemsðX #I;Y #I;X \ Y ¼ ;Þ.
A rule template defines the general structure of a set ofassociation rules. All rules with the same template arecharacterized by the same tags (but not necessarily thesame values) in the body and in the head of the rule.
Definition 5. Association rule template. Let T ¼ ft1;
t2; . . . ; tng be a set of tags in a network trace. An associationrule template is represented in the form X ) Y , where Xand Y (the body and the head of the rule template,respectively) are disjoint conjunctions of tagsðX #T;Y #T;X \ Y ¼ ;Þ.
Each rule is characterized by the support (Definition 7)and confidence (Definition 8) quality indices [18].
Definition 6. Itemset support. Let X be an itemset. Thesupport of X is the prior probability of X, i.e., the proportionof records including X to all the records in the networktrace.
Definition 7. Association rule support. Let X ) Y be anassociation rule. Its support sðX ) YÞ is the support ofthe itemset X [ Y .
Definition 8. Confidence. Let X ) Y be an association rule.Its confidence cðX ) YÞ is the conditional probability of Ygiven X, i.e., the proportion of records including both Xand Y to all the records including X.
The traditional association rule mining problem can bedescribed as follows. Given a database of transactions (inour case a network trace), a minimum support thresholdand a minimum confidence threshold, find all associationrules whose confidence and support are above giventhresholds. However, such thresholds in traditional associ-ation rules may prevent relevant knowledge from beinghighlighted, and enforcing very low thresholds may makethe task infeasible. Hence, a more powerful abstraction isneeded for association rules. Generalized association rulesare a powerful tool to address this issue. They extend thenotion of multi-level association rules [15,20], which pro-vide a higher level domain abstraction. Generalized associ-ation rule extraction requires the definition of a taxonomy,which describes a hierarchy among tagged items.
Definition 9. Taxonomy. Let ti be a tag (i.e., a networkprotocol attribute) and Xi its corresponding value domain. Ataxonomy Ti is a pre-determined hierarchy of aggregationsover values in Xi. Ti is a tree whose leaves are values in Xi.Each non-leaf node in the tree is an aggregation of its children,which may be further generalized by its father. Childrennodes are mutually exclusive and collectively exhaustive.
Fig. 2a shows a taxonomy for the source-address, or des-tination-address tags. Source-addresses are aggregated with
778 D. Apiletti et al. / Computer Networks 53 (2009) 774–789
respect to their subnet. Fig. 2b shows a taxonomy for thesource-port or destination-port tags. source-ports may be-long to the well known class if lower than 1024, registeredif in the range 1024-49151, dynamic otherwise.
Definition 10. Generalized association rules. Let E ¼ ft1 ¼expression1; t2 ¼ expression2; . . . ; tn ¼ expressionng be a setof tagged items. expressioni is either (a) a value vi in thedomain Xi of network protocol tag ti, or (b) an aggregationvalue in the taxonomy Ti over ti, where at most onetaxonomy may be defined over each tag ti. A generalizedassociation rule is represented in the form X ) Y , where (i)X and Y are disjoint conjunctions of tagged items inEðX #E;Y #E;X \ Y ¼ ;Þ, and (ii) each tag may be refer-enced at most once in X [ Y .
For example, the association rules
fsource-address ¼ 130:192:1:1g) fdestination-address ¼ 130:192:254:1g
fsource-address ¼ 130:192:1:2g) fdestination-address ¼ 130:192:254:1g
may not be frequent (i.e., they may have a lower supportthan the minimum support threshold), but the generalizedrule, obtained by aggregation of the source-address tag,
fsource-address ¼ Subnet1g) fdestination-address ¼ 130:192:254:1g
might. Hence, generalized patterns derived from rare item-sets may be relevant for network monitoring (e.g., sourcesubnets which contact a print server). Instead of extractingrules for all levels of the taxonomy and post-pruning them[15], NETMINE generalization over the taxonomy is supportdriven, i.e., if lower levels of the taxonomy are over thesupport threshold, they are not aggregated.
2.2.1. Generalized association rules miningGeneralized association rule mining is a two-step pro-
cess: (i) frequent generalized itemset extraction and (ii)
Fig. 2. Taxon
rule generation from frequent itemsets. Since itemset min-ing is considered the most computationally intensiveknowledge extraction task [1], the novel contribution ofthe Genio algorithm focuses on this step, while the secondstep is based on Goethal’s implementation [7] of the Apri-ori algorithm [1].
Given a dataset (either a network trace or a data streamview), a set of taxonomies (at most one for each networkprotocol attribute), and a minimum support threshold s,the Genio algorithm [6] extracts all generalized itemsetswhose support is above support threshold s. Instead ofextracting itemsets for all levels of the taxonomy andpost-pruning them [15], the Genio algorithm performs asupport driven opportunistic aggregation of itemsets. Morespecifically, generalized itemsets are extracted only ifitems at a lower level in the taxonomy are below the sup-port threshold.
Genio is a level-wise algorithm, which, at each iteration,generates only the frequent, possibly generalized, itemsetsof a given length. At arbitrary iteration k, three steps areperformed: (i) candidate generation, in which all possiblek-itemsets are generated from ðk� 1Þ-itemsets, (ii) data-base scan to count the support of candidate itemsets anddiscard infrequent itemsets, and (iii) infrequent itemsetmanagement to extract hidden knowledge ignored by pre-vious approaches. In the third step, a generalized version ofinfrequent itemsets is generated by means of taxonomies.Only generalized itemsets above the support thresholdare kept.
Finally, after all generalized itemsets have been ex-tracted, interesting generalized rules are generated bymeans of Goethal’s implementation of Apriori [7], possiblyenforcing a confidence threshold.
The above generalized rule extraction process is not de-signed to be performed in real-time during the capturingphase. However, experiments on the current implementa-tion of the framework show the feasibility of this approachfor appropriate sliding window update frequencies (seeSection 3.6 for more details).
omy.
D. Apiletti et al. / Computer Networks 53 (2009) 774–789 779
2.2.2. Data stream viewThe data stream view block allows the selection of a
subset of the flows obtained as continuous query output.A data stream view is a query to reduce the amount of databy means of aggregation and filtering operations. Aggrega-tion operations reduce the amount of data by collapsingmany semantically connected records in a single group.Hence, aggregation provides a higher level domain abstrac-tion. Filtering operations select a subset of records orgroups by applying a boolean combination of conditions(expressed by means of comparison operators f<;6;P;
>;<>;¼g) on records or groups.The following example shows the usefulness of data
stream views.
Example. Suppose that, to reduce the amount of storeddata, the network traffic has been aggregated with respectto address and port of both source and destination (e.g.packets differing in one of these features belong todifferent flows). For each flow the sum of the packet sizeis computed. This step is performed by running thefollowing continuous query on the data stream.
Aggregate
Select source-address, source-port, destination-address,
destination-port, Sum(size) as flow-sizeFrom
Packets [Range 60 s] Where level4 = ’TCP’ GroupBy
source-address,source-port,destination-address,destination-portSince the complete dataset contains thousands of flows,the output of the previous continuous query may be fur-ther filtered. The following query may be exploited to cre-ate a data stream view based on a filtering operation.Hence, flows whose size is lower than a threshold x arethe output of the data streaming processing phase.
Filter
Select source-address, source-port, destination-address,destination-port, flow-size
From Aggregate Where flow-size < xThe refinement analysis, performed on the results of thedescribed data stream view, extracts a small number ofgeneralized association rules characterized by high sup-port. These rules highlight more effectively any specifictraffic behavior.
Among the extracted information, rules with the tem-plate {destination-address, destination-port}) {flow-size}, may highlight attempts of SYN flooding attacks,where the left term is the victim fingerprint, while the rightpart is the size of the flow (which is supposed to be small,due to the filtering condition in the data stream view).
Recall that a SYN flooding attack occurs when a victimhost receives an excessive number of incomplete connec-tion requests that it cannot handle. To make this attack
more difficult to detect, the source host randomizes thesource IP address of the packets used in the attack. An at-tempt of SYN flooding [16] may be detected by applyingthe proposed framework in the described conditions.
Other continuous queries may be exploited to definedifferent data stream views, which may lead to the sameresults. For example, the following query exploits theinformation on the number of packets in a flow.
Aggregate
Select source-address, source-port, destination-address,
destination-port, count(�) as flow-packetsFrom
Packets [Range 60 s] Where level4 = ’TCP’ GroupBy
source-address,source-port,destination-address,destination-portThe filtering phase changes accordingly as follows.
Filter
Select source-address, source-port, destination-address,destination-port, flow-packets
From aggregate Where flow-packets > yThe query counts the number of packets in a flow andkeeps flows with a number of packets higher than a thresh-old y (the filtering effect is similar to that of the sizeattribute in the previous query). Also in this case, extractedrules can reveal SYN flood attack attempts. In particular,the rule template {destination-address, destination-port}) {flow-packets} highlights them. Hence, the frameworkis able to support the detection of specific traffic issueseven if the chosen queries are not intentionally designedto address them and are not based on previous knowledgeof the desired results. h
2.3. Rule classification
Association rules highlight correlations among attri-butes and recurrent patterns. However, analyzing the (usu-ally) large number of extracted rules is not a trivial task. Toaddress this issue, we propose a classification of the rulesinto groups according to their semantics in the networkdomain. The semantics of a rule is determined by its tem-plate, i.e., by the referenced tags (e.g., source and destina-tion addresses, port numbers, etc.) and their position (i.e.,in the body or head of the rule).
Although an exhaustive analysis of all the possiblesemantic classes is infeasible, due to the high number ofdifferent attributes and their combinations, some basicgroups have been identified. Their purpose is to guide theanalysis of the extracted rules in the network domain.
The basic classes are built upon the following attributes(i.e, tags) of network data:
� Source address (abbreviated sa).� Destination address ðdaÞ.� Destination port ðdpÞ.
Table 2Length-3 rule classes.
Class Question Rule template Example
TF + PS Given a destination address, by which hostsand for which services is it contacted?
fdag ) fdp; sag fda ¼ PrintServerg ) fdp ¼ SharedPrinter; sa ¼ AdminSubnetg means that,when the Print Server is used, then it offers the Shared Printer service tohosts in the Admin Subnet
TF + PS Given a source address, when it requests aspecific service, which hosts does it contact?
fsa; dpg ) fdag fsa ¼ AdminSubnet;dp ¼ SharedPrinterg ) fda ¼ PrintServerg means that,when hosts in the Admin Subnet request the Shared Printer service, thenthey use the Print Server
TF + SU Given a source address, which services doesit use and to which hosts does it requestthem?
fsag ) fdp; dag fsa ¼ AdminSubnetg ) fdp ¼ SharedPrinter; da ¼ PrintServerg means that,hosts in the Admin Subnet request the Shared Printer service to the PrintServer
TF + SU Given a destination address, when it offers aspecific service, which hosts use it?
fda; dpg ) fsag fda ¼ PrintServer; dp ¼ SharedPrinterg ) fsa ¼ AdminSubnetg means that,when the Print Server offers the Shared Printer service, then the users arehosts in the Admin Subnet
PS + SU For a specific service, among which hosts isthe traffic exchanged?
fdpg ) fsa; dag fdp ¼ SharedPrinterg ) fsa ¼ AdminSubnet; da ¼ PrintServerg means that,when the Shared Printer service is concerned, then it is requested by hostsin the Admin Subnet and offered by the Print Server
PS + SU Given a source and a destination addresses,which services are concerned?
fda; sag ) fdpg fda ¼ PrintServer; sa ¼ AdminSubnetg ) fdp ¼ SharedPrinterg means that,when the Shared Printer service is concerned, then, it is offered by thePrint Server, and the users are hosts in the Admin Subnet
Table 1Length-2 rule classes.
Class Question Ruletemplate
Example
TF Given a destination address, by which hosts is itcontacted?
fdag ) fsag fda ¼ PrintServerg ) fsa ¼ AdminSubnetg means that the Print Server iscontacted by hosts in the Admin Subnet
TF Given a source address, which hosts is itcontacting?
fsag ) fdag fsa ¼ AdminSubnetg ) fda ¼ PrintServerg means that hosts in the AdminSubnet are contacting the Print Server
PS Given a destination address, which services doesit provide?
fdag ) fdpg fda ¼ PrintServerg ) fdp ¼ SharedPrinterg means that the Print Serveroffers a Shared Printer service to connect to
PS Given a destination port (i.e., a specific service),which hosts provide such service?
fdpg ) fdag fdp ¼ SharedPrinterg ) fda ¼ PrintServerg means that the Shared Printerservice is offered by the Print Server
SU Given a source address, which services does itrequest?
fsag ) fdpg fsa ¼ AdminSubnetg ) fdp ¼ SharedPrinterg means that hosts in theAdmin Subnet request the Shared Printer service
SU Given a destination port (i.e., a specific service),which hosts request such service?
fdpg ) fsag fdp ¼ SharedPrinterg ) fsa ¼ AdminSubnetg means that the SharedPrinter service is requested by hosts in the Admin Subnet.
780 D. Apiletti et al. / Computer Networks 53 (2009) 774–789
The source and destination addresses represent funda-mental information for tracing network flows, and the des-tination port is typically associated with a specific service1
(e.g., well known ports). Given this information, three basicclasses of rules may be defined to analyze the following fea-tures of network traffic.
Traffic flows (abbreviated TF) can be analyzed by rulesinvolving source and destination addresses.
Provided services ðPSÞ are shown by rules consisting ofdestination port (i.e., the service) and destination address(i.e., the service provider).
Service usage ðSUÞ is reported by rules having destina-tion port and source address (i.e., the service user).
To define the semantic classes that categorize the rules,in Table 1 we analyze all the possible length-2 rule tem-plates (i.e., templates for rules involving 2 tagged items)
1 The association of the destination port to a specific service is restrictedto the well-known and registered ports. However, throughout the paper wegenerally describe rule templates considering the included tags (e.g.,destination port) and not their actual value (e.g., actual port number).Hence, if a rule includes the destination port tag, then it is classified asproviding service information, independently of the port number.
built from the three chosen tags, while in Table 2 all thelength-3 rule templates are reported.
Rules which are not classified by the above classes mayfall into two categories. They may be a specialization of thebasic classes, or they may reference different tags. Special-izing rules add one or more attributes to the rules identi-fied for the basic classes (see Tables 1 and 2). Forexample, consider also the attributes source port ðspÞ andsize ðszÞ. The specializing rule template fda; dpg )fsa; sp; szg adds the sp and sz tags to the rule templatefda; dpg ) fsag. Hence, it specializes the traffic flowand service usage classes by considering also correlationswith the source port and the size of exchanged data.The rule template fspg ) fszg does not involve any of thethree tags on which classes are built. However, it canreveal unexpected correlations, extracted by an explor-atory analysis because of a high support or a highconfidence.
Example 1 – support and aggregation. Consider thefollowing generalized association rule with supports ¼ 2% and confidence c ¼ 100%.
fsa ¼ AdminSubnet;dp ¼ SharedPrinterg) fda ¼ PrintServerg
Table 3Characteristics of the datasets.
ID # of packets Trace file size [MB] Capture duration
A 25969389 2621 41’ 15sB 26023835 2625 41’ 56s
D. Apiletti et al. / Computer Networks 53 (2009) 774–789 781
It belongs to the classes TF (traffic flow between the AdminSubnet and the Print Server) and PS (it concerns a serviceprovided by the Print Server). The rule highlights that theAdmin subnet always (confidence is 100%) uses the PrintServer when the Shared Printer service service is con-cerned. Since an aggregation is performed for the sourceaddress (from single host to whole subnet), we can con-clude that no host in the Admin subnet generates enoughtraffic to exceed the minimum support threshold. Notethat if the proposed generalized rule approach had notbeen applied, neither the rule with the host, nor the rulewith the subnet as source address would have beengenerated. h
Example 2 – confidence. Consider the following gener-alized association rule with confidence c ¼ 80%.
fda ¼ PrintServerg) fdp ¼ SharedPrinter; sa ¼ AdminSubnetg
The rule confidence value highlights a usage profile where80% of the traffic directed to the Print Server comes fromhosts in the Admin Subnet which request the Shared Prin-ter service. This strongly characterizes the service providedby the Print Server (class PS) correlated with a specific traf-fic flow (class TF).
In a different network context, we could have obtainedthe following couple of rules.
fda ¼ PrintServerg) fdp ¼ SharedPrinter; sa ¼ AdminSubnetg c ¼ 40%
fda ¼ PrintServerg) fdp ¼ SharedPrinter; sa ¼ ResearchSubnetg c ¼ 30%
In this case, of all the traffic toward the Print Server, 40% isfrom the Admin Subnet, 30% is from the Research Subnet,and both subnets request the Shared Printer service. Thus,we also expect a new rule stating that at least 70% of thetraffic directed to the Print Server request the Shared Prin-ter service. Such rule has the following form (class PS).
fda ¼ PrintServerg ) fdp ¼ SharedPrinterg c ¼ 70% �
The former example shows an important property ofassociation rules. Lowering the minimum confidencethreshold and increasing the minimum support drivesthe analysis toward more frequent patterns with weakercorrelation. In the network domain, this approach leadsto profiling the traffic of the top users/providers of thenetwork.
Vice versa, lowering the minimum support thresholdand keeping the confidence high leads to the identificationof hosts with particular traffic patterns, e.g., using or offer-ing very specific services and/or contacting only specificsubnets. In the following, examples of this behavior aregiven.
Example 3 – specializations. Consider the followingrule.
fsa ¼ internal-host; da ¼ external-host;dp ¼ 443g) fsz ¼ 254Kbytesgc ¼ 100%
It expresses information on a traffic flow for a specific ser-vice (https). An interesting (and unexpected) correlationlinks such conversation with a fixed size of the flows. Thismay disclose a property of the specific protocol version inuse by those hosts or, if such hypothesis is unacceptable(based on previous knowledge of the protocol itself), itmay be a symptom of improper use of a well known portfor a different service, which is worth to be further inves-tigated. Examples of such rules find evidence in Example3 of the experimental results (Section 3.5). h
Example 4 – specializations and aggregations. Con-sider the following rule.
fsa¼ external� network;da¼ internal� host;dp¼ 1000g)fsz¼ smallg c ¼ 100%
The rule highlights that flows from the external networkto a specific host on port 1000 always have a small size,where small is defined as an aggregation of the sizeattribute, as the external-network is an aggregation ofthe external hosts. This is a generalization over the sizeand source address attributes of a rule similar to Exam-ple 3. Aggregation performed on the sa attribute indi-cates that all the external hosts behave in the thesame way as identified by the rule. Aggregation on thesz attribute shows that packet sizes may vary, but smallpackets are always exchanged. Similar patterns are alsoreported in Example 3 of the experimental results (Sec-tion 3.5). h
3. Experimental validation
A set of experiments has been performed by applyingthe proposed techniques on network traffic datasets. Weanalyzed the effect of the support and confidence thresh-olds on various classes of generalized rules (Section 3.1),the impact of the window size parameter (Section 3.2)and the effectiveness of our generalization approach inextracting hidden knowledge (Section 3.3). The selectionof interesting rules by means of the lift quality index is dis-cussed in Section 3.4. Section 3.5 presents the analysis of asubset of interesting rules in the network context andsome interesting analysis scenarios. Finally, the feasibilityof the online rule extraction process is discussed in Section3.6, in which generalized association rules are mined fromtraces of different duration.
Two network traffic datasets have been obtained by per-forming two capture sessions by means of the Analyzer tool[28] on a backbone link of our campus network. In Table 3,the number of packets, the trace file sizes (including packetheaders only), and the capture durations are reported foreach dataset. For each flow, the network attributes consid-ered in the analysis are source address, destination address,and destination port, together with source port and flow size(i.e., the size in byte of the traffic flow).
782 D. Apiletti et al. / Computer Networks 53 (2009) 774–789
Experiments addressed the execution of the samplequery presented in Section 2.1. To avoid discarding pack-ets, a proper buffer size has to be determined. The buffermust be able to store all possible flows in a time window,whose worst case value is the maximum number of cap-tured packets (i.e., each packet belongs to a different flow).Thus, the buffer size has been set to
link speed�window sizeminimum packet size
number of flows.Experiments have been performed considering a win-
dow size of 60 s and a link speed of 100 Mbps. The valueof the window step step has been set to window size
2 . The ratioparameter in the sample query (Section 2.1) has been setto 0.1.
Fig. 3. Dataset A: Number of extracted rules for different values ofminimum support with minimum confidence = 20%.
Fig. 4. Dataset B: Number of extracted rules for different values ofminimum support with minimum confidence = 20%.
Generalized itemset mining is performed by means ofthe Genio algorithm [6], which is able to generalize andaggregate infrequent items according to given taxonomies.The taxonomies used in this set of experiments aggregateinfrequent items according to the following strategies.TCP ports are aggregated into three ranges: well knownports (between 1 and 1023), registered ports (between1023 and 49,151) and dynamic ports (otherwise). IP ad-dresses which are local to the campus network are aggre-gated by subnet. IP addresses which do not belong to thecampus network are aggregated in a general external ad-dress group. Other aggregation strategies, for example dri-ven by geographical location, can be considered as well.The flow size attribute is aggregated over a taxonomy of 4
Fig. 5. Dataset A: Number of extracted rules for different values ofminimum confidence with minimum support = 1%.
Fig. 6. Dataset B: Number of extracted rules for different values ofminimum confidence with minimum support = 1%.
Table 4Number of flows yielded by the sample continuous query on each dataset.
Dataset ID # of selected flows
A 18,051B 16,783
D. Apiletti et al. / Computer Networks 53 (2009) 774–789 783
bins, whose intervals are [1,1000), [1000, 2000), [2000,3000) and P 3000 bytes.
Experiments have been performed on a 2.8 Ghz Pen-tium IV PC with 2 Gb main memory running Linux (kernel2.6.81). The current implementation of the NETMINE frame-work allows real-time execution of the stream processingat a network speed of 100 Mbit/s, with realistic packetsizes and network loads. Experiments have been per-formed on both datasets A and B. However, for some spe-cific experiments we only report results for dataset A,since they are analogous for dataset B.
3.1. Effect of the support and confidence thresholds
Different minimum support and confidence thresholdssignificantly affect the cardinality of the extracted ruleset and the nature of the rules. In Figs. 3–6 we report, forthe two datasets, the number of extracted rules for differ-ent combinations of the threshold values. Rules are dividedin three categories: basic rules, i.e., rules belonging to thesix classes described in Section 2.3 (see Tables 1 and 2),specializing rules, which add one or more tags to the for-mer rule templates (see Section 2.3), and other rules. Theabsolute number of extracted rules is different in the twodatasets, because of the different number of flows (see Ta-ble 4) and the different sparsity of original data. However,the global trend is rather similar.
As expected, the number of rules decreases when boththe minimum support and the minimum confidence thresh-olds are increased. The support threshold is rather selective.For high support values, only a small number of rules is ex-tracted (see Figs. 3 and 4). However, it is important to notethat frequent is not necessarily a synonym for interesting.A rather high number of strong correlations is instead ex-tracted also for high confidence values (see Fig. 5).
Fig. 7. Dataset A: Basic rules extracted with minimum confidence = 20%and minimum support = 1%.
The cardinality of the set of specializing rules is muchhigher than other categories. In particular, the number ofspecializing rules with low support is rather high (see Figs.3 and 4). This effect may be due to the loose constraints ontheir structure. These rules have not a fixed length andthey may include many different combinations of attri-butes. Thus, for low support thresholds, a large variety ofcombinations may satisfy the support constraint. Theserules are suitable for capturing unexpected peculiarknowledge. Furthermore, their meaning may be more eas-ily interpretable by exploiting their specialization of thebasic classes.
Many examples of specializing rules highlight correla-tions relating basic attributes (da, sa, dp) with the size attri-bute. This effect is referable to the particular taxonomy ofthe size tag. Its values are discretized into four bins only,leading to a very dense aggregation. Hence, each singleaggregation value becomes rather frequent. Diverse dis-cretization techniques or intervals may lead to a differentbehavior. A similar behavior is shown by the source portattribute, which is often aggregated as registered ordynamically-assigned. This reveals the allocation policyfor the client source port, which is typically dynamicallyassigned on the client host, always excluding the well-known ports.
The support and confidence thresholds also affect thekind of extracted rules. By setting high support thresholds,only very frequent patterns are extracted. However, theirinterest may be marginal. For example, when the supportthreshold is set to 10% and the confidence to 20%, the basiccategory contains four rules. One of the extracted rules is
fsa ¼ external-addressg ) fdp ¼ registered� portg
which is characterized by a support of 57% and a confi-dence of 87%. To satisfy the high selectivity of the mini-mum support threshold, the generalization process hasled to a rule which is too general to provide interestingknowledge. Instead, the use of a low support thresholdcoupled with different quality indices (e.g., confidence, lift)leads to the extraction of a higher number of rules wherepeculiar patterns arise. This issue is further discussed inSection 3.4, while Section 3.5 provides some interestingexamples of rules which are extracted by lowering the sup-port threshold.
Focusing on basic rules, in Fig. 7 their distributionamong the six classes is reported for dataset A. For rulesof length 2 (i.e., rules belonging to TF, PS, and SU classes)the TF class is by far the most frequent. This is an expectedbehavior because the cartesian product of the domains ofthe tags source-address and destination-address has a muchhigher cardinality than that of the domains of the othertags. Furthermore, their aggregations over the taxonomyhave a finer granularity (i.e., there are more subnets thanfamilies of ports), thus further increasing the domain car-dinality. On the contrary, rules of length 3 are equally dis-tributed among the three classes (TF + PS, TF + SU, PS + SU).
3.2. Effect of the window size
The window size directly affects the aggregation of thetraffic data performed by the continuous query. The sam-
Table 5Number of flows obtained for different values of window size.
Dataset Window size (s) # of selected flows
A 10 93,648A 30 36,205A 60 18,051B 10 89,931B 30 33,610B 60 16,783
784 D. Apiletti et al. / Computer Networks 53 (2009) 774–789
ple continuous query (see Section 2.1) is characterized by aratio parameter whose absolute value varies for differentwindow sizes. This parameter is expressed as a percentageof the whole flow size in a window. Hence, changing thewindow size in the data stream block strongly affects theinput of the refinement analysis phase, because the ob-tained flows are characterized by a different data distribu-tion. It becomes thus very difficult to detect a correlationbetween the refinement analysis results and the variationof the window size.
In Table 5, the number of flows obtained for differentvalues of the window size is reported. The results confirmthe expected behavior on both datasets, with a decreasingflow number for increasing window size values.
Fig. 8 shows, for dataset B, the number of extracted ruleswith different settings of the window size. Changing thewindow size from 30 s to 60 s does not affect the numberof extracted rules, which varies only slightly. On the con-trary, when the window size is set to 10 s, the difference be-comes significant, because very few rules are extracted.
Unfortunately, a direct impact of the window size onthe final rule set could not be identified, because of thejoint effects of this parameter. Furthermore, the numberof extracted rules for different window size values is mod-ified by the effect of the window size on the the ratioparameter. Thus, general conclusions cannot be drawn.
3.3. Impact of generalization on the rule extraction process
Since the number of different values assigned to eachattribute is huge, a network trace is characterized by a very
Fig. 8. Dataset B: Number of rules extracted with minimum confi-dence = 20% and minimum support = 1%, by executing the query withdifferent values of window size.
sparse data distribution. In similar conditions, the numberof items which are frequent in the dataset is usually small.This problem usually affects the effectiveness of traditionalassociation rule extraction.
Figs. 9 and 10 show, for different settings of minimumsupport, the percentage of generalized association rulesand the percentage of specific rules, which are built usingnon-aggregated items. This experiment gives a measureof the number of rules which would have been discardedif a traditional approach had been used with the same sup-port threshold. Other values of minimum confidence yieldanalogous results, as rules with high confidence are ratheruniformly distributed over a wide support range.
Since in the NETMINE framework infrequent items areaggregated during the extraction process, the percentageof generalized rules increases when the support thresholdis increased. Especially for high support thresholds theextraction of generalized rules allows highlighting correla-tions that would otherwise remain hidden, because of theirlow support.
The main purpose of generalized association ruleextraction is providing an extended set of association rules,including rules which do not separately meet the supportthreshold but may contribute to a generalized rule, thusnot being discarded by the frequency threshold. A qualityindex may be then applied for selecting high quality rulesfrom this extended pool. We discuss in Section 3.4 how toexploit the lift quality index to rank the set of extractedrules and select the most interesting ones. Some of the se-
Fig. 9. Dataset A: Percentage of generalized and specific rules extractedfor different values of minimum support with minimumconfidence = 20%.
Fig. 10. Dataset B: Percentage of generalized and specific rules extractedfor different values of minimum support with minimum confidence = 20%.
Fig. 11. Dataset A: Execution time for the rules extraction, and capturetime for increasing number of data packets.
2 Actual network addresses have been masked for privacy reasons.
D. Apiletti et al. / Computer Networks 53 (2009) 774–789 785
lected rules would not have been extracted with tradi-tional approaches (i.e., without generalization).
3.4. Selection of the most relevant rules
By setting the minimum confidence threshold to 40%and the minimum support threshold to 0.05%, a total of9810 rules have been extracted from dataset A. A simpletechnique to focus user analysis is the selection of rulescontaining specific combinations of attributes, or specificvalues. However, this technique does not consider theoverall quality of rules. Hence, to support the user in theselection of the most interesting rules, many quality mea-sures of the goodness of a rule may be exploited [18]. Inthis experiment rules have been automatically sorted bymeans of the lift quality index, which is a measure ofhow much a rule is likely to be interesting. For an associa-tion rule in the form
A) B
the lift measure [18], is defined as
liftðA;BÞ ¼ sðA) BÞsðAÞsðBÞ
i.e., the ratio of the support of the rule to the product of thesupport of A and the support of B. If the resulting value is 1,then the itemsets A and B are not correlated, i.e., they arestatistically independent. Instead, if the resulting value isless than 1, the itemset A and B are negatively correlated.Otherwise, they are positively correlated, meaning that
the occurrence of one implies the occurrence of the other.This index can be used to rank rules. Exploiting the liftquality index, the top 10 rules2 in the previous set of9810 are reported in Table 7.
Some interesting application examples are among thesetop rules and will be discussed in the next section. The lastthree rules in this restricted high quality subset are gener-alized rules, which would not have been extracted withtraditional techniques. Aggregation allowed the general-ized rules to meet the minimum support and confidencethresholds, while the specific rules contributing to themwould not have been individually extracted.
3.5. Application examples
We discuss in the following some interesting analysisscenarios taken from the results presented in Section 3.3and from the same experiment replicated on dataset B.
Example. Rule number 1 in Table 6 is analyzed.
fsa ¼ x:x:x:119g ) fda ¼ 130:192:y:ygðs ¼ 4%; c ¼ 100%Þ
It belongs to class TF. In particular, it identifies a trafficflow from a specific external IP address to a single internalhost. Since the rule confidence is 100%, all the traffic com-ing from this external IP goes to the specific internal host.The following rule of class TF + SU (also extracted) providesfurther insight on this traffic flow.
fsa ¼ xx:xx:xx:xxg ) fda ¼ 130:192:yy:yy;dp ¼ registeredgðs ¼ 4%; c ¼ 100%Þ
In particular, this generalized association rule showsthat the internal host is contacted on different port num-bers but they all belong to the registered ones. We can con-
Table 7Dataset A: Execution times of the extraction process with respect to thecapture times of the processed traffic data.
Capture time (s) Execution time (s) Ratio
245 77 31%520 138 27%780 254 33%1047 342 33%
Table 6Dataset A: Top 10 rules according to the lift quality index.
# Rule Lift
1 fsa ¼ x:x:x:119g ) fda ¼ 130:192:y:yg 1792 fda ¼ 130:192:y:y; sp ¼ 8080g ) fsa ¼ x:x:x:70g 1473 fsa ¼ x:x:x:70g ) fsp ¼ 8080g 1104 fsa ¼ x:x:x:70;da ¼ 130:192:y:yg ) fsp ¼ 8080g 1105 fsa ¼ x:x:x:70g ) fda ¼ 130:192:y:yg 1066 fsp ¼ 8090g ) fsa ¼ 130:192:y:yg 90.07 fda ¼ 130:192:y:yg ) fsp ¼ 8080g 79.68 fda ¼ 130:192:y:yg ) fdp ¼ 57403; sa ¼ external� addressg 70.59 fdp ¼ 57403g ) fda ¼ 130:192:y:y; sp ¼ registeredg 70.510 fsa ¼ external� address; dp ¼ well� knowng ) fda ¼ 130:192:y:y; sp ¼ registeredg 46.6
786 D. Apiletti et al. / Computer Networks 53 (2009) 774–789
clude that the specific external IP is contacting exclusivelythe internal host, which we have found to be the campusVPN concentrator.
A domain expert may be interested in knowing whetherthe identified external IP is among the top users of the VPNconcentrator. For this purpose, extracted rules of classTF + PS like the following should be considered
fda ¼ 130:192:yy:yyg ) fsa ¼ xx:xx:xx:xx;dp ¼ registeredgðs ¼ 4%; c ¼ 62%Þ
This generalized rule indicates that 62% of the trafficdirected to the VPN concentrator comes from the externalIP address previously identified. Hence, this address isactually the top user of such service. The same conclusionsmight be reached by directly querying the network trafficdataset, but the formulation of the appropriate set of que-ries would require previous knowledge of the specific pro-files to be analyzed. On the contrary, the combination ofgeneralized association rules and their categorization isable to assist the network analyst in quickly determiningwhich patterns are worth investigating. h
Example. Rule number 8 in Table 6 is analyzed.
fda ¼ 130:192:zz:zzg) fdp ¼ 57403; sa ¼ External� addressgðs ¼ 1:4%; c ¼ 86%Þ
It belongs to class TF + PS. This generalized association rulehighlights an unconventional high-volume traffic towardsa specific host of the campus network. It targets a singleinternal IP address and a single port that is not associatedto any well known service. Furthermore, the rule is charac-terized by high confidence and support despite no aggrega-tions over the destination port and address. Two situationsmay describe the identified behavior: (i) the host may beoffering a kind of unconventional service, because mostof its traffic is actually directed to a single port and comesfrom different addresses of the external network, as shownby the generalization on the source address, or (ii) the hostis downloading data from different locations outside thecampus network with the peculiar characteristic of usingalways the same port. h
Example. From dataset B with minimum support 0.05%and minimum confidence 40%, the following specializingrules have been extracted.
fsa ¼ 130:192:a:b; da ¼ 217:c:d:166;dp ¼ 110g )fsz ¼ 67281bytesgðs ¼ 0:1%; c ¼ 100%Þfsa ¼ 130:192:e:f ;da ¼ 193:g:h:86;dp ¼ 443g )fsz ¼ 254223bytesgðs ¼ 0:1%; c ¼ 100%Þ
They identify conversations between two specific hosts (aninternal host and an external address) for two well knownservices, https (port 443) and pop3 (port 110). The peculiarfeature of these rules is that the identified conversationsalways entail a fixed size for the exchanged data. Thismay either reveal a property of the specific protocol ver-sion in use by these hosts, or it can be a symptom of impro-per use of a well known port for a different service, whichis worth to be further investigated. h
3.6. Feasibility of the online rule extraction
In the NETMINE framework, packets are processed by thecontinuous query block concurrently with data capture.The flows which are produced by this block are then pro-cessed off-line for the extraction of generalized associationrules.
To assess the latency between the data capture and theextraction of the generalized association rules, we per-formed an experiment measuring the time requested forthe latter phase. In this experiment, we considered a linkcapacity of 100 Mbps operating at the maximumtheoretical speed (i.e., packets in the trace are assumedto arrive with the minimum inter-packet gap). Thesample query has been executed on dataset A, with awindow size of 60s and the ratio parameter set to 0.1.The extraction process has been performed with mini-mum support threshold = 1% and minimum confidence =20%.
The results of the experiment are reported in Fig. 11,where the capture time is the duration of the capture with
D. Apiletti et al. / Computer Networks 53 (2009) 774–789 787
the assumption of maximum theoretical speed and theexecution time is the time required to perform the ruleextraction on captured data. The execution time is (almost)linear with respect to the capture time. The time requiredfor the extraction is always less than the time required forthe data capture. More specifically, the execution time isalways roughly 30% of the capture time, as reported in Ta-ble 7. Hence the refinement analysis could be performedconcurrently with the traffic capture in the following con-ditions: (i) a sliding window selects a subset of the flowson which the extraction process is executed, (ii) the slidingwindow is updated at an interval less than a third of itssize. In this way the results of the extraction process are al-ways ready before updating the sliding window with thenew captured data.
4. Related work
A significant effort has been devoted to the applicationof data mining techniques to the problem of network traf-fic analysis. The work described in [9] presents some theo-retical considerations on the application of data miningtechniques to network monitoring. Since the network traf-fic analysis domain is rather broad, research activities haveaddressed many different application areas, e.g., web loganalysis [37], enterprise-wide management [21], and traf-fic classification [14,29,33].
Traffic data categorization, addressed by means of clas-sification techniques, is an effective tool to support net-work management [29]. In general, classificationtechniques can be divided in supervised and unsupervised.While the first group requires previous knowledge of theapplication domain, i.e., new unlabeled traffic flows are as-signed a class label by exploiting a model built from trafficnetwork data with known class labels, the second does not.Furthermore, network traffic classification can be per-formed by analyzing different features: (i) packet payloads,(ii) traffic metrics, and (iii) statistical features computed ontraffic flows.
Traditional traffic classification techniques perform adeeper inspection of packet payloads [14,33] to identifyapplication signatures. To apply these approaches, thepayload must be visible and readable. Both assumptionslimit the feasibility of these approaches [29]. First of all,payloads could be encrypted, making the deep packetinspection impossible. Furthermore, the classifier has toknow the syntax of each application payload, to be ableto interpret it. The approach in NETMINE does not requirepacket payload inspection. It analyzes instead a set ofpacket header fields and some additional features of flows(e.g., sum of packet sizes, number of packets).
Other classification approaches which target traffic datacategorization only consider a set of statistical features foreach flow of packets (e.g., packet inter-arrival time, num-ber of new sessions). These analysis techniques automati-cally infer categorization of traffic data according to theapplication level protocols, or extract patterns of normaland abnormal traffic. Both unsupervised [25,38,8] andsupervised [3,26] classification techniques have beenexploited.
In [25], several statistics are collected (e.g., packetlength, inter-arrival time, connection duration) for eachflow. Flows characterized by similar statistics aregrouped together by means of the EM clustering algo-rithm [18]. Thus, according to the clustering structure,different types of traffic data are identified (e.g., HTTP,FTP, SMTP). The approach proposed in [38] also exploitsthe EM algorithm to identify the best cluster set fromthe training data. Given the clustering result, a Bayesianclassifier is exploited to classify new incoming data. In-stead the work in [8] exploits the K-Means algorithm togroup together flows which belong to the same applica-tion level protocol.
In the context of supervised learning algorithms, sev-eral techniques (e.g., Naive Bayesian classifiers [26],Bayesian neural networks [3]) have been exploited toclassify traffic data. These approaches perform the analy-sis by considering a very large set of statistics (e.g., 248statistics [26]). However, these techniques require previ-ous knowledge of the application domain (i.e., a set of la-beled data).
Differently from the above approaches, the NETMINE
framework provides an effective and compact explor-ative approach to support network traffic analysis andnetwork management. Traffic analysis discovers a set ofrules which describe the network behavior and sup-ports the network administrator in efficiently identifyingrelevant and unexpected correlations among trafficdata.
Continuous queries have been applied in the context ofnetwork traffic management to real-time monitoring ofnetwork behavior. In particular, they have been exploitedto detect congestions and their causes [4] and to supportload balancing [11]. In these works, network data analysisis directly performed by means of continuous queries,without data materialization and further data explorationby means of data mining techniques.
5. Conclusions
NETMINE is a framework to efficiently perform networktraffic analysis. It combines two phases: data stream anal-ysis and refinement analysis. The former reduces theamount of traffic data while the latter extracts relevantand unexpected correlations and recurrence of patternsamong data. The proposed technique exploits a taxonomyto drive the extraction process by automatically aggregat-ing extracted information according to the support thresh-old. Experiments proved the capability and theeffectiveness of NETMINE to highlight meaningful character-istics of network traffic.
Acknowledgements
We are grateful to Fulvio Risso for providing the realtraffic datasets captured from the campus network, andto Claudio Testa and Felice Iasevoli for developing partsof the NETMINE framework.
788 D. Apiletti et al. / Computer Networks 53 (2009) 774–789
References
[1] R. Agrawal, R. Srikant, Fast algorithms for mining association rules inlarge databases, in: Proceedings of the 20th International Conferenceon Very Large Data Bases, 1994, pp. 487–499.
[2] A. Arasu, S. Babu, J. Widom, The CQL continuous query language:semantic foundations and query execution, The VLDB Journal TheInternational Journal on Very Large Data Bases 15 (2) (2006) 121–142.
[3] T. Auld, A. Moore, S. Gull, Bayesian neural networks for internettraffic classification, IEEE Transactions on Neural Networks 18 (1)(2007) 223.
[4] S. Babu, J. Widom, Continuous queries over data streams, ACMSIGMOD Record 30 (3) (2001) 109–120.
[5] M. Baldi, E. Baralis, F. Risso, Data mining techniques for effective andscalable traffic analysis, in: Integrated Network Management, 2005,IM 2005, 2005 Ninth IFIP/IEEE International Symposium on, 2005,pp. 105–118.
[6] E. Baralis, T. Cerquitelli, V. D’Elia, Generalized itemset discovery bymeans of opportunistic aggregation, Technical report, Politecnico diTorino, 2008, <https://dbdmg.polito.it/twiki/bin/view/Public/NetworkTrafficAnalysis>.
[7] Bart Goethals, Frequent Pattern Mining Implementations, <http://www.adrem.ua.ac.be/goethals/software>.
[8] L. Bernaille, I. Akodkenou, A. Soule, K. Salamatian, Trafficclassification on the fly, ACM SIGCOMM Computer CommunicationReview 36 (2) (2006) 23–26.
[9] K. Burn-Thornton, J. Garibaldi, A. Mahdi, Pro-active networkmanagement using data mining, in: Global TelecommunicationsConference, 1998, GLOBECOM 98, vol. 2, 1998.
[10] J. Chen, D.J. DeWitt, F. Tian, Y. Wang, Niagaracq: a scalablecontinuous query system for internet databases, in: W. Chen, J.F.Naughton, P.A. Bernstein (Eds.), Proceedings of the 2000 ACMSIGMOD International Conference on Management of Data, May16–18, 2000, Dallas, Texas, USA, ACM, 2000, pp. 379–390.
[11] N.G. Duffield, M. Grossglauser, Trajectory sampling for direct trafficobservation, IEEE/ACM Transactions on Networking 9 (3) (2001)280–292.
[12] J. Erman, M. Arlitt, A. Mahanti, Traffic classification using clusteringalgorithms, in: MineNet ’06, ACM Press, New York, NY, USA, 2006,pp. 281–286.
[13] Y. Guan, A. Ghorbani, N. Belacel, Y-Means: a clustering method forintrusion detection, Proceedings of Canadian Conference onElectrical and Computer Engineering (2003) 4–7.
[14] P. Haffner, S. Sen, O. Spatscheck, D. Wang, ACAS: automatedconstruction of application signatures, in: Proceedings of the 2005ACM SIGCOMM Workshop on Mining Network Data, ACM, NewYork, NY, USA, 2005, pp. 197–202.
[15] J. Han, Y. Fu, Mining multiple-level association rules in largedatabases, IEEE Transactions on Knowledge and Data Engineering11 (5) (1999) 798–804.
[16] B. Harris, R. Hunt, TCP/IP security threats and attack methods,Computer Communications 22 (10) (1999) 885–897.
[17] M. Hossain, S. Bridges, R. Vaughn Jr., Adaptive intrusion detectionwith data mining, IEEE Internation Conference on Systems, Man andCybernetics 4 (2003).
[18] Jiawei Han, Micheline Kamber, Data Mining: Concepts andTechniques, Series Editor Morgan Kaufmann Publishers, TheMorgan Kaufmann Series in Data Management Systems, Jim Gray,March 2006.
[19] T. Karagiannis, K. Papagiannaki, M. Faloutsos, Blinc: multilevel trafficclassification in the dark, in: SIGCOMM, 2005, pp. 229–240.
[20] M. Kaya, R. Alhajj, Mining multi-cross-level fuzzy weightedassociation rules, in: IEEE Conference on Intellectual Systems,2004, pp. 225–230.
[21] A. Knobbe, D. Van der Wallen, L. Lewis, Experiments with datamining in enterprise management, Integrated NetworkManagement, 1999, Distributed Management for the NetworkedMillennium, in: Proceedings of the Sixth IFIP/IEEE InternationalSymposium on, 1999, pp. 353–366.
[22] F. Le, S. Lee, T. Wong, H.S. Kim, D. Newcomb, Minerals: using datamining to detect router misconfigurations, in: MineNet ’06, ACMPress, New York, NY, USA, 2006, pp. 293–298.
[23] F. Lisi, D. Malerba, Inducing multi-level association rules frommultiple relations, Machine Learning 55 (2) (2004) 175–210.
[24] S. Madden, M. Shah, J.M. Hellerstein, V. Raman, Continuouslyadaptive continuous queries over streams, in: SIGMOD ’02:Proceedings of the 2002 ACM SIGMOD International Conference onManagement of Data, 2002, pp. 49–60.
[25] A. McGregor, M. Hall, P. Lorier, J. Brunskill, Flow clustering usingmachine learning techniques, Lecture Notes in Computer Science(2004) 205–214.
[26] A. Moore, D. Zuev, Internet traffic classification using bayesiananalysis techniques, ACM SIGMETRICS Performance EvaluationReview 33 (1) (2005) 50–60.
[27] A.W. Moore, D. Zuev, Internet traffic classification using bayesiananalysis techniques, in: SIGMETRICS ’05, ACM Press, New York, NY,USA, 2005, pp. 50–60.
[28] NetGroup, Politecnico di Torino. Analyzer 3.0. <http://analyzer.polito.it>.
[29] T.T. Nguyen, G. Armitage, A survey of techniques for internet trafficclassification using machine learning, IEEE Communications Surveysand Tutorials 10 (4) (2008) 56–76.
[30] A. Patcha, J.-M. Park, Network anomaly detection with incompleteaudit data, Computer Networks 51 (13) (2007) 3935–3955.
[31] A. Patcha, J.-M. Park, An overview of anomaly detection techniques:existing solutions and latest technological trends, ComputerNetworks 51 (12) (2007) 3448–3470.
[32] L. Portnoy, E. Eskin, S. Stolfo, Intrusion detection with unlabeled datausing clustering, in: Proceedings of ACM CSS Workshop on DataMining Applied to Security, PA, November 2001.
[33] S. Sen, O. Spatscheck, D. Wang, Accurate scalable in-networkidentification of p2p traffic using application signatures, in:Proceedings of the 13th International Conference on World WideWeb, ACM, New York, NY, USA, 2004, pp. 512–521.
[34] R.A. Tansel Ozyer, K. Barker, Intrusion detection by integratingboosting genetic fuzzy classifier and data mining criteria for rulepre-screening, Journal of Network and Computer Applications 30 (1)(2007) 99–113.
[35] Telecommunication Networks Group, Politecnico di Torino. Tstat1.01, <http://tstat.polito.it>.
[36] Q. Wang, V. Megalooikonomu, A clustering algorithm for intrusiondetection, Proceedings of SPIE 5812 (2005) 31–38.
[37] Q. Yang, H. Zhang, Web-log mining for predictive web caching, IEEETransactions on Knowledge and Data Engineering 15 (4) (2003)1050–1053.
[38] S. Zander, T. Nguyen, G. Armitage, Automated traffic classificationand application identification using machine learning, Conferenceon Local Computer Networks, vol. 30, IEEE, 2005, p. 250.
Daniele Apiletti is a Ph.D. student of theDatabase and Data Mining Group at theDipartimento di Automatica e Informatica ofthe Politecnico di Torino since January 2006.He holds a Master degree in ComputerEngineering from Politecnico di Torino (2005).His research interests are in the fields ofbioinformatics and sensor data analysis. In thebioinformatics area, his research activities arefocused on microarray data classificationand feature selection techniques. In thearea of sensor data analysis he is developing
data mining techniques for physiological data processing on mobiledevices.
Elena Baralis is full professor at the Diparti-mento di Automatica e Informatica of thePolitecnico di Torino since January 2005. Sheholds a Dr.Ing. degree in Electrical Engineer-ing and a Ph.D. in Computer Engineering, bothfrom Politecnico di Torino. Her currentresearch interests are in the field of databases,in particular data mining, sensor databases,and bioinformatics. She has publishednumerous papers on journal and conferenceproceedings. She has served on the programcommittees of several international confer-
ences and workshops, among which IEEE ICDM, VLDB, ACM CIKM,DaWak, ACM SAC, PKDD. She has managed several Italian and EU researchprojects.
D. Apiletti et al. / Computer Networks 53 (2009) 774–789 789
Tania Cerquitelli is a post-doctoral researcherin Computer Engineering at the Politecnico diTorino since January 2007. She holds a Ph.D.degree and a Master degree in ComputerEngineering, both from the Politecnico diTorino. She also obtained a Master degree inComputer Science from Universidad De LasAméricas Puebla. Her research activities arefocused on the integration of data miningalgorithms into database system kernels andon the exploitation of data mining techniquesto analyze streaming data collected by sensor
networks. She served in the program committee of DS2ME’08 andIADIS’06.
Vincenzo D’Elia is a Ph.D. student of theDatabase and Data Mining Group at theDipartimento di Automatica e Informatica ofthe Politecnico di Torino since January 2007.He obtained a Master degree in ComputerEngineering from Politecnico di Torino inNovember 2006. He is currently working inthe field of data mining, and time seriesanalysis. In particular, his activity is focusedon the analysis of sensor network data tosupport effective data collection by means ofdata mining techniques. His research activi-
ties are also devoted to analyze physiological signals to study athleteconditions while they perform their activities.
