“ MEHARI: A System for Analysing the Use of the Internet Services ” Presented by: Arturo Azcorra, Josep Solé-Pareta MEHARI Partners: UC3M, UPC and UPM

MEHARI - 1UC3M / UPC / UPM

M e h a r i

““MEHARI: A System for Analysing MEHARI: A System for Analysing the Use of the Internet Servicesthe Use of the Internet Services””

Presented by: Arturo Azcorra, Josep Solé-ParetaPresented by: Arturo Azcorra, Josep Solé-Pareta

MEHARI Partners: UC3M, UPC and UPMMEHARI Partners: UC3M, UPC and UPM


M e h a r i

MEHARI Project ObjectivesMEHARI Project Objectives Traffic Capture SubsystemTraffic Capture Subsystem

High SpeedHigh Speed AAL5 ReasemblyAAL5 Reasembly Modular and scalableModular and scalable Low costLow cost

Support for many Traffic Analysis tools:Support for many Traffic Analysis tools: Detailed analyisis (including contents for AUP audits)Detailed analyisis (including contents for AUP audits) Identification and agreggation of bidirectional flows Identification and agreggation of bidirectional flows Traffic classification by usageTraffic classification by usage Traffic classification by origin / destinationTraffic classification by origin / destination Internet header verificationInternet header verification ......


M e h a r i

MEHARI System Analysis Subsystem

TrafficSamples

Analysis Platform(s)

MEHARI Functional Architecture MEHARI Functional Architecture M e h a r i

Capture Subsystem

ATM 1

ATM 0ATM 1

ATM 0

Capture Platform(s)

ATM 1

ATM 0

ATMBackbone

ATMCells

Capturepoint

PPSPPSApplicationModules

IP Biflows+ symptoms

Statisticsand Reports

Auto-regulation

PreprocessingModule

Data base -patterns- addresses- ...

Operator


M e h a r i

Capture SubsystemCapture Subsystem Modular and scalableModular and scalable

N units over the same or different trunk linksN units over the same or different trunk links Requires high speed connection to the analysis Requires high speed connection to the analysis

subsystemsubsystem Senses ALL VPI/VCI in the fiberSenses ALL VPI/VCI in the fiber

Captures in promiscuous or filtered mode over VPI/VCI Captures in promiscuous or filtered mode over VPI/VCI listlist

Capture capacity for each unit Capture capacity for each unit Sustained Average of 8 Mbit/s for a 6,000 Euros unitSustained Average of 8 Mbit/s for a 6,000 Euros unit 3,000% better price/performance than commercial 3,000% better price/performance than commercial

protocol analyzersprotocol analyzers Capture rate controled by analysis rateCapture rate controled by analysis rate


M e h a r i

Information RegisteredInformation Registered

frameseq_num

timestamp UNIX(seg.µseg) VPI/VCI

length (bytes) Truncated AAL5info field

0:893083746.654070:100/1:1064 :45000428E81B40002F062E36C600B...1:893083746.654090:100/1:44:4500002C00AC400037069CF5CC4B3C...2:893083746.654101:100/1:40:45000028455840003606052FCF4F2C1...3:893083746.654280:103/224:1500:450005DC6C4B4000FD06142640...4:893083746.654288:103/224:40:45000028240440007B06401E829FD...5:893083746.654517:103/224:400:45000190B30340001D06B516238A.........1668:893083746.813551:100/1:281:4500011976710000FB04BFFCE40...#init_time=893083746.652986final_time=893083746.813582cap_time=0.160596

Files with programmable granularity


M e h a r i

Pre-processing ModulePre-processing Module

Main functionsMain functions pseudo-packet agreggation to flowspseudo-packet agreggation to flows pseudo-packet analysispseudo-packet analysis count of symptoms associated to each flowcount of symptoms associated to each flow

Produces flow list with associated information:Produces flow list with associated information: flow desc with packet and byte countflow desc with packet and byte count weighted list of symptomsweighted list of symptoms

Highly configurable:Highly configurable: symptom definition and inter-relationsymptom definition and inter-relation aggregation periodaggregation period


M e h a r i

Classification ModuleClassification Module

Current categories:Current categories: LEISURE, COMMERCIAL, ACADEMIC, UNKNOWNLEISURE, COMMERCIAL, ACADEMIC, UNKNOWN

Current heuristics (human auditing):Current heuristics (human auditing): 1º ‘known’ addresses1º ‘known’ addresses

- e.g.: banks (COM), playboy (LEI), sports newspapers (LEI)e.g.: banks (COM), playboy (LEI), sports newspapers (LEI) 2º dominant symptoms2º dominant symptoms

- e.g.: HTTP=2, PASSWD=3, VISA=1 (COM)e.g.: HTTP=2, PASSWD=3, VISA=1 (COM)- e.g.: MAIL=1, CHAT=4, SEX=3 (LEI)e.g.: MAIL=1, CHAT=4, SEX=3 (LEI)

3º non standard ports3º non standard ports- e.g.: ftp over ports other than 20/21 (UNK)e.g.: ftp over ports other than 20/21 (UNK)

4º ‘known’ ports 4º ‘known’ ports - e.g.: 6969 (LEI) e.g.: 6969 (LEI)

Academic by default


M e h a r i

Traffic origin/destination analysis Traffic origin/destination analysis modulemodule

IP Biflows

Identificationof AS

Subnetwork,CIDR, ASs, ...

Databases

Traffic Origin/Destination Analysis Module (TODM)

Pre-processing Module (TCM)

Processor

SummaryReport Files

Official IRRData Bases

NRN BGP

other...


M e h a r i

Internet headers analysis moduleInternet headers analysis module M e h a r i

Internet Headeranalysis

(session oriented)

Capture Files

Pre-processing

Unknown TrafficProcessor

Data basewith header patterns

Summary Report Files

Internet Header Analysis Module (IHM)

- % Verified traffic- % Pending traffic

Summary Report Files(unknown traffic)

Summary Report Files

-Remote and local servers


M e h a r i

Modularity and Scalability of MEHARIModularity and Scalability of MEHARI

P 1.1

P 1.2

P 1.3

P 1.1.2

P 1.1.1

P 1.3.1

P 1.1.3

Process tree structure for information flowProcess tree structure for information flow Interprocess Comunication using shared filesInterprocess Comunication using shared files May be distributed among several machines using NFSMay be distributed among several machines using NFS


M e h a r i

Some applications of these toolsSome applications of these tools

Traffic monitoringTraffic monitoring Billing and charging models for NRN and Corporate Billing and charging models for NRN and Corporate

NetworksNetworks Network configurationNetwork configuration

- Resources dimensioning Resources dimensioning - Placing Proxies, ... Placing Proxies, ...

Service usage controlService usage control Control that the services are used responsibly, i. e. Control that the services are used responsibly, i. e.

auditing the academic networks AUP (Acceptable auditing the academic networks AUP (Acceptable Use Policy)Use Policy)

SecuritySecurity


M e h a r i

ConclusionsConclusions Modular, scalable and extensible architectureModular, scalable and extensible architecture Capture systems with excelent price/performanceCapture systems with excelent price/performance Flow information aggregation with symptoms and Flow information aggregation with symptoms and

bidirectional flow correlationbidirectional flow correlation Intermediate data base of patterns and addressesIntermediate data base of patterns and addresses Application modules currently implemented:Application modules currently implemented:

Classification by usage (AUP)Classification by usage (AUP) Classification by origin/destinationClassification by origin/destination Internet header analysisInternet header analysis


M e h a r i

Future workFuture work Further improvements in capture capacityFurther improvements in capture capacity Applications to detect security attacksApplications to detect security attacks Graphical user interfaceGraphical user interface Automatic reaction to incidents:Automatic reaction to incidents:

Alarms (mail, pager, SNMP, ...)Alarms (mail, pager, SNMP, ...) Flow blocking or re-routingFlow blocking or re-routing Flow logging for off-line human analyisisFlow logging for off-line human analyisis

Other type of statistics:Other type of statistics: Traffic statistics, as those provided by the NetFowTraffic statistics, as those provided by the NetFow Top 100 lists of hosts/serversTop 100 lists of hosts/servers Main origins/destinations of trafficMain origins/destinations of traffic Most popular sites (webs, ftps, chat servers, ...)Most popular sites (webs, ftps, chat servers, ...)


M e h a r i

Trial on Spanish NRN: RedIrisTrial on Spanish NRN: RedIris

GIGACOMTelefónica ATM

Network

RedIRISRegionalNodes

Splitters

ATM Access Switch

Analysis PC(LINUX)

100 BaseTEthernet

NFSInternet(RedIris)

Remote AccessTraffic Capture PC

(FreeBSD)

1

0

STM-1 ATMOptical

Interfaces

RedIRISCore

Router

RedIRIS: the Spanish NRN


M e h a r i

Sample of Results: Sample of Results: Traffic classification by usage (I)Traffic classification by usage (I)

% Bytes (Input traffic)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Academic Leisure Commercial Unknown

User Groups (17)


M e h a r i

Sample of Results:Sample of Results:Traffic classification by usage (II)Traffic classification by usage (II)

Total Input traffic to RedIRIS (% Bytes)

Academic78%

Leisure17%

Unknown2%Commercial

3%Leisure

12%

Commercial2%

Total Output traffic to RedIRIS (% Bytes)

Unknown2%

Academic84%


M e h a r i

Sample of Results: Sample of Results: Main traffic origin/destination (I)Main traffic origin/destination (I)

RedIRISTEN-34/155IbernetRest of Internet (through USA)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%% Bytes (Input traffic)

User Groups (17)


M e h a r i

Sample of Results: Sample of Results: Main traffic origin/destination (II)Main traffic origin/destination (II)

Total Input traffic to RedIRIS

26%

21%

12%

41%36%

16%21%

27%

RedIRISTEN-34/155IbernetRest of Internet (through USA)

Total Output traffic from RedIRIS


M e h a r i

Sample of Results: Sample of Results: % of a% of academic traffic in the link with USA cademic traffic in the link with USA

(according with the IRR description)(according with the IRR description)

0 %

10 %

20 %

30 %

40 %

50 %

60 %Input traffic

% o

f cap

ture

d tr

affic

User Groups (17)


M e h a r i

Sample of Results: Sample of Results: Top 25 most visited Top 25 most visited commercial sites incommercial sites in

one of the user groups one of the user groups

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

TSA

I

RE

DE

STB

OLE

ES

-CTV

-980

527

ES

-TTD

-951

020

CA

IXA

-RE

D

CA

NA

L-P

LUS

-SP

AIE

S

AB

F

ICTN

ET

IBE

RN

ETC

OM

ES

-FC

R-9

5060

7

JETN

ET

GR

N

CO

NE

XIS

IP-M

ULT

IME

DIA

SE

RV

ICO

M2-

NE

TS

AB

CTE

LEM

ATI

C

SP

RIT

EL

SE

RV

ICO

M1-

NE

TS

DA

UC

OM

2ME

G-E

S

RA

N

FUT

INFA

SE

RS

INTE

RC

OM

Oth

er S

ub-N

etw

orks

: 958

% Bytes (Input traffic to one of the user groups)


M e h a r i

Sample of Results (January-February´99): Sample of Results (January-February´99): Top 25 most visited TEN-155 ASs inTop 25 most visited TEN-155 ASs in

one of the user groups one of the user groups

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

AS

1275

DFN

-IP s

ervi

ce a

nd D

FN c

usto

mer

net

wor

ks

AS

786

The

JA

NE

T IP

Ser

vice

AS

1653

SU

NE

T S

wed

ish

Uni

vers

ity N

etw

ork

AS

1103

SU

RFn

et

AS

2856

BTn

et U

K R

egio

nal n

etw

ork

AS

224

UN

INE

TT, T

he N

orw

egia

n U

nive

rsity

& R

esea

rch

Net

wor

k

AS

1717

RE

NA

TER

AS

3301

Tel

iaN

et S

wed

en

AS

2852

CE

SN

ET

z.s.

p.o.

- TE

N34

-CZ

AS

513

CE

RN

AS

1853

AC

One

t Bac

kbon

e

AS

1239

AS

559

SW

ITC

H, S

wis

s A

cade

mic

and

Res

earc

h N

etw

ork

AS

8761

RE

TEN

ET

Aut

onom

ous

Sys

tem

AS

1741

FU

NE

T au

tono

mou

s sy

stem

AS

8743

Hig

hway

One

Aut

onom

us S

yste

m

AS

1835

DE

Net

- D

anis

h N

etw

ork

for R

esea

rch

and

Edu

catio

n

AS

3269

TE

LEC

OM

ITA

LIA

AS

6805

med

iaW

ays

Aut

onom

ous

Sys

tem

AS

3215

RA

IN

AS

5470

AU

TH-N

ET-

AS

AS

5556

Tel

enor

dia

AB

AS

8209

A20

00 /

Kab

elte

levi

sie

Am

ster

dam

bv

AS

2529

Dem

on In

tern

et L

td

AS

1290

PS

INet

UK

Ltd

.

Oth

er A

ss: 4

33

% Bytes (Input traffic to one of the user groups)


M e h a r i

Sample of Results : Sample of Results : Internet Headers Verification Internet Headers Verification

0.1 %

84.9 %

13.5 % 1.5 %

PendingVerifiedUnknownRejected

Documents

“ MEHARI: A System for Analysing the Use of the Internet Services ” Presented by: Arturo Azcorra, Josep Solé-Pareta MEHARI Partners: UC3M, UPC and UPM