Yi-Lei Chang Advisor: Dr. Kai-Wei Ke 2013/06/17

1

分散式網路事件分析記錄系統之研製

The Design and Implementation of

Distributed Network Event Analyzing and Recording

SystemYi-Lei Chang

Advisor: Dr. Kai-Wei Ke

2013/06/17

2

Outline

Introduction

Background

System Design

Compare to simulate system

Demo

Reference

3

Introduction

Network Event An observable occurrence on network that can be recognize as a

specific protocol activity or behavior (e.g., FTP Login, HTTP web browse).

System Goals Record and analyze network event

FTP

HTTP

VoIP

Abnormal behavior

Distributed system

High flexibility and extensibility

4

Background - Jpcap

Packages

Jpcap

JpcapCaptor

PacketReceiver

Jpcap.packet

ARPPacket

ICMPPacket

IPPacket

TCPPacket

UDPPacket

5

Background - Jpcap

Jpcap (Java API)

Jpcap.dll

WinPcap(Windows) / Libpcap(Linux)

Network Interface Card

Java application

6

System DesignInterception System

Analyzing and Recording System

Packet Capture

Packet Pool

Network functions

Protocol Parsers

TemporaryStorage

Network functions

HTML Analyzer

Voice Decoder

Storage

SQL DB

7

Interception System

Capture packets

Track relative connections

Record supported network event

8

Packet Capture

Receive packets from NIC in promiscuous mode

Set basic packet filter

IP

ARP

Not Interception System’s IP

Add packets to PacketPool

PacketCapture

Winpcap

PacketPool

JpcapCaptor

PacketRxer

9

Packet Pool

Maintain all packets capture by PacketCapture

Each ProtocolParser register to PacketPoolhave a random integer key to access it’spacket list iterator

Remove useless packets when buffer full

Synchronize needed

PacketPool

PacketCapture

PacketList

Key-Iterator Management

ProtocolPaeser

10

Protocol Parser

Abstract class ProtocolParser implements Runnable

Define basic steps for a standard protocol parser

Implement Runnable.run() with 4 abstract function called in sequence

isRelative()

processPacket()

isContinue()

endProcess()

The implementation of these abstraction function will change the use of class extends ProtocolParser(e.g., FTPProtocolParser).

11

Protocol Parserstart

Is relative?Try get a

Packet from PacketPool

Process Packet

Thread continue?

Get success?

Sleep some time

Ending process

Thread terminated

Y

N

Y

N

N Y

12

Protocol Parser - FTP

Relative: port 21

Process:

Create a connection key“clientIP|clientPort|hostIP|hostPort”for identify every FTP command connection

For every unhandled FTP connection create FTP command Parser

Continue: always

Ending process: unregister with PacketPool

13

Protocol Parser - FTP

Process Packet

Make keyPut key into handled map

Handled?New FTP command

Parser

Start command

Parser

Process Packet End

N

Y

14

Protocol Parser – FTP command

Relative: specific connection represent by connection key

Process: USER/PASS/230 – login event

PORT/227 – tract data connection

STOR/RETR – create FTP recorder to record transmitted file

Continue: Connection not close

Connection not idle

Ending process: Unregister with PacketPool

Remove handled state in FTP Parser

15

Protocol Parser – FTP commandProcess Packet

Take out command msg. from

packet

USER?

Process Packet End

PASS?PORT?227? STOR?RETR?230?

Record user account

Record user password

Log login event to DB

Record PASV connection

IP/Port

Record active connection

IP/Port

New FTP Recorder

Start FTP Recorder

Y

N

Y Y YY YY

N N N N N

16

Protocol Parser – FTP recorder

Relative: specific connection and direction represent IP and Port

Process: Put data packet to TCPReorderBuffer

Set acknowledge number for TCPReorderBuffer to reference

While buffer full flush data to file

Continue: Connection not close

Connection not idle

Ending process: Unregister with PacketPool

Flush all remain data in buffer to file

Log file transmit event into DB

17

Protocol Parser – FTP recorderProcess Packet

Process Packet End

Buffer full?

Data packet?

Set buffer ack_num

Put into buffer

Flush buffer in to file

Y

N

N

Y

18

TCPReordreBuffer

A buffer can store jpcap.packet TCPPacket and reorder packet’s data by sequence

Put:

TCPPacket

ack_number

Get:

in order packet TCP payload in byte array

Missing part info

19

TCPReordreBuffer - putPut Packet

Put packet in to map<sequence, packet>

Put Packet End

Put Ack_num

Put Packet End

Ack_num > stored ack_num?

Set ack_num

N

Y

20

TCPReordreBuffer - get

Get in order data

Get/sort all keys(seq.)

Get packet with smallest

seq.

PktSeq = nextSeq

PktSeq > nextSeq

Remove Packet from

map

Put packet in IOPacket list

PktSeq+PktDataLen >=

ack_num

Map empty?

Get in order data end

IOPacket list to byte array

Record missing part

Y

Y

Y

Y

N

N

N

21

Protocol Parser - HTTP

Relative: port 80

Process:

Create a connection key“clientIP|clientPort|hostIP|hostPort”for identify every HTTP connection

For every unhandled HTTP connection create HTTP recorder

Continue: always

Ending process: unregister with PacketPool

22

Protocol Parser – HTTP recorder

Relative: specific connection specific connection represent by connection key

Process: Put data packet to TCPReorderBuffer

Set acknowledge number for TCPReorderBuffer to reference

Cut HTTP header, record header information

Log HTTP event into DB

Store HTTP body into DB if its not too big

Continue: Connection not close

Connection not idle

Ending process: Unregister with PacketPool

Flush all remain data in buffer to file

Log file transmit event into DB

23

Protocol Parser H.323

Relative: port 1719(H.323RAS), port 1720(Q.931/H.225)

Process:

Maintain device list using gatekeeper RAS message

For every unhandled H245 connection create H245 Parser

Continue: always

Ending process:

Unregister with PacketPool

24

Protocol Parser H.245

Relative: specific connection represent by connection key

Process:

While openlogicchannel message detected, create RTP recorder

Continue:

Disconnect message undetected

Connection not idle

Ending process:

Unregister with PacketPool

Log calling event into DB

25

Protocol Parser RTP

Relative: specific UDP packet with specific source and destination

Process:

Record RTP content

Real-time decode/play if needed

Continue:

Disconnect message undetected

Connection not idle

Ending process:

Unregister with PacketPool

26

Protocol Parser - Abnormal behavior Relative: ICMP Packet, ARP Packet, TCP SYN packet

Process:

ICMP ping attack

Count ICMP packet for both source and destination

If > 3 ping packet/sec log ping attack event into DB

ARP attack

Record MAC/IP mappings

If MAC/IP mappings changing > 10 times/min log ARP attack event into DB

TCP SYN packet

Record SYN request, remove when 3 way established

If to many SYN request unestablished log SYN attack event into DB

Continue: always

Ending process: Unregister with PacketPool

27

Analyzing and Recording System

HTML page recovery

PCM decode

File storage and presentation

28

Analyzing and Recording System - HTTP Analyzer

Search http response with content-type text/html to get html page file

Search [src=“”] pattern in html file

Search relative http request in DB

Recover/rename relative file and replace links in html file

Cross match DB and html file to recover as much as possible

29

Improvement

FTP active/passive mode, upload, download support

HTTP absolute direct link resolve

H.323 support

Better program structure with higher flexibility and extendibility

30

Compare to other system 本系統 Wireshark ClearSight Analyzer

系統特性比較

使用者介面 簡易 複雜 複雜

開放原始碼 是 是 否

擴充性 高 高 低

價格 免費 免費 昂貴

系統功能比較

網路協定量統計 無 有 有

分散式架構 有 無 無

儲存側錄檔案支援檔案及

HTML頁面還原只針對封包內容儲存 只針對封包內容儲存

語音即時監聽 有 無 無

記憶體需求 小 大 大

可分析之協定 較少 多多

適合長時間之網路監測 是 否 否

31

Demo

32

Reference

[1]林佑民，「基於雲端運算之網路通訊監察分析系統之研製」，碩士論文，國立台北科技大學資訊工程系碩士班， 2012

[2]黃威穎，「 H.323網路電話音訊監控與錄製系統之研製」，碩士論文國立台北科技大學資訊工程系碩士班， 2008