IEEE IEMCON 2016 Keynote 0.4cm Improving Computer Network ... · IEEE IEMCON 2016 Keynote Improving Computer Network Monitoring, a brief Tour Ruediger Gad Terma GmbH, Space, Ground

IEEE IEMCON 2016 Keynote

Improving Computer Network Monitoring, a brief Tour

Ruediger Gad

Terma GmbH, Space, Ground Systems, Darmstadt, Germany

2016-10-15

Assure Operational Computer Networks

BasisInformation

DetailedAccurateUp-to-date. . .

Network Monitoring(“Network Reconnaissance” or “Network Analysis and Surveillance”)

Ruediger Gad – Terma GmbH, Darmstadt, Germany

IEEE IEMCON 2016 – Keynote – Improving Computer Network Monitoring, a brief Tour

Network Monitoring

Challenging

DistributionSizeChangeTimelinessData Volume. . .



What to improve in network monitoring?

Scope & Coverage(Overarching)

Convergence ofHeterogeneous DataSources

Flexibility

Performance

Complexity vs. Usability

. . .



How to improve network monitoring?

Integration of Technologies

Use the “best” tools.

Cooperation

Mitigate resource limitations.

Dynamic Distributed Network Monitoring

Scope, Coverage, and Flexibility

Event-driven Architecture (EDA) and Complex Event Processing (CEP)

Convergence of Data Sources, Data Processing, Flexibility, & Correlation

Domain Specific Language (DSL)

Data Extraction & Transformation, Ease Usage, Dynamic Capabilities

Self-adaptation

Usability, Responsiveness, and Operation






Cooperation








Self-adaptation







Cooperation








Self-adaptation







Cooperation








Self-adaptation







Cooperation








Self-adaptation







Cooperation








Self-adaptation





Integration of TechnologiesUse the “best” tools.

Cooperation








Self-adaptation




Integrating Technologies: “A Tale of Two Worlds”

“Lower-level” Data Acquisition

Raw Data, e. g., Packet Capturing (Pcap), Connection Tracking, NetflowClose to HardwareProgramming Languages: Assembly, C, C++, Rust, . . .

“Higher-level” FrameworksProvide abstractions for powerful functionality.

Communication, Persistence, Analysis, Processing, . . .

Proramming Languages: Java, Python, Ruby, Clojure, . . .

Use strengths from both worlds. → Combine both worlds.

Pcap & JVM as Example

Pcap as “Worst Case” Scenario



Network Monitoring (Pcap) & JVM, Use Cases

Real-time Support Vector Machine Based Network Intrusion Detection System Using Apache Storm, M.A. Manzoor, et al., IEMCON 2016

Monitoring Traffic in Computer Networks with Dynamic Distributed Remote Packet Capturing, R. Gad, etal., IEEE ICC 2015

Scalable Hybrid Stream and Hadoop Network Analysis System, V. K. C. Bumgardner, et al., 5thACM/SPEC ICPE 2014

An event-based platform for collaborative threats detection and monitoring, G. Lodi, et al., InformationSystems 2014

hadoop-pcap - Hadoop library to read packet capture (PCAP) files, RIPE-NCC, [Online] @ github, 2014

Event Stream Database Based Architecture to Detect Network Intrusion: (Industry Article), V. Kumaran,DEBS 2013

Hadoop-based multi-classification fusion for intrusion detection, X.-Y. Ren, et al., Journal of AppliedSciences, 2013

. . .



Packet Capturing Data Flow Overview

Based on: Comparing and Improving Current Packet Capturing Solutions Based on Commodity Hardware, L. Braun, et al., IMC 2010Improving Network Traffic Acquisition and Processing with the Java Virtual Machine, R. Gad, M. Kappes, and I. Medina-Bulo, 20th IEEE ISCC 2015

Clojure and Java Packet Capturing Library: https://github.com/ruedigergad/clj-net-pcap



https://github.com/ruedigergad/clj-net-pcap

Overview of Intermediate Double Buffering

Improving Network Traffic Acquisition and Processing with the Java Virtual Machine, R. Gad, M. Kappes, and I. Medina-Bulo, 20th IEEE ISCC 2015





Packet Capturing with the JVM: Improved Method vs. Old Method

0 1000 2000 3000 4000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 25 50 75 100

[kpp

s]

Rel

. Std

. Dev

. [%

]

Packet Size [x100 byte]

Th.Pkt.Rt. 1 Gbps [kpps]Cap.Rt. (Dbl.Buf.) [kpps]CR Rel.SD (Dbl.Buf.) [%]

Th.Pkt.Rt. 10 Gbps [kpps]Cap.Rt. (Non-B.) [kpps]CR Rel.SD (Non-B.) [%]






CooperationMitigate resource limitations.







Self-adaptation




Cooperation for Improving the Performance, Foundations

Header Field Based Partitioning of Network Traffic for Distributed Packet Capturing and Processing, R. Gad, et al., 28th IEEE AINA 2014



















Our Partitioning Approach

Criterion

Header Data

Operation

x mod yResidue Classes

Assigned to sensors.

x = Header Field Valuey = 2n

→ Bit-wise AND and Equality

E. g.: tcpdump −i eth0 ”ip[10:2] & 3 = 0”

Ressource Efficient






Cooperation


Dynamic Distributed Network MonitoringScope, Coverage, and Flexibility





Self-adaptation




Distributed Remote Packet Capturing (DRePCap)

Sensors: Hosts A to D

Controller: Host E

Communication InfrastructureLogicData MergingData Consumer

Traffic Generation

Host A → Host D

Monitoring Traffic in Computer Networks with Dynamic Distributed Remote Packet Capturing, R. Gad, et al., IEEE ICC 2015

Distributed Remote Packet Capturing (DRePCap): https://github.com/fg-netzwerksicherheit/drepcap



https://github.com/fg-netzwerksicherheit/drepcap

Cooperative Sensors: Performance, Scalability, and Traffic Load

0

25

50

75

100

100 200 300 400 500 600

Cap

ture

Rat

io [%

]

Packet Rate [kpps]1 Sensor2 Sensors

3 Sensors4 Sensors

0

100

200

300

400

500

600

100 200 300 400 500 600

Tra

ffic

Loa

d [M

bps]

Packet Rate [kpps]1 Sensor2 Sensors

3 Sensors4 Sensors









Cooperation


Dynamic Distributed Network MonitoringScope, Coverage, and Flexibility





Self-adaptationUsability, Responsiveness, and Operation



Cooperative Sensors: Improving Operation and Usability via Self-adaptivity

Example

On-demand Cooperation

Aims

Capture as much as possible.Avoid overload.Reduce # of sensors.

Apply cooperation as necessary.






Cooperative Sensors: Improving Operation and Usability via Self-adaptivity

Example

On-demand Cooperation

Aims

Capture as much as possible.Avoid overload.Reduce # of sensors.

Apply cooperation as necessary.






Example Results

0

100

200

300

400

500

0 10 20 30 40 50 60 70 80 90 100

[kpp

s]

Time [s]

Send 3Send 2Send 1Drp. 1Drp. 2Drp. 3Pkt. Rt.Recv.






Cooperation




Event-driven Architecture (EDA) and Complex Event Processing (CEP)Convergence of Data Sources, Data Processing, Flexibility, & Correlation



Self-adaptation




EDA & CEP for Network Monitoring















Cooperation






Domain Specific Language (DSL)Data Extraction & Transformation, Ease Usage, Dynamic Capabilities

Self-adaptation




Dynamic Data Extraction Configuration via Domain Specific Language

Listing 1: Extraction DSL Example[ { : o f f s e t 0 , : name : ts , : t r a n s f o r m a t i o n : t imestamp }{ : o f f s e t 12 , : name : l e n , : t r a n s f o r m a t i o n : i n t 3 2 }{ : o f f s e t : ipv4−dst , : name : ipDst , : t r a n s f o r m a t i o n : ipv4−a d d r e s s }{ : o f f s e t : udp−dst , : name : udpDst , : t r a n s f o r m a t i o n : i n t 1 6 } ]

Listing 2: Extraction Function based on DSL( f n [ ba o f f ]

( doto ( j a v a . u t i l . HashMap . )( . put ” t s ” ( t imestamp ba (+ o f f 0 ) ) )( . put ” l e n ” ( i n t 3 2 ba (+ o f f 1 2 ) ) )( . put ” i p D s t ” ( ipv4−a d d r e s s ba (+ o f f 4 6 ) ) )( . put ” udpDst ” ( i n t 1 6 ba (+ o f f 5 2 ) ) ) ) )






Data Extraction Performance

Table: Data Extraction Performance (Old Method)

Method Cap.Rt. [x ; sd(x)]

jNetPcap (Map) 265.7 kpps ; 10.4 kpps

Table: Comparison of DSL Extraction Performance

Method Cap.Rt. BS=1 [x ; sd(x)] Cap.Rt. Max. [x ; sd(x)] BSMax.

DSL 1 612.2 kpps ; 8.8 kpps 669.0 kpps ; 7.2 kpps 35DSL 2 726.4 kpps ; 9.1 kpps 798.5 kpps ; 5.6 kpps 50DSL 3 1114.8 kpps ; 46.4 kpps 1364.4 kpps ; 19.2 kpps 30DSL 4 1478.7 kpps ; 146.9 kpps 2053.6 kpps ; 24.5 kpps 90






Cooperation






Domain Specific Language (DSL)Data Extraction & Transformation, Ease Usage, Dynamic Capabilities

Self-adaptationUsability, Responsiveness, and Operation



Performance-based Self-adaptation, Feedback Loop Overview

Problem: Overload, Drops

Solution

Reduce “Columns”On-demand Adjustment

Aims

Avoid overload.“Maximize” processingfunctionality.

Adjust as necessary.






Example Results of Self-adaptive Performance-based Adjustment

0 200 400 600 800

1000 1200 1400

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 0 2 4 6 8 10 12 14

[kpp

s]

# of

Rul

es

Time [s]

Cap. Rt. [kpps]Pkt. Rt. [kpps]

Drop Rt. [kpps]Min. # Rules

# Act. Rules



Summary

Computer Networks: Critical Importance

Assuring Operating Networks → Information

Network Monitoring(Network Reconnaissance, Network Analysis and Surveillance)

“Good” Information → Challenging

(Contradicting) Requirements and Properties

Solutions?

Integration of Technologies: Use the best tools.Cooperation: PerformanceDynamic & Distributed: Flexibility, Scope, CoverageEDA & CEP: Convergence of Data Sources, Data Processing, Flexibility, & CorrelationDSL: Data Extraction & TransformationSelf-adaptation: Usability, Responsiveness, and Operation



Summary






Solutions!

Integration of Technologies: Use the best tools.

Cooperation: PerformanceDynamic & Distributed: Flexibility, Scope, CoverageEDA & CEP: Convergence of Data Sources, Data Processing, Flexibility, & CorrelationDSL: Data Extraction & TransformationSelf-adaptation: Usability, Responsiveness, and Operation



Summary






Solutions!

Integration of Technologies: Use the best tools.Cooperation: Performance

Dynamic & Distributed: Flexibility, Scope, CoverageEDA & CEP: Convergence of Data Sources, Data Processing, Flexibility, & CorrelationDSL: Data Extraction & TransformationSelf-adaptation: Usability, Responsiveness, and Operation



Summary






Solutions!

Integration of Technologies: Use the best tools.Cooperation: PerformanceDynamic & Distributed: Flexibility, Scope, Coverage

EDA & CEP: Convergence of Data Sources, Data Processing, Flexibility, & CorrelationDSL: Data Extraction & TransformationSelf-adaptation: Usability, Responsiveness, and Operation



Summary






Solutions!

Integration of Technologies: Use the best tools.Cooperation: PerformanceDynamic & Distributed: Flexibility, Scope, CoverageEDA & CEP: Convergence of Data Sources, Data Processing, Flexibility, & Correlation

DSL: Data Extraction & TransformationSelf-adaptation: Usability, Responsiveness, and Operation



Summary






Solutions!

Integration of Technologies: Use the best tools.Cooperation: PerformanceDynamic & Distributed: Flexibility, Scope, CoverageEDA & CEP: Convergence of Data Sources, Data Processing, Flexibility, & CorrelationDSL: Data Extraction & Transformation

Self-adaptation: Usability, Responsiveness, and Operation



Summary






Solutions!

Integration of Technologies: Use the best tools.Cooperation: PerformanceDynamic & Distributed: Flexibility, Scope, CoverageEDA & CEP: Convergence of Data Sources, Data Processing, Flexibility, & CorrelationDSL: Data Extraction & TransformationSelf-adaptation: Usability, Responsiveness, and Operation



End

Thank you for your attention!

Questions?

Rudiger Gad – http://ruedigergad.com

[email protected] – [email protected]



http://ruedigergad.com

End

Thank you for your attention!

Questions?

Rudiger Gad – http://ruedigergad.com

[email protected] – [email protected]



http://ruedigergad.com

Documents

IEEE IEMCON 2016 Keynote 0.4cm Improving Computer Network ... · IEEE IEMCON 2016 Keynote Improving Computer Network Monitoring, a brief Tour Ruediger Gad Terma GmbH, Space, Ground