ITG The Future of Trading. TM SAGE 2003 Monitoring Firewalls and Internet Usage with CRICKET

ITGThe Future of Trading. TM

SAGE 2003

Monitoring Firewalls and Internet Usage with CRICKET

2

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Agenda

MRTG Shortcomings Cricket IPFW Joining them up Examples More Examples

3

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

MRTG

For monitoring router interfaces by SNMP Run from cron every 5 minutes Generates HTML & GIF files PERL & some C Some capability for non-SNMP data sources by

specifying a program to run

4

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

… problems

Efficiency OK for small sites, less so for 100+ routers

Config complexity Interface renumbering in SNMP

Specificity “In”, “Out”, “Uptime” – or nothing!

Integer-only Scaling tricks for load average etc

(later versions of MRTG address these issues)

5

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

RRDTool

From the author of MRTG C library, PERL-callable Low-level tool for

Storing Summarizing Graphing

Time-series data Not a complete application

No data capture No control logic or configuration

6

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Cricket

Uses RRDTool Efficiency a prime requirement 3 parts –

A config tree A collection agent A CGI to generate & display graphs

About 7000 lines of Perl

7

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Cricket Config Tree

Tree structured, inherited defaults Implemented as a UNIX directory tree Entries in Dictionaries (Perl Hashes) Various dictionary types:

Single-line (OIDs etc) Multi-line (HTML boilerplate) Tag-Value (hash of hashes)

Compiled to a single DB file for speed

8

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Cricket Config Dictionaries

Cricket application uses 7 config dictionaries: Target TargetType DataSource Graph RRA OID HTML

9

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Data Sources

A method of fetching data URL-like specifier 4 built-in DSS types

SNMP: FILE: EXEC: FIELD:

Extensible by writing a simple (100 line) Perl module

10

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Instance Mapping

SNMP can re-number interfaces Need to map a constant identifier to dynamic

SNMP interface number Can specify an SNMP instance variable which

identifies the interface (e.g. ifAlias for CISCOs which is like “FastEthernet0/0”)

Cricket SNMP collector will walk the table to look up & cache the interface number

Will check and re-walk if necessary at each collection cycle

11

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

IPFW

Firewall uses IPFW on FreeBSDNumbered rules processed in order until accept/denyFor gateways, ruleset is processed twice!

IPFW Rules natd

InputI/F

OutputI/F

Routing Table

Host Applications

12

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

IPFW Rules

Action + byte/packet counts + match rules Match on:

Protocol (TCP, GRE, ICMP etc) Source & Dest IP/Net Source & Dest ports or port range (if appropriate) UID/GID for local packets Various protocol-specific flags (TCP SETUP) Source/Dest interface

Common actions: Accept, Deny, Divert, Count, Skipto

13

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Other IPFW Features

NAT via user-space NAT daemon & DIVERT rule - Slower due to extra context switches + Not limited by kernel KVM

Incoming connection forwarding Traffic Shaping Bandwidth limiting Packet Loss simulation Transparent Proxying

14

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Sample IPFW rules

02200 1283 82079 deny ip from any to 10.0.0.0/8 via fxp0



03000 6505264 1595692435 divert 8668 ip from any to any via fxp0

03100 6 280 deny ip from 10.0.0.0/8 to any via fxp0



15

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Joining Them Together

IPFW rules make a great tool for monitoring Very flexible packet matching COUNT rules don’t change Firewall operation Packet & Byte counts on each rule

How to make IPFW rules available to Cricket? SNMP?

SNMP not a secure protocol SNMP agents not security audited No access to rules

Rsh to run ipfw? (!!)

16

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

ipfwlogd

Daemon listens on socket on inside net Accepts connection & returns text report Never reads from socket (no buffer overruns) Small (<200 lines) and can be audited Can also run from inetd if your firewall has it enabled Sample output:

100 4156780 477197448

200 0 0

300 0 0

900 68654 48194803

1000 5865819 5797218169

1100 5086311 624784246

17

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Using ipfwlog with Cricket

Each 5 minutes, collect script is run Telnet to ipfwlog port & save report in a file (bunch of other data collection tasks) Run the Cricket collector Cricket targets use the FIELD: DSS to extract

counts for named rule number from the saved file

18

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Example – Chargeable Internet Use

Over-committed link, paying per byte Can’t use interface counters:

DMZ traffic “Free” traffic (ISP mail/FTP server) Non-IP traffic (spanning tree & management)

Add IPFW rules:

00900 skipto 1101 ip from 202.53.40.208/29 to202.53.40.208/29 via fxp0

01000 count ip from any to any in recv fxp0

01100 count ip from any to any out xmit fxp0

19

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Internet Usage – Defaults file

target --default--host = %auto-target-name%data-file = %dataDir%/%host%.ipfwin-rule = 1000out-rule = 1100target-type = ipfw-host

targettype ipfw-hostds = "total-in, total-out"

datasource total-in ds-source = "field:%data-file%:%in-rule%:3:1: "

datasource total-out ds-source = "field:%data-file%:%out-rule%:3:1: "

graph --default--y-axis = "Bytes per second"Units = "bytes/sec"

graph total-indraw-as = AREAlegend = "Bytes Received"

graph total-outlegend = "Bytes Transmitted"

20

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Internet Usage – Targets file

# NB: remember to edit ~/bin/run-cricket.sh if you# change/add hosts here!target melfw

short-desc = "Melbourne Firewall"target melproxy

short-desc = "Melbourne Web/FTP proxy"target proxy

short-desc = "OLD Melbourne Web/FTP proxy“collect = no

target fwshort-desc = "Old Firewall"in-rule = 48out-rule = 28

21

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Examples – VPN Usage

PPTP (Windows) VPN TCP for session setup, payload in GRE GRE well supported by ipfw

NAT knows enough to re-write GRE packets NAT even knows that GRE will follow TCP setup! Recognised for IPFW rules

ipfw add 1200 count gre from any to any in recv fxp0

ipfw add 1300 count gre from any to any out xmit fxp0

22

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

VPN Usage – cont

target --default--vpnin-rule = 1200vpnout-rule = 1300

datasource vpn-in ds-source = "field:%data-file%:%vpnin-rule%:3:1: "

datasource vpn-out ds-source = "field:%data-file%:%vpnout-rule%:3:1: "

targettype vpn-hostds = "total-in, total-out, vpn-in, vpn-out"view = "Total Traffic: total-in total-out, VPN Traffic: vpn-in vpn-out“

target melfwtarget-type = vpn-hostshort-desc = "Melbourne Firewall"

23

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Examples – Web Usage

Squid with SNMP agent (very thorough) Stand-alone proxy box (very easy) Or monitor using the IPFW rules

ipfw add 1400 tcp from ${oif} to any port 80,81,445

ipfw add 1500 tcp from any port 80,81,445 to ${oif}

(Note use of shell variables - ${oif}) Cricket config basically the same as previous

example

24

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Example – Port Scans

Just for kicks, count various port scans

add 1310 count tcp from any to any 1433 in recv ${oif}


add 1330 count udp from any to any 1434 in recv ${oif}

add 1340 count udp from any to any 137-139 in recv ${oif}


Minor change in cricket config:

datasource probes

ds-source = "field:%data-file%:%rule%:3:2: "

25

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Examples – VPN Link Usage

Count number of ngX interfaces with IP addressnetstat -in | sed -n '/^ng.*10\.132\.4\./p' | wc –l

And use an EXEC: DSS with GUAGE variabledatasource vpn-links

ds-source = "exec:0:/usr/local/etc/mpd/linkcount.sh"rrd-ds-type = GAUGE

targettype links-typeds = vpn-links

graph vpn-linksy-axis = "Users Active"units = "users“legend = "Users Active"show-max = 1show-ave-max = 1

target linkstarget-type = links-type

26

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Examples – Keeping Usage Stats

Graphs are great for trouble-shooting But verifying bills needs NUMBERS! Cricket can save data to a copy-to

SNMP: SQL: FILE: with a small patch from sourceforge

target --default--archive-file = %dataDir%/%auto-target-name%.archive

copy-to = file:%archive-file%

Summarise with some shell/perl scripts

27

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Examples – Ping Times

Consider SmokePing for complex cases For simple cases…. A script (this for FreeBSD)…

ping -c $1 -n -q $2 | \

awk -F/ '/round-trip/ { print a[split($4, a, " ")];

print $5; print $6}‘

Run this from the cron jobpinghosts="storm asahi adm-ns05 pooka"

for i in $pinghosts; do

$HOME/bin/doping.sh 5 $i > $basedir/pings/$i.ping &

done

28

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Examples - cont

target --default--

target-type = host-ping

ping-file = %dataDir%/%host%.ping

TargetType host-ping

ds = "pingMin, pingAve, pingMax"

datasource --default--

rrd-ds-type = GAUGE

datasource pingMin ds-source = "file:0:%ping-file%"

datasource pingAve ds-source = "file:1:%ping-file%"

datasource pingMax ds-source = "file:2:%ping-file%"

target Sydney

host = storm

29

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Examples - Mail Block Stats

target spam-refusetarget-type = spamcountshort-desc = "SPAM email connections dropped"

TargetType spamcountds = "spamcount"

datasource spamcountds-source = "exec:0:grep Rejected /var/log/maillog

| wc"graph spamcount y-axis = "Bounces per minute" units = "Bounces/min" legend = "SPAMs Bounced" bytes = 0 scale = 60,*

30

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Examples – System Monitoring

SNMP agent on the host Net-SNMP Vendor agent (e.g. Solaris)

SE Toolkit Scripting language for performance analysis “Orca” to graph data “VirtualAdrian” for advice Solaris only!

31

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Case Study

The original MRTG implementation paid for itselfLatency from Melbourne to HK

Our link had been rerouted via Tokyo

32

ITG

Th

e F

utu

re o

f T

rad

ing

. TM

Case Study – Again!

And the same thing happened again!Melbourne – Sydney linkRerouted via Brisbane!

Documents

ITG The Future of Trading. TM SAGE 2003 Monitoring Firewalls and Internet Usage with CRICKET