title:

What info can you get from log files? How do you get it?

October 11, 2010

Examples

How many dynamic address clients do we have in a day?

How many address leases have been used? Are any of our clients taking more than their fair

share of addresses (i.e. one)? How many iPods are on cruznet? How many unique names logged into the

authentication system? What DNS servers are the kids using? Latvia?

ANSWER

Find the right log file and look through it.

I don't want to talk about graphing today, except to say that regardless of how I graph,

I will need to produce “time series” data.

DHCP transaction logs

But first we need to review the DHCP protocol...

DHCP protocol

D.O.R.K.And other stuff . . .

RA RA RA !!

If you believe DORK, this should be RK.

Clients renew (or try, anyway) by sending a request to the server that last gave them a lease. If they get an ack, then they get to keep that address. If they get a nak, the address is not available. Or the client might get no response at all.

Sample log file

Example: How many MAC addresses?

Make a list of all the ethernet addresses in the transaction log. Eliminate the duplicates. Count what is left. Pretty simple.

We need some tools . . .

Unix tool: grep

Grep applies a pattern to each line of a file. If the pattern exists in the line, the line is printed. If the pattern doesn't match, the line is skipped and the next line is considered.

Print can mean print or it can mean pass to the next program in a chain.

Unix tool: awk

Awk also operates on lines. It can reference the words within the line, testing them for a match, doing arithmetic on numbers, or just printing them.

Awk uses curly braces { } to surround its arguments.

Finding the MAC address

Oct 10 00:01:34 dhcpa dhcpd: [ID 702911 local7.info] DHCPREQUEST for 169.233.121.110 from d4:9a:20:a5:32:ab (iPod-touch-3) via 169.233.127.254

Oct 10 00:01:34 dhcpa dhcpd: [ID 702911 local7.info] DHCPREQUEST for 128.114.143.179 (128.114.142.212) from 00:13:72:bc:f5:81 (se81) via 128.114.143.240

In both cases, the address follows the word 'from'

Example w grep and awk

grep DHCPREQUEST dhcploga | awk '{if ($12 == "from") print $13; if ($13 == "from") print $14}'

Can also be written as:

grep DHCPREQUEST dhcploga | \ awk '{if ($12 == "from") print $13; \ if ($13 == "from") print $14}'

Output looks like:

7c:c5:37:72:49:cd7c:c5:37:a2:58:ee00:13:72:bd:c4:6d60:33:4b:1c:d4:f9c4:2c:03:8a:ea:d8d4:9a:20:1e:fc:8f70:f1:a1:ad:51:b500:13:02:b6:13:70

On Friday, there were 280K lines on dhcpa

Unix tool: uniq

Think unique

This tool compares lines in a file. When two or more lines are identical, it replaces them with a single line. But for this to be useful, all the addresses that are the same need to be adjacent in the file. Said another way, the file needs to be sorted.

Unix tool: sort

sort has lots of options. It can sort numerically or alphabetically. It can start sorting on a column other than the first word. It can reverse sort. Here, it doesn't make any difference because we only need all occurrences of the same MAC address to be on adjacent lines.

Putting this together, we get . . .

grep DHCPREQUEST dhcploga | \ awk '{if ($12 == "from") print $13; \ if ($13 == "from") print $14}' | \ sort | uniq | wc

wc counts the lines and words in the file. The number of lines is our answer: the number of different MAC addresses seen by the system.

Answers for last week

Friday 17391 Thursday 19451 Wednesday 19435 Tuesday 19668 Monday 19040

Anything wrong here?

There are two servers – dhcpa and dhcpb. I should have combined their logs.

DHCP assigns addresses to interfaces, not computers. One computer that is on wireless and wired at the same time will have two addresses.

A computer that is used on resnet (wired) in the dorm and cruznet (wireless) on campus will show us both MAC addresses, even if they're not used at the same time.

Another example: iPhone count

There is a hint how this might be done in the log file slide.

If we collect all the Client-hostnames, sort and run uniq we will not get the right answer.

Ethernet MAC addresses are unique. But is no reason to assume Client-hostnames are unique. Monicas-iPhone? Could be lots of Monicas.

How could we fix this?

Client-hostname

For MAC address we had:

grep DHCPREQUEST dhcploga | \awk '{if ($12 == "from") print $13; \ if ($13 == "from") print $14}'

Change this to:

grep DHCPREQUEST dhcploga | \awk '{if ($12 == "from") print $14, $13; \ if ($13 == "from") print $15, $14}'

This looks like . . .

(Austin1) 00:26:bb:0d:26:54(Austins-iPhone) cc:08:e0:1e:3b:43(Austins-phone) 00:26:08:7c:87:fa(Autumn-Saga) 00:25:bc:e1:5e:98(Avery) 00:26:4a:18:da:7a(Aviel-PC) 00:1b:24:f6:9d:0a(Aviv-PC) f0:7b:cb:0c:04:fb(AwesomeIV) f8:1e:df:d8:90:95(AwesomePossum) 00:16:d4:a5:af:6c(Awesomeness-PC) 00:23:4d:73:72:8c

Phone strategy

Select client names that contain phone with grep -i -i option ignores case so we get both phone and

Phone Retain the MAC addresses of the phones, and look

for unique ones as before, so

[stuff from previous page] | grep -i phone | \

awk '{print $2}' | sort | uniq | wc

Apple counts:

399 iPhones 552 iPods 42 iPads

How do we know this is right?

We can check the MAC vendor code to make sure it is Apple

But really, we don't know.

DHCP leases with awk

The DHCP server keeps a record of current leases in a file on the server disk. If the server needs to restart, the leases file tells it about leases already granted. There is one record for each managed IP address. Each address is in one of several states:

DHCP lease statesAn IP-address can be:

FREE – address is available for assignment BACKUP – address is free but on other server ACTIVE – unexpired lease held by client RELEASED – half way toward FREE ABANDONED – address conflict

? ABANDONED ?

An address is put into abandoned status when a client trys to use it and finds that it is already in use. ABANDONED is the same as FREE except that the address will not be offered if untainted free addresses are available.

A leases entry looks like:

Lease 169.233.58.234 { Starts 3 2010/10/06 02:29:38; Ends 3 2010/10/06 05:15:00; Tstp 3 2010/10/06 05:15:00; Tsfp 3 2010/10/06 17:32:08; Cltt 3 2010/10/6 02:29:28; Binding state released; Next binding state free; Hardware ethernet a4:ba:db:b3:5c:cc; Client-hostname “Catalina-PC”;}

Lease entry (con't)

There may be other lines in the file, depending on the client. For example, dhcp option 82 might give switch/port information.

There is a block like this for every dynamic address the server is managing.

Writing to the leases file

When an address changes state, a new entry for that address is written to the end of the leases file. For any address, the only entry that is current is the LAST one. The server never changes entries in the middle of the file.

Problems

Each address lease block is 10+ lines. Since grep acts on lines individually, it won't work to select whole blocks.

It would sure be nice if each address was one line. awk commands could get to be pretty long and hard

to type. Commands can be placed in a file. This file might be called an awk script. Then it becomes:

awk -f script.awk

Script to fix the multi-line problem{## Input is dhcp.leases. Output is one line per addr.#if ($1 == "lease") printf "\n%15s ",$2if ($1 == "starts") printf "%s %s ",$3, $4if ($1 == "ends") printf "%s %s ", $3, $4if ($1 == "binding") printf "%8s ", $3if ($1 == "hardware") printf "%s ",$3if ($1 == "uid") printf "%s ", $2if ($1 == "client-hostname") printf "%s ", $2}

This produces long lines like:

169.233.146.200 2010/10/11 02:32:42; 2010/10/11 14:32:42; active; 00:22:fa:df:6d:c8; "\001\000\"\372\337m\310"; "Kunal-PC";

169.233.144.244 2009/07/21 12:16:06; 2009/04/09 15:54:05; backup; 00:23:6c:05:8a:4f; "\001\000#l\005\212O";

The UID is weird binary. Time is GMT. Client-ID is optional for the client.

Usually, interest in one subnet

We'll use the 2-net as an example

awk -f script.awk dhcp.leases | \awk -F. '{if ($3 == 2) print}'

-F. means to use the period as the word separator instead of white space. The line begins with an address like 128.114.2.10 so the third word is the number 2.

For each IP, keep only the last line

# script name is lastline.awk{line[$1]= $0} END {for (item in line) print line[item]}

$0 is the whole line. $1 is the IP address.

Count of leases in use on 2-net

awk -f script.awk dhcp.leases | grep 128.114 \awk -F. '{if ($3 == 2) print}' | \awk -f lastline.awk | grep active | wc