Happy easter everyone who happens to be celebrating that holiday. At least here in Jacksonville FL, its a beautiful warm day. If its cold and miserable were you are: what a great day to go hunting for easter eggs in your logs! This is prompted by a post from a reader who sent in the "sludge" as he called it: A quick summary of reports he typically sends to DShield. Did I mention lately that y'all should be sending your logs to DShield?

Anyway. He used some nice shell scripting to summarize his logs. I like this technique and frequently use it myself. You should all become familiar with the little gems hidden in your respective shells. Even Windows got something to offer in that department (search for past "Windows Shell Kung-Fu" posts by Ed for example).

In my opinion, the goal is to find "unusual" log entries. I am typically less interested in the thousands of SQL slammer infected hosts hitting me each day. And yes, they still exist :-(. What I am interested in are the "one off" hits.

Here a quick shell command I like to summarize my iptables firewall logs:

sed 's/.*SRC=//' < /var/log/firewall | sed 's/ .*DPT//' | sort | uniq -c | sort -n

Its not exactly pretty... but well, it works.

Browsing through the output, this one entry caught my attention:

1 192.228.79.200=27639 LEN=541

Its UDP (otherwise you would see more details about flags and such given the command sequence I used), it got payload, and it is sent to an odd port

Next lets look at the full line:

grep 27639 /var/log/firewall
Mar 23 18:17:20 defianta kernel: filter: cable-dmz IN=eth2 OUT=eth0
   SRC=192.228.79.200 DST=10.0.0.2 LEN=561 TOS=0x00 
   PREC=0x20 TTL=43 ID=0 DF 
   PROTO=UDP SPT=53 DPT=27639 LEN=541 

hm. from port 53. So maybe just a DNS response? But why did it get blocked? Now it is helpful to have full packet captures. I tend to capture each packet entering and leaving my network. Its not a huge amount of data (few gbytes per day), so I can keep about a month worth.

tcpdump -r snort.log.1206295201 -nn host 192.228.79.200
reading from file snort.log.1206295201, link-type EN10MB (Ethernet)
18:17:20.655144 IP 192.228.79.200.53 > 70.91.145.9.27639:  64286- 0/13/16 (533)

Interesting. The only thing I see is a DNS reply, with no answers, but plenty of authority records. Its also rather large. Unless edns0 is used, DNS replies should stick to 512 Bytes!

off I am into packet land to figure out what this was exactly about. More to come soon... it will involve an AAAA record (hehe... what a cliffhanger)

If you want to take a look, see http://isc.sans.org/odddns for the packet. (no... finding the packet was not staged. That's exactly how I got it.)

Johannes B. Ullrich, Ph.D.
jullrich at sans.org