Extracting gzipped or Unix script files from pcap data

During an incident response, it's often handy to be able to examine the actual attack traffic, and if you're using Sguil, you probably have it handy. One common situation is that an intruder has transferred files to or from your network, and you'd really like to see what's in them.

There's a great tool for extracting arbitrary files from pcap dumps, tcpxtract. Similar to the way hard drive forensic tools look through bytes on the disk to find the "magic headers" at the beginning of various types of files, tcpxtract combs through pcaps looking for file types it knows, regardless of the transport protocol used to ship them over the network. When it finds them, it writes them out to individual files for manual analysis.

Tcpxtract is a great tool for NSM practicioners, and should be in everyone's standard kit. There are a few common types of files that it doesn't support, but you can easily fix this by simply editing the tcpxtract.conf file to add support for new types if you know their magic numbers.

My friend geek00L has already blogged about adding Windows PE executable support. Now I'm here to tell you how to add support for gzipped files and Unix script files like "#!/some/file" ("#!/bin/sh" or "#!/usr/bin/perl" for example).

Just add the following two lines to the end of tcpxtract.conf:

gzip(1000000, \x1f\x8b\x08);
script(1000000, \x23\x21\x2f);

A little anti-climactic after all that buildup, wasn't it? I've had some advice that the script detection is likely to throw lots of false positives in an SSL session, so maybe you should keep it commented out until you know there are script files in the session that you need to find.

Canonicalizing funky IP addresses

I've been playing around with some phishing data recently, to see if I can correlate known phishing attempts with my outgoing network sessions in order to alert me when my users fall victim. I don't have any results to report for this yet, but I did have to do some research into canonicalizing IP addresses.

Most people don't realize this, but there are a few different IP address formats. We're using to seeing the dotted decimal quad form (e.g. "192.168.1.1") but there are others, and the phishers are using them. Thus, it behooves us to know how to work with them.

I was unable to find a comprehensive list of available formats online for some reason (if you know of one, please leave a comment!) so I examined my corpus of phishing URLs and found the following examples:

1. 192.168.1.1
   
2. 1127353292
   
3. 0x43320bcc
   
4. 192.0x43.9.3 (i.e., mixed decimal and hex quads)
   

Of course, phishers are also using normal hostnames as well, giving us at least 5 different types of possible inputs to deal with. Most of our tools use the decimal dotted quad format, so we need to normalize these formats into something useful.

Here's a Perl function that should do this for you. If you pass it any of the above formats (including the hostname), it will normalize the input and return a decimal dotted quad in string form, suitable for your favorite security tool. If it can't be converted (maybe it's a hostname that no longer resolves) the function returns undef instead.

I've wrapped this in a simple command-line tool and also incorporated it into other Perl scripts. I'm sure you'll find other uses for it as well. And hey, if you come across any other legal address formats, please let me know.

use Socket;

# Take any valid IP address or hostname and return the normalized IP address.
# Recognizes the following formats:
#     some.host.com
#     192.168.0.1
#     0x43320bcc
#     1127353292
#     192.0x1c.0x10.9  (ie, mixed decimal and hex quads)
# The function returns the string value of the IP address if successful, or
# undef if not.
#
# Warnings:  
#     1) This will return only a single address, so if the hostname resolves
#        to more than one address, you won't get them all.
#     2) It will generate a DNS query when looking up hostnames
sub normalize_funky_addrs {
    my($host) = @_;
    my($addr);

# If this is a hex address (e.g., "0x01234FAc") then first convert it to 
# decimal form
    if($host =~ m/^0x[a-fA-F0-9]+$/i) {
 $host = hex($host);
    }

# If the entire address wasn't hex, individual octets might be.  If so,
# find them and convert them.  Split the address into octets, check and 
# covert hex in each octect as necessary, then paste them all back together
# into one string and continue processing.  Yes, I could just return here,
# but I'd rather have only one function exit point.
    if($host =~ m/^([^.]+)\.([^.]+)\.([^.]+)\.([^.]+)$/) {
 ($octet1, $octet2, $octet3, $octet4) = ($1, $2, $3, $4);
 $octet1 = hex($octet1) if ($octet1 =~ m/0x/i);
 $octet2 = hex($octet2) if ($octet2 =~ m/0x/i);
 $octet3 = hex($octet3) if ($octet3 =~ m/0x/i);
 $octet4 = hex($octet4) if ($octet4 =~ m/0x/i);
 $host = "$octet1.$octet2.$octet3.$octet4";
    }

# Now we've either got a hostname, a normal IP address or an integer-form
# IP address.  We can work with any of those.
    $addr = gethostbyname($host);
    if($addr) {
 return inet_ntoa($addr);
    } else {
 return undef;
    }
}

Extracting email attachements from pcap files

From time to time, I need to examine email attachements that may have been delivered to my users, usually to see exactly what type of malware was included. Now, I could call them up and ask them if they got the message, but there are several reasons I don't like to do that. For example, I hate playing telephone tag, and I don't really want to encourage them to open the message for any reason.

Since I use Sguil, I have another option. I can extract the attachement from the network data directly. Here's how.

1. Capture the session of interest as a pcap file. If you're also using Sguil, you can probably just do a SANCP search to find the network session that contained the message as it was delivered to your SMTP server. Open it up in Ethereal, which causes the Sguil client to copy the pcap file to your analysis workstation. Close Ethereal now, because you won't be using it for this.
   
2. Once you have the pcap file, use the tcpflow tool to extract an ASCII version of the conversation. This will create two files, one for the server side of the session, and one for the client side. Here's an example, where xxx.xxx.xxx.xxx is the sending host, and zzz.zzz.zzz.zzz is your mail server:
   

   
   % tcpflow -r xxx.xxx.xxx.xxx_yyyy_zzz.zzz.zzz.zzz_25-6.raw
   % ls
   xxx.xxx.xxx.xxx_yyyy_zzz.zzz.zzz.zzz_25-6.raw
   xxx.xxx.xxx.xxx.yyyy-zzz.zzz.zzz.zzz.00025
   zzz.zzz.zzz.zzz.0025-xxx.xxx.xxx.xxx.yyyy
   

   
   The file you need to concern yourself with now is the one that ends in .00025. That's the one that contains the email message sent by the client.
   
3. Edit the *.00025 file in your favorite text editor. As it is now, it's a complete record of all the SMTP protocol events, and what you really want is just the data. The easiest thing is just to delete everything up to (but not including) the first line that starts with "Received:".
   
4. Use the following perl command to read the mail file in via stdin and dump out the decoded MIME message. The script uses the MIME::Parser module (available from CPAN) to handle MIME decoding.
   

   
   %  perl -e 'use MIME::Parser; \
   $parser = new MIME::Parser; \
   $parser->output_under("/var/tmp"); \
   $entity = $parser->parse(\*STDIN); \
   $entity->dump_skeleton;' < *25
   
   Content-type: multipart/mixed
   Effective-type: multipart/mixed
   Body-file: NONE
   Subject: Message could not be delivered
   Num-parts: 2
   --
       Content-type: text/plain
       Effective-type: text/plain
       Body-file: /var/tmp/msg-1150990039-2841-0/msg-2841-1.txt
       --
       Content-type: application/octet-stream
       Effective-type: application/octet-stream
       Body-file: /var/tmp/msg-1150990039-2841-0/letter.zip
       Recommended-filename: letter.zip
       --
   

   
   As you can see, this command creates a directory /var/tmp/msg-XXXXXXX-XXXX-0/ which contains files for each piece of the MIME multipart message, including the attachement (in this case, /var/tmp/msg-1150990039-2841-0/letter.zip).
   

Now that you've got the attachement, you can use your favorite reverse engineering tools on it to figure out exactly what you've got on your hands.

Laptop encryption? I have a better idea.

I just read an AP article from last week, entitled Encryption can save data in laptop lapses. We hear about this problem in the news all the time now: someone loses a laptop with personally identifiable information about thousands of {customers|taxpayers|veterans|others}.

This is a real problem, make no mistake, but articles like this one are starting to piss me off. Instead of talking about encryption, let me propose a much more revolutionary idea: don't put the freakin' data on the laptop in the first place!

Laptops are designed to be mobile, and are easily lost, stolen or broken. Good encryption practices are rare (hint: it's not just a matter of software, but also of implementation, policy and user acceptance). Instead of worrying about who's laptop is going to be stolen next, let's try to address the real issue: does that data really need to be stored on the laptop in the first place?

If mobile users need access to data in the field, make them VPN back to the corporate network and work on it there. And don't tell me things like, "But we need to work on the plane" because you don't. You may want to work at 26,000 feet (it's better than watching Firewall on the tiny screen), but my privacy rights outweigh your productivity crisis anytime.

Let me end this rant by saying that if your company insists on placing your customers' personal information on laptops or other mobile devices, encrypted or not, I hope they hammer you in court.

Update 2006-06-09: Finally, some mainstream coverage for this apparently radical idea! In a story entitled Why Do Laptops Schlep Such Data?, the unnamed AP author raises the question about whether carrying such data around on mobile devices is appropriate. Short answer: "No, but some people will inevitably find a way to do it anyway." No doubt this is true, and that's why routine encryption of mobile data remains important. Just don't be confused about the proper use of mobile encryption: It's not the first line of defense, it's the last. It ideally only comes into play once someone has violated corporate policy by copying data onto the device.

Unfortunately, the article does miss (badly!) on an important point. The second-to-last paragraph describes software that "shreds" data on the hard drive if the laptop is stolen. The idea behind this type of so-called protection is absurd. The laptop's OS and security software has to be running in order for this to be effective, but any competent computer thief can swap out a hard drive and copy the data with only a few moments' work.

Tracking your most active network protocols with Sguil

All you Sguil users out there might find this interesting. I wanted to figure out a way to track the top most active protocols flowing over my perimeter, where "active" in this case means "most bytes transferred either in or out of my network". This is actually a fairly simple database query, but I had a special requirement.

You see, I wanted to know not only the total number of bytes transferred, but I wanted to see it broken down by whether the traffic was entering my network or leaving my network. SANCP (Sguil's session collector) tracks bytes by source and destination, but that's not good enough. The source of one connection could be an Internet host, while the source of another connection could be a system on my own LAN. Just because SANCP records how many bytes the source host sent doesn't mean it knows whether they were entering or leaving my LAN.

Here's a SQL query that really can tell the difference:

select dst_port, 
       sum(to_mynet) as bytes_to_mynet, 
       sum(from_mynet) as bytes_from_mynet,
       sum(to_mynet + from_mynet) as total_bytes
       from
       (
          select dst_port, 
          src_bytes as from_mynet,
   dst_bytes as to_mynet
   from sancp where
   (start_time between DATE_SUB(CURDATE(), INTERVAL 1 HOUR) 
                and CURDATE())
   and (src_ip between INET_ATON("192.168.0.0") and
        INET_ATON("192.168.255.255")) and
   not (dst_ip between INET_ATON("192.168.0.0") and
        INET_ATON("192.168.255.255"))
          UNION ALL
          select dst_port, 
   dst_bytes as from_mynet,
          src_bytes as to_mynet
   from sancp where
   (start_time between DATE_SUB(CURDATE(), INTERVAL 1 HOUR) 
                and CURDATE())
   and not (src_ip between INET_ATON("192.168.0.0") and
        INET_ATON("192.168.255.255")) and
       (dst_ip between INET_ATON("192.168.0.0") and
        INET_ATON("192.168.255.255"))
       ) as test_table
       group by dst_port
       order by total_bytes desc
       limit 5;

The trick here is to do two queries, each limiting itself to either sources on the Internet or sources on my LAN. I can then use the "SELECT" statements to order the columns to reflect the direction of the data flow. In this query, the second column will always be the number of bytes flowing out of my network, and the third column will always be the number of bytes coming into my network. Of course, the first column is the destination port number, which corresponds roughly to the network protocol used.

These two queries are joined by a "UNION ALL" statement to make them into one large result set. The SQL code above treats this as a temporary table, from which the outer SELECT statement grabs it's data and does the final work of summing both incoming and outgoing data flows and creating a neat table.

The output of this query will look something like the following table. Note that I ran it against some very odd sample data, so it doesn't show a full 10 rows. Also, I messed with the numbers because I was uneasy about posting my real data here. Still, you can see what the report looks like.

+----------+---------------+-----------------+-------------+
| dst_port | bytes_to_mynet| bytes_from_mynet| total_bytes |
+----------+---------------+-----------------+-------------+
|       80 | 1816          | 51101804400     | 51101806216 |
|       22 | 1816          | 51101412328     | 51101414144 |
|      443 | 1816          | 51096413496     | 51096415312 |
|       53 | 1804400       | 18093887        | 18274327    |
+----------+---------------+-----------------+-------------+

To use this in your own network, simply change the IP addresses above to represent the beginning and ending of your own address space. As written, the query will track only a single hour of data (from 23:00 to 00:00 of the current day), but you can play with the time specification to get other reporting periods.

HTTP PUT Defacement Attempts

I've seen the number of website defacement attempts on my network rise by about 3 orders of magnitude since yesterday, as evidenced by this report:

mysql> select date(timestamp) as dt, count(*) from event
where timestamp > "2006-05-30" and signature = "HTTP PUT
Defacement Attempt" group by dt order by dt ASC;
+------------+----------+
| dt         | count(*) |
+------------+----------+
| 2006-05-30 |        3 |
| 2006-05-31 |     2006 |
| 2006-06-01 |     1301 |
+------------+----------+
3 rows in set (0.55 sec)

The count for 2006-05-30 is pretty typical (an average day sees less than 10 defacement attempts). Other analysts I've talked with don't seem to be noticing anything unusual. Is it just me?

The reason for this, it turns out, is that I've forgotten to contribute the Snort rule I wrote to detect these attacks. So here it is, in case you're interested in tracking these yourself:

alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS 
(msg:"HTTP PUT Defacement Attempt"; 
flow:to_server,established; content:"PUT "; depth:4; 
classtype:web-application-attack; sid:1000003; rev:1;)

The attack itself is quite simple. The attackers simple use a built-in HTTP method designed to allow an authorized user to upload a file directly into the web space. It's not even really an exploit, since they're using the PUT method the way it was designed to be used. They're just counting on a misconfiguration that allows anyone to do it.

Here's an example:

PUT /index.html HTTP/1.0
Accept-Language: en-us;q=0.5
Translate: f
Content-Length:153
User-Agent: Microsoft Data Access Internet Publishing Provider DAV 1.1
Host: education.jlab.org

spykids spykids spykids spykids spykids spykids spykids
spykids spykids spykids spykids spykids spykids spykids 
spykids spykids spykids spykids spykid\n\n

As you can see, they're trying to replace the site's default page ("/index.html") with a page consisting entirely of their own text ("spykids" repeated several times).

There seem to be a constant low-level of these attacks on the Internet, and if your servers are configured correctly, you have nothing to worry about. Still, if you're like me, you still want to track these defacement attempts. Using my rule, now you can. I'll be submitting it to the Sourcefire Community Ruleset, so hopefully it'll show up there soon.

Update 2006-06-01 14:37: I'm not the only one who has noticed this anymore. Apparently this group has been defacing a lot of sites lately. Fortunately, I didn't notice it in the quite the way this guy did. I guess they've decided to crank up the defacement machine recently.

InfoSeCon 2006 Report

Earlier this month, I was very fortunate to be an invited speaker at InfoSeCon 2006. This was my second year speaking there, and there are two things you should know about this conference:

1. It's a small conference, because the organizers are have created an atmosphere where the speakers and attendees mingle freely, discuss issues over dinner and drinks, and actually talk rather than just attend presentations. They've built a community, and this works extraordinarily well for both speakers and attendees.
   
2. The location is spectacular. Dubrovnik is a UNESCO World Heritage Site for a reason, and it's no joke when they call it the "Pearl of the Adriatic." In fact, the location is so spectacular that the biggest danger is that your boss will want to go instead. Fight this, or you'll miss the spectacular evening events that make this very much resemble a working vacation.
   

The conference as a mixture of combined sessions and individual "Management" and "Technical" tracks. The speaker list was headed this year by two big names in the industry: Eugene Kaspersky (of Anti-Virus fame) and Marcus Ranum (of "I invented the firewall and we've pretty much all screwed it up since" fame). Mr. Kaspersky's talks were a little light on the technical content, I thought, having been written by his marketing department. Still, it was interesting to hear him speak about the relationship between malware and organized crime. Nothing that isn't already common knowledge, but he's on the cutting-edge of this fight, so just hearing his thoughts was instructive. He also spoke about the organization of his company's malware lab, but I was fairly disappointed with this, as it lacked substantive detail and mostly just emphasized the fact that they do have such a lab.

Marcus Ranum's talks were much more interesting. In fact, he was something of a lightning rod of controversy, as I understand him to be elsewhere, too. He gave a pair of talks, about the state of the security industry and about the evolution of the firewall. The overarching theme, which became a sort of conference catchphrase, was that "we're doing security wrong." Not that anyone has a comprehensive solution yet, but if the first step is admitting that we have a problem, then I think he's done his job. I don't always agree with Marcus, but after hearing him speak and spending some time with him informally, I think his points are valid.

By the way, remember when I said that the organizers try hard to create opportunities to mingle with the speakers? It works. I really appreciated the opportunity to sit down and have several conversations with Marcus. I also got to spend rather a lot of time with some friends, both new and old, in the Croatian, Slovenian and Slovakian IT industry. We prowled the midieval streets of Cavtat looking for vampire photos and listened to a restaurant full of boisterous Croatian tourists singing along with an accordian. We sailed the Adriatic and scaled the Dubrovnik city walls. We even learned a few things about the discipline of Information Security. All in all, a valuable and enjoyable trip, and I can't wait for InfoSeCon 2007!

Scan detection via network session records

I needed a quick, easy way to detect scanners on my LAN. Since I'm running Sguil, I have plenty of data about network sessions in a convenient MySQL database. I thought, why not use that?

My premise (cribbed from Snort's sfPortscan preprocessor) is that network scanning activity stands out due to it's unusually high number of failed connection attempts. In other words, most of the time the services being probed are not available, and this makes the scanner easier to spot. Here is a simple perl script I wrote that implements this type of scan detection by querying Sguil's SANCP session database.

To use it, you'll need to edit the script to modify the $HOME_NET variable. It's a snippet of SQL code, but it's very simple and documented in the comments. I'm looking for scanning activity on my own LAN, so the default is to set it to search only for activity with two local endpoints. If you are reasonably fluent in SQL, though, you can customize this to find other types of scans. There are a few other variables you can tweak (read the comments) but nothing terribly critical.

The script identifies two types of scanning activity. Portsweepers are systems that try a few different ports across a variety of addresses. For example, malware that looks around trying to find open web servers would fall into this category. On the other hand, portscanners are systems that try a lot of ports, usually on a relatively few systems. Attackers that are trying to enumerate services on a subnet would be a good example.

I haven't really tested this anywhere but a RHEL/CentOS 4 system. If you get it working on some other platform, or if you have any other comments/suggestions/improvements, please post them here.

The value of privacy

In his latest editorial, The Eternal Value of Privacy, Bruce Schneier provides a compelling answer to the question, "If you aren't doing anything wrong, what do you have to hide?"

Schneier's argument is slanted more towards the national security issues of domestic wiretaps, video surveillance and the like, but his points are also valuable from a cybersecurity perspective.

Many of us are tasked with monitoring for abuses of and intrusions into our computing infrastructure. This means collecting and analyzing data, much of which may be considered personal or private. IDS analysts especially may come into contact with the contents of personal communications such as emails, instant messages or even VOIP calls. We may see web URLs, chat room logs or forum postings detailing medical conditions, alternate lifestyles, or even employment concerns. It is our duty (and in many cases, our legal obligation) to protect this information against disclosure, even to others within our own organizations, unless there is a clear security concern which must be acted upon. Even then, discretion and need-to-know are critical.

If you're reading my blog, you're in a group of people who are likely to tackle thorny privacy issues on a regular basis. I highly recommend taking the time to read Schneier's short editorial and think about how this applies to your own situation.

A traffic-analysis approach to detecting DNS tunnels

I had the chance to see a very interesting presentation last week about DNS tunnels. I understand the concepts well enough, but I had never run into one "in the wild." I realized, however, that I probably wouldn't have noticed one in the first place, so I decided to try to do a little digging to see if I could come up with a detection mechanism.

Fortunately, I'm using Sguil to collect all my Network Security Monitoring (NSM) data. As you may know, Sguil collects (at least) three types of data:

1. Snort IDS alerts
   
2. Network session data (from SANCP)
   
3. Full content packet captures (from a 2nd instance of Snort)
   

The packet captures are not typically used for alerting purposes, leaving me with the possibility of using either an IDS signature based approach, or something using traffic analysis on the network sessions.

Some DNS tunnel detection signatures already exist (The Sourcefire community ruleset already has signatures to detect the NSTX tunnel software), but I think this approach is doomed to failure from the start. A signature-based approach would be good for detecting specific instances of DNS tunnelling software, but for real protection, a more general detection capability is required. I chose to try the traffic analysis approach instead.

I started with the assumption that there were basically three uses for DNS tunnels, to wit:

1. Data exfiltration (sending data out of the LAN)
   
2. Data infiltration (downloads to the local LAN)
   
3. Two-way interactive traffic
   

Of course, the fourth possibility is that a tunnel could be used for some combination of the above, but I left that out of the scope for now. I also ignored the possibility of two-way traffic for now, since it's harder and I'm just starting to look into this. My main goal was to identify data exfiltration by looking at "lopsided" transfers. By extension, the same technique could also be used to discover data infiltration, as I'll show.

Once I identified the types of tunnels I wanted to detect, I tried to deduce what their traffic profiles would look like. Specifically, I assumed the following:

1. At least one side of the session would be associated with port 53.
   
2. The tunnel could use either TCP or UDP (though UDP is the most likely), but since SANCP is able to construct psuedo-sessions from sessionless UDP traffic, I could effectively ignore the difference between TCP and UDP sessions.
   
3. The upload/download ratio would be lopsided. That is, the tunnel client would either be sending more data than they recieve (exfiltration) or vice versa (infiltration).
   

If you're not familiar with Sguil, just know that all the network session data is stored in a MySQL database. Although you normally access this data through the Sguil GUI, it's also easy to connect with the normal MySQL client application and send arbitrary SQL queries. I
started with this one:

 select start_time, end_time, INET_NTOA(src_ip), src_port,
 INET_NTOA(dst_ip), dst_port, ip_proto, (src_bytes / dst_bytes) as
 ratio from sancp where dst_port =  53 and start_time between
 DATE_SUB(CURDATE(), INTERVAL 7 DAY) and CURDATE()
 having ratio >= 2 order by ratio DESC LIMIT 25;  

The intent of this query is to identify at most 25 DNS sessions during the last week where the initiator (the client) sends at least twice as much data as it receives in response. Sounds good, but it is actually a bit naive. It turns out that many small transactions fit this profile, especially if the DNS server doesn't respond (the response size is 0 bytes).

I fixed this problem by also looking for a minimum number of bytes transferred (in either direction). After all, the purpose of a DNS tunnel is to transfer data, so if there are only a few bytes, the odds are very, very good that it's not a tunnel. In this version, I've set up the query also to check that the total number of source and
destination bytes is at least 50,000:

 select start_time, end_time, INET_NTOA(src_ip), src_port,
 INET_NTOA(dst_ip), dst_port, ip_proto, (src_bytes + dst_bytes) as
 total, (src_bytes / dst_bytes) as ratio from sancp where dst_port =
 53 and start_time between DATE_SUB(CURDATE(), INTERVAL 7 DAY) and
 CURDATE() having ratio >= 2 and total > 50000 order by total DESC,
 ratio DESC LIMIT 25;

This brings the number of false positives down quite a bit. If this still isn't good enough, you can tune both the ratio and the total bytes to give you whatever level of sensitivity you feel comfortable with. Increasing either number should decrease false positives, but at the expense of creating more false negatives (ie, you might miss some tunnels).

By the way, I mentioned before that you could use the same approach to detect either exfiltration or infiltration. The queries above are written for exfiltration, so if you would like to look for infiltration, simply change the definition of the "ratio" parameter from this:

(src_bytes / dst_bytes) as ratio

to this:

(dst_bytes / src_bytes) as ratio

That'll look for data going in the other direction. The rest of the query is identical for either case.

As I mentioned, I've never encountered a DNS tunnel outside of a lab environment (yet), so I can't say how well my approach holds up in the real world. If anyone with more experience would like to comment, I'm all ears. Similarly, if anyone decides to try out my method, I'd love to hear how it works out for you.

Update 2006-05-04 14:33: My fellow #snort-gui hanger-on and all-around smart guy tranzorp has posted additional analysis of my ideas on his blog. Specifically, he points to two easy ways to evade this simple type of analysis, and I recommend that you read his post for yourself, because he's got a lot of experience with this topic.

If you just want a quick 'bang-it-out-in-an-emergency' check for tunnels, the following updated query isn't bad. It's the same as above, but corrects for the fact that the DNS reply contains a complete copy of the DNS query, as tranzorp pointed out:

 select start_time, end_time, INET_NTOA(src_ip), src_port,
 INET_NTOA(dst_ip), dst_port, ip_proto, 
(src_bytes + (dst_bytes - src_bytes)) as  total,
(src_bytes / (dst_bytes - src_bytes)) as ratio 
from sancp where dst_port =  53 and 
start_time between DATE_SUB(CURDATE(), INTERVAL 7 DAY) and
 CURDATE() having ratio >= 2 and total > 50000 order by total DESC,
 ratio DESC LIMIT 25;

Based on feedback I've received from tranzorp and others, I'm looking at a more statistical approach to the problem. In short, I'm experimenting with computing a psuedo-bandwidth for each session (bytes sent / session duration), then looking for abnormally high or low bandwidth sessions. I've got a prototype, but it's in perl so it's horribly inefficient for the mountain of DNS session data I collect. I'll post something about it when I have had a chance to tweak on it some more.

Unintentionally acknowledging the truth

Here's a funny typo in a recent article about Verizon upgrading the bandwidth on it's fiber-to-the-home service:

The 30-Mbit premium tier was left untouched, in bot performance and price.

Too true, I'm afraid. Too true.

New HRSUG Website Launched

I'm pleased to announce the grand opening of the new HRSUG website at hrsug.org. The site will carry the latest information about HRSUG meetings and happenings, and separates the information from my personal blog (this one).

I also hope to publish network security-related notes or articles contributed by HRSUG members. If you want blog posting privileges, let me know and I can add you to the list.

Let's keep this announcement short and sweet. Remember, HRSUG is for the members, so let me know if you have ideas for improving the site.

If you read this far, it may also interest you to know that the May HRSUG meeting is coming up this week.

EIDE/SATA USB adapter cable, good for forensics?

This product looks like it could really be nice for acquiring images of suspect hard drives.

One interesting feature is that you can connect both the EIDE and the SATA drive at the same time. I guess it could be a bit slow trying to acquire an image when both the source and destination drives are on the same USB port, but on the other hand it's a lot more portable than a full tower rig, so things might balance out.

The obvious problem is the lack of an integrated write-blocker, but I guess you could use your own along with this device.

Anyone tried one yet?

April HRSUG Meeting Reminder

Hi, all. I just want to remind everyone that the April meeting of the Hampton Roads Snort Users Group (HRSUG) is coming up:

Date:  Thursday, 6 April 2006
Time:  7:00PM
Place: Williamsburg Regional Library
       515 Scotland Street
       Williamsburg, VA
       (757) 259-4040

The topic for the meeting will be "EZ Snort Rules", an introduction to writing basic IDS rules in everyone's favorite pig-themed IDS. See you there!

Update 04/06/06 22:30: I've posted the slides for my presentation on the Vorant downloads page. The full title of the talk is "EZ Snort Rules: Find the Truffles, Leave the Dirt".

MySQL 4.1.x authentication internals

While reviewing the deployment plans for a new MySQL server yesterday, I started wondering about the security of the authentication protocol it uses. Most of the users of the new database would be accessing it via the LAN, without the benefit of SSL encryption. It's well known that the MySQL 3.x authentication protocol is vulnerable to sniffing attacks, so MySQL 4.1 introduced a new authentication scheme designed to be more secure even over a plaintext session. But was it good enough?

I first spent some time to understand exactly how the protocol works. Thanks to some kind folks in the #mysql channel on irc.freenode.net, I located the MySQL internals documentation page the describes the authentication protocol. Unfortunately, this page doesn't really describe what's going on in the 4.1 protocol, so I turned to the source code for the definitive scoop.

Just so no one else has to do this again, here's a better description of how the protocol works, based on the 5.0.19 release. Look in libmysql/password.c at the scramble() and check_scramble() functions if you want to check the source code for yourself.

The first thing to know is that the MySQL server stores password information in the "Password" field of the "mysql.user" table. This isn't actually the plaintext password; it's a SHA1 hash of a SHA1 hash of the password, like this: SHA1(SHA1(password)).

When a client tries to connect, the server will first generate a random string (the "seed" or "salt", depending on which document you're reading) and send it to the client. More on this later.

Now the client must ask for the user's password and use it to compute two hashes. The first hash is a simple hash of the user's password: SHA1(password). This is referred to in the code as the "stage1_hash". The client re-hashes stage1 to obtain the "stage2_hash": SHA1(stage1_hash). Note that the stage2 hash is the same as the hash the server stored in the database.

Next, the client hashes the combination of the seed and the stage2 hash like so: SHA1(seed + stage2_hash). This temporary result is then XORed with the stage1 hash and transmitted to the server.

Now the server has all the information it needs to know whether the user supplied the correct password. Remember that the server stores the stage2_hash in the database, and it originally created the seed, so it knows both of those. What it wants to recover is the stage1_hash. Fortunately, the data transmitted from the client is simply the seed and the stage2 hash, XORed with the stage1 hash. To recover the stage1 hash, the server simply has to recompute the hash SHA1(seed + stage2_hash), then XOR it with the data provided by the client. The result will be the stage1 hash. Since the stage2_hash is simply SHA1(stage1_hash), the server can simply compute a new stage2_hash and compare it to the hash in the database. If the two hashes match, the user provided the proper password.

Ok, I admit that was probably a little confusing, but here's where I think it gets really interesting. There is a flaw in this protocol. The security depends on keeping two pieces of information a secret from attackers: the stage1 and stage2 hashes. If the attacker knows both hashes, they can construct a login request that will be accepted by the server.

Stage2 hashes are protected by keeping the database files away from prying eyes, but if an attacker were to steal the database backup tapes, for example, the hashes for all database accounts would be in his possession.

Still, just having the stage2 hashes is not enough. The final piece of the client authentication credential is XORed with the stage1 hash, yet the server doesn't know the stage1 hash. Therefore, the protocol provides it with all the information it needs to know in order to derive the stage1 hash. Unfortunately, that means that the same information could be derived by anyone who could sniff a valid authentication transaction off the wire and who also possessed a copy of the stage2 hash.

So the weakness is that if an attacker has the stage2_hash stored in the db, and can observe a single successful authentication transaction, they can recover the stage1_hash for that account. Using both hashes, they can then create their own successful login request, even without knowing the user's actual password.

I've confirmed this with the MySQL team, who say that this is a known issue. It is difficult to exploit, I think, since you need a fair amount of information in order to make the attack successful. Frankly, if you already have the password hashes, you probably also already have the rest of the info in the database as well, so this might be overkill. Still, I had fun working through this last night, so I thought others might enjoy it as well. At the least, maybe I'll save someone else the time I spent understanding how the protocol works.

If this vulnerability really concerns you, you can protect the stage1 hash information by using MySQL's built-in SSL session encryption support. You should probably also be encrypting your backup tapes to protect the stage2 hash and all the data.

Detecting common botnets with Snort

I've been reading up on the mechanics of how popular bot software actually works, with an eye towards detecting it on the wire. I've been using the BLEEDING-EDGE ATTACK RESPONSE IRC - Nick change on non-std port and its cousins, which attempt to detect botnets and other "covert" channels that think they're clever by sending IRC protocol traffic over ports that are not normally associated with IRC. These work well, have two problems:

1. Some IM programs (like ICQ and MSN Messenger) use the IRC protocol on various ports and thus trigger this rule
   
2. The rule doesn't detect botnet traffic if they happen to use the standard IRC port (6667)
   

One of the papers I've been reading recently was this great overview of four popular bot packages, An Inside Look at Botnets by Paul Barford and Vinod Yegneswaran. This isn't groundbreaking new research by any means, but they have started to put together some basic practical information about how these things work on the wire.

Having read that paper, I decided it would be fun to write a set of Snort rules to detect traffic generated by the four bots they mention. That would at least address the part of the problem #2, so could be a very worthwhile effort as well. Here are the rules, suitable for inclusion in your own local.rules file.

For those who are curious about what these do, there are three functions. The first rule (sid 9000075) attempts to define IRC protocol traffic, no matter what port the server is using. It sets the "is_proto_irc" flowbit on the session, so the later rules can depend on this to be set and won't do a lot of work inspecting non-IRC packets. This rule will never generate any alerts; it's only there to make the other rules in this file more efficient.

The second rule (sid 9000076) isn't necessarily related directly to botnets. It looks for IRC servers that seem to be on your local network and are communicating with outside hosts. If you actually do run a legitimate server, modify or comment this out as appropriate.

All the rest of the rules look for specific commands used by AgoBot/PhatBot, SDBot, SpyBot and GTBot. The GTBot commands are slightly more complex. All the others just use plain words (like "portscan ") for commands, but GTBot uses a command character to distinguish commands from other data (like "!portscan "). Since it seemed to me that the command character was likely to change from botherder to botherder, I coded in a regular expression that tried to allow a variety of possible cmdchars. You'll see those in the rules.

Anyway, feel free to use these rules if you like. I won't guarantee they work well yet, since I haven't been running them long, but so far so good. They probably won't crash your snort process, but that's about all I can say about them. I will give you one note of caution: If you find an active botnet, you may generate a lot of alerts. Consider using thresholding to avoid overwhelming your IDS analyst.

If you actually detect a botnet through these rules, please let me know. As far as I know, I have no active botnets right now, so I've only been able to test them against contrived traffic and not live data. I'd like to know if they do or do not work in the wild.

Update 4/4/2006: These bot rules are now included as part of the snort.org community ruleset. Thanks to Sourcefire's Alex Kirk, who made the rules better by pointing out that I forgot to add the "nocase" directive to make the rules case-insensitive. If you're using my rules, I recommend that you remove them and have a look at the community rules instead.

I'm finally caving in...

...and sitting the CISSP exam. I mentioned this to a few people today and got one of two reactions:

1. They immediately lost all respect for me, or
   
2. Ok, I lied. There was only one reaction.
   

I admit this was based on a rather limited sample, but wow. People really seem to hate the CISSP, all it stands for, it's dog and the horse it rode in on.

I've read some of Richard Bejtlich's thoughts on the matter (which made quite the splash when they were originally published). I agree with him that the ISC(2) code of ethics is pretty much all the CISSP has going for it, but I don't think that explains the vituperousness of the reactions I see whenever CISSP is mentioned. Nor do I necessarily think it's jealousy, because most of the people I get this from could easily pass the exam (if they were willing to pony up the $500 fee).

It seems to me as though I've detected a bit of a pattern, though: the more comfortable a person is working with technical matters, the less likely they are to respect the CISSP. Is it just our geeky tendency to look down upon those things which can be done by any suit and tie lackey? Is the perception that
CISSP == management == PHB? I don't think it's really that simple, but maybe us geeks just need someone to love to hate and Bill Gates wasn't available today.

If anyone has any theories they'd like to share, or if you hate people who hold CISSPs and would be kind enough to let me know why, please leave a comment.

March HRSUG meeting

Hi, all. I just want to remind everyone that the March meeting of the Hampton Roads Snort Users Group (HRSUG) is coming up:

Date:  Wednesday, 8 March 2006
Time:  7:00PM
Place: Williamsburg Regional Library
       515 Scotland Street
       Williamsburg, VA
       (757) 259-4040

As of this moment, there is no topic assigned to the meeting. I'll work on that, but if anyone has a suggestion or would like to volunteer to talk about something, please let me know.

Weird Cisco attacks happening all over the world

On February 4th, I started seeing small numbers of the following Snort alert:

Cisco IOS HTTP configuration attempt

Here's a breakdown of the number of alerts per day for the first couple of weeks of the month:

| 2006-02-04 |        6 |
| 2006-02-05 |        4 |
| 2006-02-13 |        2 |
| 2006-02-14 |        1 |
| 2006-02-15 |       30 |
| 2006-02-16 |       26 |

There were two things that drew my attention to this activity. First, it's an old exploit (detailed in this advisory. Second, there was a sharp spike in activity starting on the 15th.

Around the same time as I noticed this, the SANS ISC Handler's Diary posted an entry about it, and they didn't know what was going on either. I contacted them about it, and have since heard nothing in reply.

I didn't think too much about this at first, but after collecting a few more days' worth of data, I noticed the following: Each source IP only attacked one destination IP, and while is usually a steady stream of 3 or 4 alerts per hour, there are some periods where I go for several hours without seeing anything. These things tend to suggest that the attacks are coordinated, probably through a single botnet.

Today, I heard from several other analysts around the planet who have seen similar traffic. I got the IPs and timestamps from a few of them and am in the process of analyzing some of the data. What I can see so far is that we've got around 280 individual attacks from ~250 unique IPs. Each of the three sites reporting has at least a few IPs in common with the others, further supporting my idea that this is all due to one big botnet.

I don't have much information yet, but big thanks to fellow #snort-gui regulars David Lowless and hewhodoesnotwishtobenamedbutknowswhoheiseventhoughIrefusetousehiscodenameJesus for providing data and sharing thoughts & ideas about this. It may all just turn out to be nothing, but the coordination and stealth involved make it interesting if nothing else.

Update 2006-02-24 15:53: I asked around on the #shadowserver channel, and have found that attacks like the ones I observed have been found in at least one active botnet (not sure if the scattered sightings are from different botnets or not).

Why they are probing for 5 year old vulnerabilities in Cisco routers is still a mystery. Maybe they just want to use them as proxies for IRC/spam/other attacks. I hope that's the extent of their ambitions.

Update 2006-03-06 11:41: Based on conversations I've had with others who are experiencing these sorts of attacks, I've concluded that the most likely scenario is that the perpetrator is hoping to gain access to Cisco devices in order to use them as IRC proxies to hide their true network address while trading credit card numbers and other illicit info. In other words, they can log on to the devices and initiate outbound network connections to IRC servers. One analyst speculated that there may be an IRC client plugin of some sort to facilitate this, and that seems like a reasonable assumption.

BTW, if this technique is used to disguise IRC, it may also be used to disguise connections to other services as well (HTTP?). This all seems like a pretty roundabout way to achieve the hypothesized goal, but certainly in keeping with technology that existed at the time the original exploit was discovered. Maybe someone is just using some old scripts they found online somewhere?

Update 2006-03-07 06:53: The Philippine Honeynet Project (which I reached via this SANS ISC diary entry) has also noticed these probes. Their analysis includes the name of the tool they believe responsible (ciscoscanner). Unaccountably, though, they're suggesting that the tool is looking for the vulnerability listed in this advisory, which doesn't seem to fit the observed data. It's good to read that some other people have finally noticed this, at least. They kind of stick out like a sore thumb if you're paying attention.

Dshield for IDS systems?

Most of us are probably familiar with the Dshield project, which collects firewall logs from users around the planet. They process this data to extract attack trends and produce reports we can use to help evaluate the potential "hostileness" (my term) of a given IP address.

While going through my morning sguil operations, I thought to myself, "Why haven't we done this for IDS events yet?"

My idea is very simple. Sguil requires an analyst to assign each event to a different category (e.g., "successful admin compromise", "attempted compromise", "virus activity", "no action required", etc). For those categories which correspond to confirmed malicious intention, why not collect data about the alerts and forward them to a central clearinghouse?

Of course, you wouldn't want to collect all the data, for privacy reasons. You'd probably only want the attacker's IP, the timestamp and a common reference number to show what event occurred (the snort sid, for example, though that only applies to a single IDS). You could further apply some basic rules to the data before submission, to delete all records originating from your own network, for example.

So what would be some of the benefits of this system? First, the clearinghouse would be in a position to notice hosts which were attacking large segments of the Internet. This determination would be based on confirmed data submitted by human analysts and would therefore be more inherently trustworthy than data garnered from firewall logs. Second, the clearinghouse could directly observe and report on the types of attacks being seen in the wild, something the existing Dshield project cannot do. Finally, this data could be fed back down into the IDS analysis tools to help adjust the severity of observed events based on how similar events were previously categorized by other analysts. Sort of a Cloudmark for IDS.

I'm not suggesting that there's anything wrong with Dshield. They're providing a great service that I rely upon to help me make decisions every day. It's just that they're only capturing one type of data (firewall logs). I think their model could logically be extended to IDS operations, to the benefit of the entire community.

Update 02/14/2006 11:36: Someone on the #dshield IRC channel mentioned that this idea also sounds similar to Shadowserver. True enough, though they're using their own net of mwcollect and Nepenthes collectors to track malware distribution, and so they're not capturing IDS data. Great for tracking automated attacks and generating high-quality blacklists of sites actually distributing malware. Not quite what I had in mind, but also a very valuable dshield-esque service. Check them out, but keep in mind that they haven't gone public with their data yet. If you ask nicely, though, they might give you access to the IRC channel that keeps track of the botnet reports in realtime. I'm going to check it out.