Tuesday, February 14, 2006

Dshield for IDS systems?

Most of us are probably familiar with the Dshield project, which collects firewall logs from users around the planet. They process this data to extract attack trends and produce reports we can use to help evaluate the potential "hostileness" (my term) of a given IP address.

While going through my morning sguil operations, I thought to myself, "Why haven't we done this for IDS events yet?"

My idea is very simple. Sguil requires an analyst to assign each event to a different category (e.g., "successful admin compromise", "attempted compromise", "virus activity", "no action required", etc). For those categories which correspond to confirmed malicious intention, why not collect data about the alerts and forward them to a central clearinghouse?

Of course, you wouldn't want to collect all the data, for privacy reasons. You'd probably only want the attacker's IP, the timestamp and a common reference number to show what event occurred (the snort sid, for example, though that only applies to a single IDS). You could further apply some basic rules to the data before submission, to delete all records originating from your own network, for example.

So what would be some of the benefits of this system? First, the clearinghouse would be in a position to notice hosts which were attacking large segments of the Internet. This determination would be based on confirmed data submitted by human analysts and would therefore be more inherently trustworthy than data garnered from firewall logs. Second, the clearinghouse could directly observe and report on the types of attacks being seen in the wild, something the existing Dshield project cannot do. Finally, this data could be fed back down into the IDS analysis tools to help adjust the severity of observed events based on how similar events were previously categorized by other analysts. Sort of a Cloudmark for IDS.

I'm not suggesting that there's anything wrong with Dshield. They're providing a great service that I rely upon to help me make decisions every day. It's just that they're only capturing one type of data (firewall logs). I think their model could logically be extended to IDS operations, to the benefit of the entire community.

Update 02/14/2006 11:36: Someone on the #dshield IRC channel mentioned that this idea also sounds similar to Shadowserver. True enough, though they're using their own net of mwcollect and Nepenthes collectors to track malware distribution, and so they're not capturing IDS data. Great for tracking automated attacks and generating high-quality blacklists of sites actually distributing malware. Not quite what I had in mind, but also a very valuable dshield-esque service. Check them out, but keep in mind that they haven't gone public with their data yet. If you ask nicely, though, they might give you access to the IRC channel that keeps track of the botnet reports in realtime. I'm going to check it out.


Chas Tomlin said...

This is a fantastic idea. I've bodged up a quick RSS feed into some of my sguil data, its available here;


This gives you the last 10 WEB-ATTACKS alerts along with the src IP and timestamp


David Bianco said...

Cool, Chas! Any code you could share?

Chas Tomlin said...

Sure, take a look at;



This gives a feed of wget attempts which could be a new way of collecting malware as typically these are attempts at downloading malware on to vulnerable servers.

If I have time i'll knock up some scripts to parse the RSS feed and collect any malware.


Richard Bejtlich said...

Hi David,

Talk to Bamm about his OpenNSM idea. It's what you are discussing.

Also, there was support for Snort logs in DShield, but it doesn't seem to have been that popular.

[0] http://lists.sans.org/pipermail/list/2001-April/000482.html
[1] http://groups.google.com/group/mailing.unix.snort/browse_thread/thread/3fa3460b688ecc24/26bb40b42959bf22?lnk=st&q=dshield+snort&rnum=2#26bb40b42959bf22