Monday, November 12, 2007

Holy Grail Found

...or not. Who knows? This article published by the University of Wisconsin-Madison, claims that UWM's Paul Barford has developed technology (called "Nemean") to automatically identify botnet traffic. So far, so good. To be honest, the industry could really use this kind of solution.

Here's the part I have trouble with, though:

The Achilles’ heel of current commercial technology is the number of false positives they generate, Barford says. Hackers have become so adept at disguising malicious traffic to look benign that security systems now generate literally thousands of false positives, which Nemean virtually eliminates.

In a test comparing Nemean against a current technology on the market, both had a high detection rate of malicious signatures — 99.9 percent for Nemean and 99.7 for the comparison technology. However, Nemean had zero false positives, compared to 88,000 generated by the other technology.

I'm not even sure I know where to start here. These numbers sound quite impressive, but without knowing much more about how the tests were conducted, they really don't mean very much. First of all, how many unique types of botnets were tested, and what were they? Also, who chose the test cases? I can achieve a 99.9% detection with 0% false positives too, if I control both the software to be tested and the selection of the test cases.

Next, I wonder what specific technology they tested against, and whether or not they tuned it properly for their test environment. I suspect not, as 88,000 false positives is quite high. I doubt they had 88,000 unique botnet samples, either, so I'm sure there are multiple alerts for each, which can easily be tuned down.

Finally, and most importantly, even if you do achieve near perfect results on a selection of current botnets, this is likely to be only temporary. The Bad Guys are virtually guaranteed to step up their game too.

To be fair, this is addressed in the article:

While Barford has high hopes for Nemean, he says Internet security is a continuous process and there will never be a single cure-all to the problem.

“This is an arms race and we’re always one step behind,” he says. “We have to cover all the vulnerabilities. The bad guys only have to find one.”

At least it sounds as though Dr. Barford does have an idea of the scope of the challenge. Perhaps there is more to this project than was presented in this fluff article. I really would like to hope so, but I'm a bit skeptical based only on the information presented here.


Chris Carpinello said...

My snake oil alert went off as soon as I read “The technology we’re developing here really has the potential to transform the face of network security”. For statements like that, my favorite litmus test consists of the guy defending his revolutionary technology with a DefCon demonstration. How long do you think he'd last? I wouldn't give him more than 10 minutes.

DavidJBianco said...

You got it in one. I'm pretty sure we could find at least *one* guy at DefCon with his own botnet. 8-)

Marcin Antkiewicz said...

It seems that they've run their system along snort on one of the UW networks. Default snort sig library will spit out 88k matches without any problem, and above 99% fp rate sounds about right for a generic snort sig library.

That writeup is not the best, but you have to remember the source - non-technical, internal "about us" publisher at UW.

I got a walkthrough through the system some time ago, and it was quite nice - definitely not a cure-all, and under heavy development, but it had the potential to compete with snort, or other sig based IDSes.

for an old paper on the idea ref: