Holy Grail Found

...or not. Who knows? This article published by the University of Wisconsin-Madison, claims that UWM's Paul Barford has developed technology (called "Nemean") to automatically identify botnet traffic. So far, so good. To be honest, the industry could really use this kind of solution.

Here's the part I have trouble with, though:

The Achilles’ heel of current commercial technology is the number of false positives they generate, Barford says. Hackers have become so adept at disguising malicious traffic to look benign that security systems now generate literally thousands of false positives, which Nemean virtually eliminates.

In a test comparing Nemean against a current technology on the market, both had a high detection rate of malicious signatures — 99.9 percent for Nemean and 99.7 for the comparison technology. However, Nemean had zero false positives, compared to 88,000 generated by the other technology.

I'm not even sure I know where to start here. These numbers sound quite impressive, but without knowing much more about how the tests were conducted, they really don't mean very much. First of all, how many unique types of botnets were tested, and what were they? Also, who chose the test cases? I can achieve a 99.9% detection with 0% false positives too, if I control both the software to be tested and the selection of the test cases.

Next, I wonder what specific technology they tested against, and whether or not they tuned it properly for their test environment. I suspect not, as 88,000 false positives is quite high. I doubt they had 88,000 unique botnet samples, either, so I'm sure there are multiple alerts for each, which can easily be tuned down.

Finally, and most importantly, even if you do achieve near perfect results on a selection of current botnets, this is likely to be only temporary. The Bad Guys are virtually guaranteed to step up their game too.

To be fair, this is addressed in the article:

While Barford has high hopes for Nemean, he says Internet security is a continuous process and there will never be a single cure-all to the problem.

“This is an arms race and we’re always one step behind,” he says. “We have to cover all the vulnerabilities. The bad guys only have to find one.”


At least it sounds as though Dr. Barford does have an idea of the scope of the challenge. Perhaps there is more to this project than was presented in this fluff article. I really would like to hope so, but I'm a bit skeptical based only on the information presented here.