tag:blogger.com,1999:blog-7652481.post3207576328334394365..comments2008-11-23T05:19:24.605-05:00David Biancohttp://www.blogger.com/profile/09760835714791462863noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-7652481.post-48945636404374579922008-11-23T05:19:00.000-05:002008-11-23T05:19:00.000-05:00H,<BR/><BR/>Is SGUIL capable to handle huge mount of raw data capture in ISP with say 300Mbytes of data minimum ?<BR/><BR/>In this case we need to equip with a huge harddisk for data capture in one week and does it make sense in practice ?<BR/><BR/>Or SGUIL will only store raw data traces for those IDS alerts being fired? ThanksVictorhttp://www.blogger.com/profile/17528849061159411850noreply@blogger.comtag:blogger.com,1999:blog-7652481.post-51668459444479228632008-05-18T12:12:00.000-04:002008-05-18T12:12:00.000-04:00Martin, thanks for the informative comment. I found a similar speed increase, although I also found a performance bottleneck with the speed of the disk reads for the index files. To get the best performance, it looks like the index files should probably be on a different disk than the pcaps, especially for more active networks such as yours. I don't think SANCP provides a convenient way to do this yet, but perhaps I've simply overlooked it. <BR/><BR/>As for your point about typical searches and the retrieval time, you're right for UDP and TCP searches. They're usually unique enough to give good performance, because at least one of the port numbers is bound to be ephemeral, and therefore acts like a database key. For things like ICMP, which doesn't use ports, you could be getting more data than you wanted, though hopefully the sheer volume of ICMP between any two hosts isn't going to be very high.David Biancohttp://www.blogger.com/profile/09760835714791462863noreply@blogger.comtag:blogger.com,1999:blog-7652481.post-22126204727171965442008-05-17T12:49:00.000-04:002008-05-17T12:49:00.000-04:00That's great to hear that you've updated SGUIL to use the pcap indexing. For those of us who deal with big pipes, SGUIL really doesn't scale without some help. I first suggested the indexing idea to John over a year ago, and he took it and ran with it, and now it's pretty solid. <BR/><BR/>I got similar disk cost results as you did. Obviously, the smaller the average packet is, the larger the index will be with respect to the pcap. Search performances were <B>several orders of magnitude</B> faster. My pcaps are somewhere in the neighborhood of 30 GB for a 15 minute time period, and indexing was able to retrieve arbitrary packets in about <I>5 seconds</I>. <BR/><BR/>However, the indexing speed increase is only really apparent if the needle in the haystack you're looking for is fairly unique. That is, if you're searching on some characteristic that exists many times throughout the index, (e.g. searching traffic for a very busy host) then you will see less of a speed increase because the bottleneck is not search but retrieval. That said, typical searches are usually unique enough to see get the big performance boost.<BR/><BR/>Many thanks to John for another great feature in SANCP and thanks to you for getting it into the hands of the larger community via SGUIL.Martinhttp://www.blogger.com/profile/03975313410819886706noreply@blogger.com