So, this week, the OISF has been on a media blitz about Suricata, their open-source Intrusion Detection System. As always, my preference is for you to review the information yourself, so before I give you my thoughts about the state of Suricata, here are some links:
http://www.computerworld.com/s/article/9179436/DHS_vendors_unveil_open_source_intrusion_detection_engine?taxonomyId=82
http://www.darkreading.com/smb-security/security/intrusion-prevention/showArticle.jhtml?articleID=225900192&cid=nl_DR_DAILY_2010-07-20_h
http://www.openinfosecfoundation.org/
When I talked to Matt Jonkman about Suricata this past December, I was excited. He talks a good game and his pitch of Suricata as a "Next Generation Intrusion Detection and Prevention Engine" was catching. It is always good to step outside the box and take a good hard look at how any industry is approaching things and innovation is always welcome. The fact that Suricata is Open Source and driven by government money meant that the innovation would be available and easily ported to Snort, helping not just Suricata users, but Snort users as well. To me, it looked win-win all the way.
I was so impressed by the tack that Suricata appeared to be taking that I gave my best wishes to them on their first release, back in December:
"Congrats to Matt Jonkman and the team at OISF. It's a big step, and I look forward to seeing your work (after then new year :))
Matt". -- [Snort-users] Suricata IDS Available for Download!, 12/31/09
http://marc.info/?l=snort-users&m=126229251430261&w=2
But at this point, having worked with Suricata and looked at what the OISF has actually delivered, I'm just disappointed with where they've ended up and what they've delivered.
Suricata's developers harp on a lot of different issues, some of which are valid, and some are simply wrong. More than anything else, they beat the multi-threaded drum:
".For example, Suricata's multi-threaded architecture can support high performance multi-core and multiprocesser systems, Jonkman said." -- (Computerworld, above)
"This is 2010 and not a single IDS supports multi-threading"
"The flaw in every IDS is that it is single threaded"
"Multi threading alone is worth moving over to Suricata for".
-- (Suricata team, various)
We've talked about this before, with an extensive, technical discussion on multi-threading and IDS from Marty (http://vrt-sourcefire.blogspot.com/2010/06/single-threaded-data-processing.html). Go there and check out what Marty has to say. The essence of his post is that there are sound architecture reasons why traditional multi-threading is not the appropriate approach for IDS implementations. Certainly, as commodity hardware evolves (we do work very closely with Intel), we'll continue to evaluate the technology. In addition to the long technical brief, I'd point out this simple fact of life: IDS vendors are judged on both detection accuracy and speed. No report about an IDS engine comes out that doesn't identify how many GB/s throughput the engine was capable of sustaining. Trust me, if multi-threading were the answer, the industry would have moved there in short order.
Of course, there is no third-party review of Suricata's performance. I'm going to give you some numbers, but I don't expect you to put any special weight on them, they are more to show that we have looked at the engine and haven't found anything that we would take from it. I asked Chris McBee, one of the researchers here on the VRT to install Snort and Suricata on the same box. I told him to make it run, take all the steps necessary to maximize performance. He even changed out the network card to support the pf_ring buffer. We did not, however install a CUDA-capable network video card (edited after publish, my bad -- molney), since, in the words of one Suricata developer (it should be noted that the 1.0.0 release notes indicated an increase in performance):
"Currently on my desktop CUDA actually slows things down"
--http://www.inliniac.net/blog/2010/02/20/suricata-has-experimental-cuda-support.html
So the key to the test is that they were on the same, commodity hardware. We ran Snort first, then Suricata on the same box. In each test, Snort and Suricata were loaded with the latest default open source ruleset from the VRT. In the case of Suricata, some rules that used unsupported options failed to load, and there is no .SO rule support, so none of those rules ran either. This means that Snort was running a larger ruleset than Suricata.
Test Set 1: The same packet set sent at between 200 and 5200 MB/s (30 runs total)
Total Packets sent: 30,000,000 (Across all 30 tests)
Total Packets dropped, Snort: 61264 (0.0204%)
Total Packets dropped, Suricata: 17,438,542 (58%)
Test Set 2: The same packet set sent at between 200 and 5200 MB/s (30 runs total) (Hyperthreading Disabled)
Total Packets sent: 30,000,000 (Across all 30 tests)
Total Packets dropped, Snort: 511 (0.0017%)
Total Packets dropped, Suricata: 15,714,211 (52%)
Now, we did rerun the tests when Suricata 1.0 was released, briefly:
"Suricata peaked at about 300 Mb/s without dropping packets, provided no rules are loaded.
With rules loaded, Suricata runs up to about 200Mb/s.
Snort, with rules, hits 894Mb/s with no drops" -- Internal VRT Report on Suricata Performance
Suricata's performance isn't just bad; it's hideously, unforgivably bad. This is especially true for a project that is hawking the performance issue. There are other issues I have with Suricata, but I think the one that is most concerning is the following statement:
"We made a conscious decision last year that we're not going to go with an obfuscated rule set, we're not going to pick up an SO rule format" -- Matt Jonkman
This decision is going to limit any detection capability to the rule language provided by Suricata. In today's incredibly complex environment, this is an unacceptable limitation. Sourcefire developed the C capability to give the VRT and high-end response organizations the ability to build detection with all of the power of C available to them and not lock them into our the Snort rules language. The loss of this capability dramatically reduces the usefulness of Suricata. As an administrative issue, for organizations working in a TS/SCI environment, rules often need to be provided in an obfuscated format to protect against unnecessary disclosure of data. This isn't available without .so rules.
All of this came up in a discussion of how to engage vendors to get data to write rules. It should be noted that Sourcefire is a partner with several large vendors, including the most targeted: Microsoft. In agreement with these vendors, we protect our rule set for vulnerabilities that are not being exploited in the wild to help protect against exploit development. We do this by obfuscating the rules associated with data we receive from them into .so rules. There is no impact to performance by doing this, but it does pose a hurdle to potential attackers looking to glean information about potential vulnerabilities from how we go about detecting them.
Matt Jonkman specifically said: "We're hoping to get enough market share to say to vendors if they want to use our product if they want to feel safe, we're going to have to be transparent enough that rules can be written". This is simply not going to happen, and trying to adjust reality to justify your philosophical approach will simply result in a product that is not usable.
Look, I generally work very hard to avoid looking like a corporate shill, I try to keep to purely technical information. But in this case, I have to call a spade a spade, even if it leaves you thinking I'm just a mouth piece. But know that I honestly believe this:
The OISF has spent nearly a million dollars to fulfill their obligation to the DHS to deliver the next generation in IDS engines. They have since engaged in all manner of wishful thinking, self-aggrandizement and Snort bashing. They've failed, utterly, to deliver on their promises. This is forgivable on the performance front, that problem is non-trivial. But in the end, what they've built is a poorly functioning Snort-clone, missing the most powerful detection capability that Snort has. There isn't anything in the way of innovation; they are taking the same approach as everyone else from a detection standpoint. Simply put, rehashing isn't innovation.
If you want to see what innovation looks like, come to Vegas and let the VRT show you the Razorback system. It isn't Snort, it isn't ClamAV, and it isn't Suricata. It's a new approach to the detection problem, and was built from the ground up in close collaboration with groups that are facing APT-level threats. It may not be perfect, it may not even be the right answer (but we think it is), but it is truly innovative.
And we didn't even cost you a million dollars.