Well...it's that weird period between Christmas and New Years, and I've realized that I hadn't gotten anything for those wonderful people that keep the VRT employed.So as a gift to you, software vendor, I present Matt's Guide to Vendor Response.Now...this is a complicated subject.It has to be, because every company I've dealt with has had to work through it painfully.Hopefully, this guide should help whoever follows in the footsteps of vendors past to more easily move from "security through no one is looking at me" to a company who operates in balance with business drivers and obligations to customers.

First, let’s cover some concepts:

VALUE OF VULNERABILITY -- The most important thing to understand about vulnerabilities is that they have value.Depending on several factors (the same ones your end users use to gauge risk:ease of exploit,how widely deployed the software is, remote/local etc...) it can be worth a great deal.When someone comes to you with vulnerability data, you should be grateful.There are a number of commercial entities that will purchase vulnerability data and most of them will then pass it on to you.But there are a number of other avenues that a security researcher could take.Various government agencies in the U.S. Government will purchase vulnerability data, especially if it’s been researched fully to the point of exploit.Other organizations around the world, both governmental and non-governmental, will also pay money for vulnerabilities.

Know that when I say pay money, I mean good money.(Matt Watchinski in a presentation to DojoSec about the Adobe JBIG2 issue estimated the value of the bug from between 75 to 100 thousand dollars).Here's a pro-tip:Head to Vegas and hit some of the late night DEFCON parties.Search through the crowd and find the group wearing the custom suits, nice watches and Italian shoes.Congrats, you've found some of the real deal.(By the way, don't discount the dude in the green shirt with green spiky hair...he's just keeping it on the down-low).Here is my point, when you get vulnerability data, someone has given up what could be a substantial sum of money to provide you with that data.Treat them well.Be respectful, when you announce the patch, put their name in it.They've done you and your customers a great service, and it costs you nothing more than having someone on your staff act like a half-way decent human being.

IMPACT OF 0-DAY -- I really believe that a lot of the software development community simply haven't internalized what an 0-day means in terms of impact to customers.When there is an in-the-wild exploit of your software, you should turn all of your resources to alleviating that problem.Most operations folks flail when 0-day comes out, no one will give them enough information to mitigate the issue, they are at the mercy of the vendor's patch cycle and at the core of it all is a gaping hole into their data.Depending on your application, this could be state secrets, medical records, credit card numbers etc...So when you sit in your meeting to figure out how to handle the mess that the failure in your secure software development lifecycle (you do have one of those, oh...please don't make Michael Howard cry...) has caused, keep in mind the real world impact (big or small...) that the problem is causing.

Now, I'm going to keep this simple.There are a ton of policies and spread sheets and threat matrices and God knows what else that some CISSP somewhere will help you build.But here is how, at the absolute root of the issue, to handle security issues in your products:

CASE 1:You've been confidentially notified by a security researcher that there is vulnerability in your product.

RESPONSE:

First, thank the researcher and let him have a point of contact on your security team to discuss the issue.As soon as your triage of the bug is done, notify the researcher of the timetable to patch.Feel free to put in whatever caveats you need to, but keep that communications channel open.You never know when it will be useful to you.

Second.Don't tell anyone who doesn't need to know anything about the vulnerability.Configure your bug handling software to have a security flag that allows viewing only by a dedicated security response development team.If you have an external bug tracking system, remove immediately any information about the bug.Many of you might wonder why this is...especially after I've repeatedly demanded that as much information as possible be given to operations folks.But here is the deal...as soon as any information about an unpatched vulnerability is out, attackers will start the process of converting that information into a working exploit.

For example, when the Adobe JBIG vulnerability came out, Sourcefire was only aware that there was an issue in PDF files that had JBIG tags.So Lurene and Matt set the fuzzer on it, starting at the JBIG tag.It took less than 10 minutes to find the vulnerability, and a short time later we understood the execution path.We're good...but so are the attackers.So until such time that there is information in public, lock down any info you have.

Third.Triage appropriately and patch as soon as is reasonable.I'm not suggesting that you bend over backwards to get a patch out tomorrow for undisclosed bugs.Keep an eye on your security feeds, notify your security vendor partners to be on the lookout for particular strings or conditions and get to work on patching within your normal development framework.Just ensure that you have a process in place to expedite the patch process if the vulnerability becomes actively exploited or the vulnerability details are released.

A quick note on "security vendor partners". If you, as a company, are truly concerned about the operational impact of vulnerabilities in your software on your customers, you should create relationship with security vendors NOW so that they can help protect your customers when vulnerabilities arise. Contact IPS/HIDS/SIM/AV vendors so that when you are under the gun, you have both resources you can reach out to and the ability to tell your customers that while you are working on the patch, their security vendors can provide coverage to protect them. This goes a long way towards making security operations people feel better about your product. Microsoft's MAPP program (http://www.microsoft.com/security/msrc/collaboration/mapp.aspx) that Sourcefire participates in is an excellent example of this kind of program.

Finally, deliver the patch.When delivering the patch, remember to credit the researcher and, if appropriate, his or her company.Also, understand that once you deliver the patch, sufficient data is now in the public arena to create an exploit.So provide as much data as you can so operations folks can make an intelligent decision on how quickly to roll out the patch, and what mitigations are available prior to that patch roll out.Remember that, for better or worse, if there is no active exploit, many organizations require extensive testing prior to a patch roll-out, so give the ops team options to handle things prior to patching.

CASE 2:Information has been publically disclosed regarding a vulnerability, but you have seen no active attacks.

RESPONSE:

This is the tricky one, and is one of the reasons you need to have a dedicated team with the appropriate skill set to handle security issues.First, you need to interpret the level of detail that has been provided to the public.If there is a Metasploit module, just skip straight down to active attacks, because it's on like Donkey Kong (TM, or something).If it is an announcement to bugtraq or full-disclosure, have someone who understands exploitation look at the data and help you determine the risk.If you are completely clueless on exploit development (and really, you have to be willing to admit this) reach out to your security vendor partners for assistance.

Consider reaching out to the original poster of the data.Yes, this can hurt your pride, but remember that your company has caused (minor|moderate|severe) issues and you should feel some obligation to suck it up and fix it.Remember at this point few people are really going to beat you up over the fact that your code has bugs (whose doesn't?) but it’s how you respond that will define how you are perceived.

Now you've got enough data to make some decisions.Based on that data, and your policies, pick from the following:

1)If you feel that the level of disclosure is such that it creates a near-immediate threat to your customers, you need to skip immediately to the active exploit case.

2)Announce the fact that you are aware of the vulnerability and that a patch is in the works.Give a set of mitigation options for people to use until a patch comes out.Provide no unnecessary data.Engage your security vendor partners.

3)Announce the fact that you are aware of the vulnerability and that a patch is in the works.Provide all the technical details you can including locations of exploit code, how error logs look when the exploit is attempted, mitigation strategies and suggested detection techniques.Engage your security vendor partners.

Number three will probably look pretty extreme to some folks, particularly of the severe white-hat persuasion.But let me tell you a secret:There are no secrets.If the vulnerability is out there, you and every ops group on the planet are in a race against the hackers.In all likelihood, in some areas, you've lost the race already. But either way the only fair way to treat customers when the threat is imminent is to provide them every tool they can to get in place to weather the storm while you get your patching in order.

I'm going to bang on Adobe pretty hard here in a minute, so let me point out something they did that was actually close to right.There is exploit code available for Adobe Illustrator CS3 and CS4 for an .eps file parsing overflow.It has not, to the best of my knowledge, been seen in the wild, but it would be an excellent targeted attack vector.Adobe announced that it was aware of the issue, gave a date for the patch and advised that customers not open .eps files from untrusted or unknown providers.

Now, while I would love a better set of mitigations, I'm not sure what you could do in Illustrator.But I have enough information at this point to write mail filters, IDS rules and log parsers looking for .eps transfers.(You can find the notification here:http://www.adobe.com/support/security/advisories/apsa09-06.html)

All that being said, here is why I said "close to right". There is publicly available exploit code that includes the Metasploit windows/adduser and windows/shell_bind_tcp shellcodes. You can find that code here: http://www.exploit-db.com/exploits/10281. This is important because it allows you to write much more specific detection, allows you to test your detection with actual exploits and the mere fact that it exists tells you that there is a true threat to you if you use Illustrator and .eps files. Adobe did not provide a link to the exploit, which is very useful to organizations with advanced security practices. Adobe did not even disclose the existence of exploit code (which changes the response priority in most organizations).

CASE 3:You discover that a previously unknown vulerability is being exploited in the wild.

RESPONSE:

First, have a meeting and explain to everyone that your company has screwed up.Say it just about in those terms, get people in the mindset that you have an obligation to your customers.Make them understand that there are companies that will be adversely affected, perhaps dramatically, by the vulnerability in the software that your company makes.

Second, notify your dedicated security development team that they will now be earning their money.Overtime as necessary until it is resolved.Notify your QA team that they will have to put on hold whatever they are doing and prep for testing of a patch.Notify your web and infrastructure team that you will be having an out-of-band patch and they need to be ready whenever the patch is ready to publish.

Third, immediately notify your customers that there is an active exploit.Give every possible piece of data you have.Exploit code, sample files, log snippets, IP addresses of known attackers...everything.You owe them every piece of information you can possibly give them while your team slaves away on a patch.Now is not the time to be sly or hide behind your website.Get out there and earn the trust of your customers.

OK...now is Adobe smashing time.Let me tell you the absolute WRONG way to go about it. I'll use Adobe twice, because they've screwed up at least twice this year, and they are dead-center of the client-side bullseye right now.

First, with the JBIG2 problem, it was abundantly clear that they simply didn't have a plan in place for handling active exploitation.The vulnerability, which was very, very easy to take to exploitation was first announcedpublically sometime around the 19th of February by ShadowServer (http://www.shadowserver.org/wiki/pmwiki.php/Calendar/20090219).Also, it was noted that this vulnerability was actively exploited in the wild.It became apparent that this had been exploited since at least January 1st (we're thinking earlier...which is a bang against anti-virus vendors (us included) for not finding it earlier).We also find out that Adobe was notified by McAfee around January 11th of the problem.Long, sad story short, no patch was issued until March 23rd and then only for one version of Reader and no versions on the Unix platform.

A second, actively exploited 0-day was found in December.This time Adobe released a security advisory with well documented mitigation techniques (find the advisory at http://www.adobe.com/support/security/advisories/apsa09-07.html) including some accurately worded (reduced risk...) information about the role of DEP in the exploit.

Now, I missed this on the first pass through the bug, as I assumed the JavaScript Blocklist Framework technote was their standard blurb about the Blocklist. But the technote is actually where most of the data I would want to have from an ops side is located, including naming the function with the problem (DocMedia.newPlayer). With the information in the technote more accurate mitigation is available. I would have liked to of had that on the front page, but a diligent ops guy would find it. (Find the technote at http://kb2.adobe.com/cps/532/cpsid_53237.html)

The real issue, though, is that the patch would not be available until January 12th. For organizations where the JavaScript Blocklist Framework wasn't an option, such as those groups running non-Windows systems, or Windows users that don't have the ability to fix up all of their end-user registry settings or home users who most likely have no idea there is a threat at all, this time period is one where there is very little in the way of mitigation. Microsoft patches every month and still out-of-band patches when there is a significant 0-day threat. Why can't Adobe break into its quarterly patch cycle for the same reason?

ComputerWorld seemed to think this was a problem also, and had an article entitled "Adobe explains PDF patch delay" (http://www.computerworld.com/s/article/9142479/Adobe_explains_PDF_patch_delay).The first paragraph pretty much sums up Brad Arkin's (Adobe's director for product security and privacy) explanation of the delay:

"Adobe chose to wait until mid-January to patch a critical PDF bug because issuing an emergency update would have disrupted its quarterly security update schedule, the company said today."-- ComputerWeek's Gregg Keizer, December 18, 2009

Unacceptable.Seriously, there is no other way to describe this situation.You cannot justify pushing out an actively exploited 0-day vulnerability because "it would disrupt the quarterly update schedule".I invite you to read the article to get Brad Arkin's exact wording.Part of his justification was that they had deployed the JavaScript Blocklist Framework, which if you were at the correct version, would allow you to lock down JavaScript enterprise-wide.

What is so important in the quarterly patch that it can't be put off? The obvious answer is the Adobe Illustrator bug with public exploit code...maybe that should be rolled into the out-of-band patch, don't you think? If this weren't a blog that represents a group of people that are part of a company, I'd say this more plainly than this, but here is my final thought on Adobe over the past year:

WHAT THE HELL?

Look, Adobe has been doing better lately, but I think they still have a ways to go.

CONCLUSION

If you skipped to the conclusion (and who could blame you, it was almost too long, did not write...), let me lay it out for you:

1)Have a plan

2)If an exploit is underway, give customers every piece of information you have and patch as quickly as you possibly can.

3)If no one knows but you, keep your mouth shut while you patch.

4)Treat your security researchers well, they are doing you a huge service.

5)Know that you have an obligation to your customers to protect them from vulnerabilities in your software, give them every tool and piece of information possible to protect themselves while you code the patch.

Not surprisingly, the VRT is part of the Sourcefire vulnerability response process. We really do try and follow the ideas I've expressed here. Hell, we even credited Neil Mehta (an ISS employee, you know...one of our competitors) for the work he did on the Back Orifice buffer overflow. Which we patched, tested and released within 80 hours of learning of the problem.

We need to demand more from our software vendors (including Sourcefire, keep us honest). Use whatever leverage you have (and it is pitifully small in many cases) to get a better response from your software providers.

As always, I welcome your comments in the comments field below.