This post is entirely for developers. Feel free to read, but approach it with that in mind.
There is no such thing as bug-free code, and any code, even the most secure, can, with time, can be used for nefarious actions. We ourselves find weaknesses in our code, internally and externally, and have to work extra hard to stay ahead of the issues. We all have oversights and missteps and issues that we could never have anticipated getting introduced. Some of these issues might have security implications affecting the software we build, and those are the things that get categorized as potential software vulnerabilities.
I think it’s fair to assume that we can all relate with this to some level. What seems to be a problem however is how we, using the collective we, handle the disclosure of these vulnerabilities when brought to our attention.
- What happens when someone identifies a vulnerability in the code we write?
- What if it can or is being misused to hack websites that employ that same code?
- How should we as developers respond and handle these situations?
I want to share some thoughts that I hope will provide some insights on a good disclosure and engagement strategy.
Accounting for Vulnerability Disclosures
If you are a builder and are shipping code the first thing you should plan for is a method to allow researchers to engage you if and when issues are identified. This is first and foremost; you want to encourage folks to review your work especially if it’s going to be used publicly and should welcome as many eyes as possible. You don’t have to set up official bug-bounty programs or anything along those lines like the big companies, but some method to ensure that you get notified immediately of issues. We should all want to know if there is serious problem with our code.
Here are 6 steps we encourage every builder take into consideration when an if a vulnerability disclosure occurs:
1. Do Not Panic
Take a deep breathe. Everyone makes mistakes and this is likely not the end of the world. It may seem like a big deal and you may get worried that you will lose business and the trust from your users. However, if you deal with the disclosure properly, it may actually have the reverse effect and you may actually gain their trust from this experience.
I think as humans we have a predisposition for jumping straight to the negative. I’m going to lose business!! I’m going to go out of business!! We should flip this mindset to one that is more approachable, What can I learn from this? You’re in essence getting a free security review; this is a positive thing and you want to encourage an open dialog with the researchers.
As soon as you get the notification, let the reporter know that you have received the notification and are looking into it. We sometimes fail to realize how important this small step can be. I can tell you first hand, researchers have a low tolerance for unnecessary delays.
In the business of disclosures, it’s not uncommon for organizations to shun them away, and over time it’s built a culture of distrust; forcing them to believe the organization doesn’t care. This reduces the tolerance for lack of acknowledgement and can often lead to bad outcomes for everyone involved.
A simple, Thank you so much, we’re looking into this and will get back to you within X hours will pay you huge dividends.
3. Do Your Homework.
Take all the information provided by the researcher and try to confirm the issue. If you can’t reproduce the issue, don’t jump the gun and tell them they are wrong, instead ask for something known as Proof of Concept (PoC). Remember, this researcher is taking time out of their day to report something to you to help you, attacking them or blowing them off will do little in your favor.
Once the PoC is provided, you should work to provide the researcher a timeline. In the world we live in today, with web applications, timelines that extend beyond 2 weeks really aren’t acceptable. If you must extend beyond that, be sure to communicate with the researcher on the reasons behind the delay.
The pressure to release sooner will honestly be determined by its severity, ease of exploitation and impact to the larger audience.
4. Patch It!
Please patch the problem effectively. One common mistake is to patch it where it was identified, but leave it susceptible in other parts of your code. Don’t be lazy.
Take the time to review the rest of your code for other implementations, if you fix the weakness show that you’ve learned something and fix other, yet similar, examples in your code base. There is nothing more frustrating to a researcher if they find the same issue riddled through your code even after the disclosure; it shows a lack of care or concern and that will make them less likely to engage.
5. Coordinated Disclosure
Before you release the patch, involve the researcher. Send the fixed version to the researcher, ask them to review, engage them in the process and watch it pay you dividends. They will often go above and beyond to help you think through the problem, and will help built a rapport. Once this is done, you have to plan a release – there really isn’t a way around this, especially for severe vulnerabilities.
Let the researcher know when you plan to release. Work a coordinated disclosure release schedule, most researchers love to identify problems but also love to write about their findings. They do it to share their knowledge but also to share with their peers, it’s why they do it.
A good email to the researcher would be:
I will be publishing a patch on DAY X at Y time (GTM). I will follow the release with a blog post of my own shortly and will email all my clients. Can you wait 24 hours for your own disclosure? I Want my users to hear from me first.
This is a very acceptable email and most reasonable researchers will understand and respect your request. Please do not do a silent release – a silent release is a release where you avoid any mention of the vulnerability in the hopes that no one will ever find out. Especially if coordinating with a researcher. The researcher is monitoring all patches, if they see you do this without any proper disclosure they are bound to call you on this act, publicly.
When writing a blog post (and in the release notes), you should explain what happened, the severity, who reported the issue to you and what you did to fix it. Provide links for the patch and a timeline if possible. Your users will appreciate and trust you more when you do this. Again, do not try to hide if it is a security issue. The idea that by not sharing you don’t have to worry about hackers exploiting is an antiquated way of thinking. A true cracker worth their salt, can run a diff and see the issue right away; not saying anything does not solve the problem.
6. The Backlash.
This seems to be the biggest fear most builders have. What will my community think? What will clients think? Will my company survive?
These are normal feelings, we can assure you though this will pass. In many instances, buyers will gain renewed faith in their provider understanding that sometimes things happen. If you get a client that is extremely upset, do what you can for them to explain the situation, but sometimes it’s ok to Fire or let them go.
If you can ride the storm you will often come out with the upper hand. What we encourage you not to do is attack the researcher, the disclosure space is highly finicky and in the eyes of many if a builder starts to attack a researcher defenders from around the web will come to their aid. This is where things go bad really fast, so just take the high road.
Yes, an issue was found, but more importantly this is what we learned and what we’re doing moving forward.
Example of Good Disclosures
Remember, no one is perfect and even if we are, with time, we’re all shown how imperfect we truly are. I want to leave you with a good example from the JetPack team. They recently handled a vulnerability disclosure and did so brilliantly in our eyes: http://jetpack.me/2014/04/10/jetpack-security-update/.
They patched the issue right away and notified everyone, including hosts and partners about the problem. Kudos to them!
I welcome thought and opinions in the comments, and if something makes sense we’ll address it accordingly. As mentioned before, this was drafted to provide you some thoughts and guidance, not dictate. Yet, we hope you find it helpful.