Useful data points for shaping your SDLC program
By Robert Auger
1/11/11
If you work in infosec for a large organization it can be difficult to easily track the state of every software level vulnerability throughout your various code bases. This is particularly true when groups outside of infosec such as the business unit, development, or QA are filing these defects and fail to loop in infosec (possibly because they don't know how!). Getting a grasp on how issues are being identified, and handled is essential for improving your orgs security program/s. By making a few changes to your bug tracking system it can become easier to understand the issues being discovered, effectiveness of certain testing tools and strategies, effectiveness of defenses, and can help improve processes addressing security related defects. This may involve tweaking your security testing approaches/processes, defect handling process, and security related frameworks. In the sections below we'll walk through a series of attributes that you can add to your bug tracking system that can be queried to help identify patterns that can be reviewed to improve your company's security posture.
Security Flag
The goal of this flag is to have a true/false attribute to indicate if a defect is security related in nature. This allows you to easily query all defects for the existence of this flag and pull up (hopefully) all security related defects. From here you can gather metrics on security defect priorities, business units with the most defects, time to close, and other data points which we'll discuss later in this article. The simplicity of this flag also allows people who are not very familiar with attacks and weaknesses to clearly label an issue as a security concern. In large companies people may not know what to do when a defect that is, or may be security in nature. While educating groups on how to engage security is out of scope for this post, the existence of this flag can alert infosec to potential problems, both defect related, and related to user education.
It's worth noting that there will likely be security related defects in your system already lacking this flag, so it is important to go back and apply this flag to previous issues. Lastly you should periodically query your bugtracker for security related keywords and to apply this flag to any missing items.
Security bug source
While a lot of folks track where in the development lifecycle an issue is discovered, which is a good metric to track, many don't track the circumstances of how exactly an issue is discovered. Being able to track the origin of a defect allows you to determine
- How effective certain security tools are
- How effective certain manual security testing approaches are
- How effective penetration testers are
- Where in the product lifecycle issues are discovered
If you're using multiple security tools, tracking the effectiveness of one tool over another for certain vulnerability types can be beneficial to your testing program. If for example you're using a static analysis tool and a blackbox tool running the same types of tests, there will be situations where one tool will be better at finding a given vulnerability class. This can give you insight into what these types of tools are, and are not good at finding (hopefully your organization has a good understanding of how/when these tools are helpful already). Below are some examples of sample 'Security bug source flags' which can be listed within a drop-down on your defect form.
Sample 'Security bug sources'
- Internal - Dev
- Internal - QA
- Internal - Static Analysis
- Internal - Blackbox Security Tool
- Internal - Other
- Penetration Test
- External
- Company Partner
Security Bug Type/Vulnerability Type
By providing a drop-down list of the most common types of security vulnerabilities it becomes easier to track the types of issues being discovered. Tracking your org's vulnerability types can help to identify:
- Gaps in your organization for certain types of threats: From vulnerability identification, to mitigation.
- Gaps in the security testing process: This can give insight into gaps within tools, manual testing, or external testing.
- Number of issues by vulnerability type: This gives your department insight into the most common issues, allowing you to target the focus accordingly.
If for example your most common security defect was Cross-site Scripting this will allow you to investigate if
- XSS protections are lacking from certain code bases/technology stacks
- Any existing XSS protections are doing a bad job, or are not covering all contexts
- The protections are not being used correctly, or are being disabled
The WASC Threat Classification is a good starting place to obtain a list of the most common attacks and weaknesses to pre-populate this list with. If you're looking for a more comprehensive list I would strongly suggest checking out Mitre's CAPEC/CWE projects.
Getting notified of defect updates
Depending on your bug tracking system you may have the ability to receive notifications when certain actions are performed. This can give you insight into how development is handling each issue, and if they are handling it according to process. Below are specific things that I've found worth tracking, and why.
- Priority change: Useful to identify if an issue has been incorrectly downgraded, or has been escalated.
- Assignment or removal of the security flag: This gives you into insight when people don't think an issue is a security related, or later identify an issue to be a security concern.
- Defect state change (open, close, duplicate, etc): To identify if issues are new, or have been fixed. Tracking open issues with the security flag set to true is particularly useful.
Following up on a defect when any of these changes occur is essential. Last but not least tracking the time that it takes to close a defect by its priority is a great metric to determine if bug handling processes are improving, or are not being followed.
In Closing
My motivation for this post was to share some of the approaches that I've found useful in my job as an infosec engineer focused on application security since little has been discussed (if at all) on the subject. While the general tone of this article is targeted towards infosec, many of its points can be driven out of development, qa, or the business unit.
About the Author
Robert Auger co-founded the Web Application Security Consortium (WASC) - a group dedicated to developing and promoting "security standards of best practice" for the World Wide Web - in 2004 where he currently leads multiple projects. He also runs the oldest application security site on the web CGISecurity.com and also runs QASec.com where he addresses security throughout the Software Development Lifecycle. Robert currently is an application security engineer at PayPal where and works on application security, and Secure Development Lifecycle program development.