Putting security test cases into your test plan
By Robert Auger
1/5/2007
Part of software testing involves replicating customer use cases against a given application. These use cases are documented in a test plan during the quality assurance phase in the development cycle to act as a checklist ensuring common use cases aren't missed during the testing phase. People within the quality assurance community are starting to understand that checking an application for security issues (defects) isn't just the responsibility of the security department (if one exists), or the software architects. While typical QA Engineers don't understand the scope or inner working of specific software vulnerabilities, they do go about testing an application in a similar fashion to how the penetration testing community does. Unlike typical penetration testing QA has access to internal documents and insider information giving them advantages to aide in the testing of an application. In addition to documenting customer use cases it's important to begin the process of documenting what an attacker may attempt against your application as well and incorporating these attacker 'use cases' into a security section of your standard test plan.
In Preparation For Your Test Plan
Before writing a test case it is important to identify the scope of your testing. I've broken up prep work into three sections below.
Identify Application Input
- Files
- Environmental Variables
- Parameters (This may be received by a web application, command line argument, application form textbox or control, etc...)
- Configuration Files
- The Windows Registry
- Database Source (SQL Database, LDAP, etc...)
- Hidden/Debugging Commands
- Other (Ask your development team for anything else taking in input)
Identify every vector that input comes from that could possibly be modified by a user. Perform basic bounds testing along with security related input validation tests. The Threat Classification document written by The Web Application Security Consortium contains a detailed list of the types of vulnerabilities caused by modifying input along with examples. If an application takes a value by the user and performs a database/ldap query check for issues such as SQL Injection and LDAP Injection. You may have heard of Buffer Overflows, or Format String Vulnerabilities. If you enter in a large amount of data into an application input and the application errors, crashes, loops, acts funny, or shuts down this may be a sign of a potential buffer overflow. Your application developer should be able to track down the cause of the crash and identify the exact cause. Simply put when a large or poorly formatted input crashes an application there is cause for concern from both the product stability perspective as well as security.
Data may travel various paths and each path may be a vector for an attacker to exploit the application. Your development team should be able to inform you of how the application input is used, where it goes, and provide suggestions on how to test for security, and non security related issues.
Identify Application Output
- Files
- Environmental Variables
- Network Traffic
- The Windows Registry
- Console/Form
- Database Source
- Hidden
- Other
Identify every way that the application outputs data. This data doesn't have to be text displayed on the screen, it could be binary data in the form of a file, network stream, or other. Once again I'm going to refer to the Threat Classification document for common types of output vulnerabilities. One of the most common is Cross-site Scripting (XSS) a vulnerability type whose impact is only starting to become more apparent. Originally thought to only affect web based applications it is starting to show up in various other places.
Take into consideration that data outputted to a file may come from a location modifiable by the user and during the input phase is properly sanitized and appear harmless, however could cause damage to other portions of your application trying to read this file later on. XPath Injection, XML parser denial of service, XML Eternal Entity Attacks, Parser Overflows, and even image rendering engines have been found to contain security flaws related to processing of data.
Installation and Deployment
- Installation
- Using an installer
- Using instructions
- Using a zip file or other archive
- Deployment
- Trusted Environment
- Untrusted Environment/Third Party shared hosting environment
Installed applications must take into consideration of the permissions being applied to files and registry keys during installation and during the applications execution. Even temporary files existing for no longer than 1 second during installation or during the applications execution could be all an attacker needs to gain access to sensitive application data or a user's machine. Unix machines have the concept of a user mask which defines how a newly created file's permissions are set. During deployment or installation consider that the user may have a poorly defined user mask which could leave your application open to attack. Ensure that any installer or deployment tool sets permissions appropriately for files, newly created databases, and registry keys (when applicable). If you provide instructions to the user for how to set up your application review the steps involved, permissions suggested and the possible impact of this. One common mistake in relation to web based applications involves telling the user to chmod a certain file or directory to 777. More than a handful of public advisories outline the risks of this permission.
Sample Security Test Cases For A Shopping Cart Application
Below I've split up security test cases into Functional and Logical. The look and feel of your own Test plan format may vary however you should get the point. This is not an extensive sample of what a shopping cart test plan may look like but should convey what sorts of tests cases one might add as well as how uniform many of them are.
Functional Tests
Function Category
- Specific Test Case
Customer Order File
* Ensure that 'orders.txt' file permissions are as restrictive as
possible. If these permissions are loosely defined then this as a
severity 1 security issue.
* Ensure that sensitive data within the 'orders.txt' file is encrypted
using a known strong algorithm. This is a severity 1 security issue.
Customer Data Stored in a SQL Database
* Ensure that sensitive data within the SQL Database is encrypted using
a known strong algorithm. This is a severity 1 security issue.
Registration Form
* For each user input perform common security related input validation tests. See The Web
Application Security Consortium's Threat Classification
for a list of common input vulnerability types. For each input perform
each vulnerability type. The severity level of a vulnerability will be
determined by the vulnerability type, and probability.
* (If SQL is Used) Perform both standard SQL Injection, and Blind SQL Injection tests as outlined by http://www.spidynamics.com/whitepapers/Blind_SQLInjection.pdf and http://www.securiteam.com/securityreviews/5DP0N1P76E.html. If SQL
Injection is present file this as a severity 1 issue.
Login
* For each user input perform common security related input validation tests. See The Web
Application Security Consortium's Threat Classification
for a list of common input vulnerability types. For each input perform
each vulnerability type. The severity level of a vulnerability will be
determined by the vulnerability type, and probability.
* (If SQL is Used) Perform both standard SQL Injection, and Blind SQL Injection tests as outlined by http://www.spidynamics.com/whitepapers/Blind_SQLInjection.pdf and http://www.securiteam.com/securityreviews/5DP0N1P76E.html. If SQL
Injection is present file this as a severity 1 issue.
Buying Items
* Ensure that the user is unable to modify the price for a given item. Ensure that the price is not exposed in a web form,
cookie, query string, or POST data. If the price is exposed through one of these vectors ensure that if changed, the
application detects the modification on the server side and refuses to sell
the item for anything other than the stated price.
* For each user input perform common security related input validation tests. See The Web
Application Security Consortium's Threat Classification for a list of
common input vulnerability types. For each input perform each vulnerability type.
Search Engine
* For each user input perform common security related input validation tests. See The Web
Application Security Consortium's Threat Classification for a list of
common input vulnerability types. For each input perform each vulnerability type.
* (If user text is echo'd back) Test for Cross site scripting vulnerabilities. If discovered file a severity 2 issue.
Logical Tests
Authentication
* Disallow the user from directly accessing a page requiring the user
to be logged in. If you are able to access a page requiring
authentication
and are currently not logged in, file a security ticket of severity 2-3
depending on the page.
Login
* If the username or password is incorrect do not display a message
such as 'Your Username is Incorrect'. Instead ensure that a generic
message
stating 'Invalid Credentials' is displayed so as not to leak
information about existing user accounts. If either the username or
password is specifically mentioned file as a severity 3 issue.
Email Confirmation
* When emailing the order confirmation do not leak sensitive
information such as the full credit card number, social security
number, full name and address, or other similar sensitive information.
If the card number or social security referenced in the email displays
more than the last 4 digits (Unless otherwise defined) file a severity
2 issue.
Password Reset
* Ensure that if the password is reset that the temporary password
expires after a reasonable amount of time as defined by the product
manager (default is 48 hours). After 48 hours the temporary password
will expire and will need to be regenerated.
Filing A Security Ticket
I won't get into false negatives and false positives in this document as it is an entire document in itself, but will mention that it's important to identify the validity and probability of a security defect. Some vulnerabilities only become apparent when a non standard configuration is used and it's important to properly identify the severity of these sorts of issues with your development team. If you start flooding them with unlikely scenario's and unreasonable severity levels they will start ignoring you just like with any other bug that you may file. It is also important not to shrug off something that you feel is non standard, communicating with customer support is an important step which should be performed to identify common customer configuration and deployment scenarios.
Regression Testing Security Issues
Just like with any other bug, just because it gets fixed doesn't mean it isn't going to make a come back. As part of your security test plan it is important to set aside an additional section labeled 'Security Regression Testing' which provides a list of all previous security issues. Search your bug tracking system (if you don't have one, you have bigger issues) for security related defects and during each development cycle (this will vary from project to project and may be every X cycles) retest (regression) them. This assumes of course that you have a bug naming convention, and have somehow flagged it with 'security' or other identifier. Like other test cases try to automate regression of these issues as often as possible to reduce the manual testing burden.
Conclusions
For many companies hiring a dedicated security team or engineer isn't possible for one reason or another. Ensuring that your product or site is as safe as possible is something that can be accomplished with careful development cycle planning. This article outlines how the QA portion of a code's cycle can have security implemented into its everyday process, however these checkpoints should be something scoped out within each phase of the remaining development cycle.