Black box testing is a technique involving examination of the system in the capacity of an outsider, with the help of the tools used in detecting the attack surface and probing the system for internal information/data. Having no access to internal knowledge of the system, the tester typically builds an understanding of the system. Information leakage is especially significant to the black box testing technique as it enables a tester to generate more understanding of the system than otherwise obtained by manipulating a leak-free program.
Generally, black box testing lets the tester probe all of the attack surfaces and generate test data for functionality that may not be in the design. It helps protect against many common mistakes such as not removing debug-only commands before production, throwing in last-minute functionality that is not properly documented in the design etc. Such flaws, being difficult to otherwise notice through white box testing, are brought to notice with the help of black box testing.
Black box testing is primarily used in cases where a system is using security by obscurity to ensure the protection of data, as is often the case in digital rights management (DRM) systems. At times, the only successful defence against DRM attacks is to increase the security, which is ensured by Black Box tests. Notwithstanding this. black box testing by a skilled reverse-engineering team is also used to test the strength of the obscurity used, though this might require expertise outside the norm for a quality assurance team and can, at times, also be expensive.
In black-box testing, access to the source code is not necessary, although it will help in improving the tests. Black-box testing is built on the responses to a set of inputs fed to the software through selected interfaces, such as-
- User interfaces: GUI, command line;
- Network protocols;
- Data structures such as files;
- System APIs such as system calls and device drivers.
Purposes of Black Box Testing
Some prominent purposes of black box testing are as follows-
- Interoperability testing-Interoperability is basically a subset of conformance testing. Interoperability testing is a practical task in which the final product or its prototype is tried against other industry products.
- Conformance testing-Conformance testing, a category of black-box testing, is a process that aims at validating the conformity of the product against the specifications.
- Performance testing-The third type of testing, performance testing, comes from real-use scenarios of the software. When the software works according to the feature set and with other vendors’ products, testers need to assess if the software is efficient enough in real-life use scenarios. There are different categories of tests for this purpose, including stress testing, performance testing, and load testing.
- Robustness testing-another major black-box testing category is robustness testing or negative testing. From the security perspective, this category is often referred to as the most important category. In negative testing, the software is tortured with semi-valid requests to assess the reliability of the software.
Security Techniques for Black Box Testing
1. Load Testing
The easiest and best-known attack to QA people includes various DoS (Denial of Service) situations. The majority of DoS attacks are based on load. In load testing, the performance limitations of the system are tested with fast repetition of a test case and by running several tests in parallel.
2. Stress Testing
A stress test will change the operational environment for the SUT by restricting access to required resources. Examples of changes include-
- Size and speed of available memory;
- Size and speed of available disk;
- The number of processors available and the processing speed of the processors;
- Environmental variables etc.
Most often stress tests are executed in a test automation framework that will enable you to run the SUT inside a controlled environment such as a sandbox or software development simulation.
3. Unit Testing
In unit testing, the SUT is a module used inside the actual application. The real application logic can be bypassed by implementing parts of the functionality in prototypes when the real implementation is still unavailable or by other replacement implementations when the target is actually a library such as a file parser or a codec.
4. Fault Injection
Traditionally, the term fault injection has meant a hardware testing technique in which artificial faults are introduced into printed circuit boards. The purpose is to test the sensitiveness of the hardware for faults (fault tolerance) emerging during manufacturing or product lifetime. Fault injection can be used to forecast the behaviour of hardware during operations or to guide efforts in making the hardware more robust against flaws.
5. Syntax Testing
The primary objective of syntax testing is verification of the fact that the system does some form of input validation on the critical interfaces. Several different kinds of anomalies (or errors) can be produced in syntax testing:
- Syntax errors: Syntax errors violate the grammar of the underlying language
- Delimiter errors: Delimiters mark the separation of fields in a sentence. Delimiters can be omitted, repeated, multiplied, or replaced by other unusual characters. Paired delimiters, such as braces, can be left unbalanced. Wrong unexpected delimiters can be added at places where they might not be expected.
- Field-value errors: A field-value error is an illegal field in a sentence. Normally, a field value has a range or many disjoint ranges of allowable values. Field-value errors can test for boundary-value errors with both numeric and non-numeric elements.
- Context-dependent errors: A context-dependent error violates some property of a sentence that cannot, in practice, be described by the context-free grammar.
- State dependency error: Not all sentences are acceptable in every possible state of a software component. A state dependency error is, for example, a correct sentence during an incorrect state.
6. Negative Testing
The most common type of negative testing is defining negative tests as use cases—for example, if a feature implements an authentication functionality, a positive test would consist of trying the valid username and valid password. Everything else is negative testing, including wrong username, wrong password, someone else’s password, and so on. The purpose is to try only negative tests and not to care about the responses from the SUT at all.
Robustness testing consists of the following steps, which are very similar to the steps in syntax testing:
- Identify the interface language.
- Define the syntax of the informal language.
- Create a model by augmenting the protocol with semantics such as dynamically changing values.
- Validate that the syntax (the protocol and the resulting model) is complete, consistent and satisfies the intended semantics.
7. Regression Testing
Post-release testing is also known as regression testing. Regression testing needs to be very automated and fast. The tests also need to be very stable and configurable.
This article discusses testing approaches that could be useful to you when integrating security testing into a standard quality assurance process. The aim of black box testing is not to prove that the software is secure but to break software by whatever means available. Follow the above steps for efficient security testing of your products.