Regression testing is the process of re-testing software that has been modified. Regression testing comprises of a great majority of testing energy in the development of software and is a crucial part of any viable software development process. Large components or systems tend to have large regression test sets. In practice, regression testing is often incorporated into a continuous integration service.
This article explains what kinds of tests should be in a regression test set, which regression tests to run and how to respond to regression tests that fail.
Identifying and Running Tests in Regression Testing
Generally, a test engineer faces an issue in determining which tests to include in a regression test set. Including every test set possible results in a regression test set that is too large to manage. The result is that the test set cannot be run as often as changes are made to the software. For traditional development processes, this period often amounts to a day; regression tests run at night to evaluate software changed that day, with developers reviewing the results the following morning. If the regression tests do not finish in a timely manner, the development process is disrupted. It is well worth throwing money at this problem in terms of additional computational resources to execute the tests, but, at some point, the marginal advantage of adding a given test is not worth the marginal expenditure of the resources needed to execute it. On the other side, a set that is too small will not cover the functionality of the software sufficiently well, and too many faults will be passed on to the users. It is also possible is to restructure tests with an eye to efficient execution.
Thus, in order to avoid the aforementioned problems, a regression test set has to be the right size. Some organizations have a policy that for each problem report that has come in from the field, a regression test must exist that, in principle, detects the problem. The idea is that customers are more willing to be saddled with new problems than with the same problem over and over. The above approach supports traceability because each test chosen in this way has a concrete rationale. For example, if node coverage in the form of method call coverage shows that some methods are never invoked, then it is a good idea to either decide that the method is dead code with a respect that particular application, or include a test that results in a call to the method.
Responding to Regression Test Failures
If one or more regression tests fail, the first step is to determine if the change to the software is faulty, or if the regression test set itself is broken. In either case, additional work is required. If no regression tests fail, there is still work to do. The reason is that a regression test set that is satisfactory for a given version of the software is not necessarily satisfactory for a subsequent version.
Changes to software are often classified as corrective (a defect is corrected), perfective (some quality aspect of the software is improved without changing its behaviour), adaptive (the software is changed to work in a different environment), and preventive (the software is changed to avoid future problems without changing its behaviour). All of these changes require regression testing. Even when the (desired) external functionality of the software does not change, the regression test set still needs to be reanalyzed to see if it is adequate. For example, preventive maintenance may result in a wholesale internal restructuring of some components. If the criteria used to select the original regression tests were derived from the structure of the implementation, then it is unlikely that the test set will adequately cover the new implementation.
Evolving a regression test set as the associated software changes is a challenge. Changes to the external interface are particularly painful since such a change can cause all tests to fail. For example, suppose that a particular input moves from one drop-down menu to another. The result is that the capture/replay aspect of executing each test case needs an update. Or suppose that the new version of the software generates an additional output. All of the expected results are now out of date and need to be augmented. Clearly, automated support for maintaining test sets is just as crucial as automated support for generating and executing the tests.
A different approach to limiting the amount of time needed to execute regression tests is selecting only a subset of the regression tests. For example, if the execution of a given test case does not visit anything modified, then the test case has to perform the same both before and after the modification and hence, can be safely omitted. Selection techniques include linear equations, symbolic execution, path analysis, data flow analysis, program dependence graphs, system dependence graphs, modification analysis, firewall definition, cluster identification, slicing, graph walks, and modified entity analysis.
It is worth emphasizing that regression tests must be automated. Indeed, it could be said that unautomated regression testing is equivalent to no regression testing. A wide variety of commercially available tools are available, some of which include JUnit and its cousins such as Selenium, NUnit, HTMLUnit, CUnit, CPPUnit, SUnit, TestNG, PHPUnit, etc.
How to accelerate your product hits the market early and on time by outsourcing software testing? Solve all your software testing needs !! Contact us.