Irrespective of how well conceived and tested a program is before being released, a software eventually has to be improved so as to fix bugs or respond to changes in user specifications. In simple terms, regression testing is retesting of previously working software after a change to ensure that unchanged software is still functioning as before the change.
This article discusses how to guide the planning and assessment of regression testing activities in large, complex reuse-based systems, when and how to execute regression tests and challenges that can occur to regression tests.
What is Regression Testing?
According to IEEE, regression testing is “Selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or components still complies with its specified requirements.”
The goal of the regression testing may differ between different organizations or parts of an organization. The goal may be either to reveal defects or to obtain a measure of the quality of the system. Traditionally regression testing is iteratively applied to the consecutive revisions of the software, where revisions refer to the variation in time of the evolving system.
Importance of Regression Testing
Regression testing is an important software maintenance activity that involves repeatedly running a test suite whenever the program under test and/or the program’s execution environment changes. By performing a regression test suite on the introduction of a new feature of a defect mix, a tester can ensure that the modification of the program is such that it does not negatively impact the overall correctness.
Efficient regression testing is vital, especially for organizations incurring huge costs in software development. It incorporates, inter alia, determination of test cases required to be re-executed (called as regression test selection) so as to verify the behaviour of the test which has been modified. The prime objective of regression testing analysis is to make sure that the changes that have been made to the software(s) have not resulted in an adverse effect on the essential aspects of the software. It must be conducted to confirm that recent program changes have not adversely affected existing features and new tests must be conducted to test new features.
However, regression testing can also be prohibitively expensive, particularly with respect to time, and thus, accounts for as much as half the cost of software maintenance. Some good practices in regression testing are-
- Run automated daily tests on module level.
- Focus automation below user interface.
- Visualize progress monitoring.
These practices are not specific to regression testing. The latter item is not specific testing at all but is a management practice that becomes critical to regression testing as it constitutes a key part of the development project progress.
When and How to perform Regression tests?
Regression testing is performed at various different levels which include- module level, component level and system level and at different stages of the development process. Some organizations regression test as early as possible while others regression test as late as possible in the process, and for some, regression testing is continuously carried out throughout the whole development process. The purpose may be slightly different for the three options; early regression test to enable early detection of defects, and late regression testing for certification or type approval purposes. As far as frequency is concerned, some organizations regression test daily while others regression test at each software integration, at each milestone, or before releases. The most common approach is, however, to regression test before releases.
Further, the tests that are utilized for regression testing may be a selection of developer’s tests, a selection of tests from a specific regression test suite, a selection of tester’s tests or new test cases are designed. The most common use is, however, to reuse test cases designed by testers. Some other ways to perform regression tests include complete retest, combine static and dynamic selection, a complete retest of safety-critical parts, select test cases concentrating on changes and possible side effects, ad-hoc selection, smoke test, and focus on functional test cases. A project may include several different regression testing activities, including manual and automatic regression testing techniques.
Regression testing is a frequent and time-consuming test activity in most industrial software projects. It is applied in many different contexts, not based on research but based on experience. It ensures that nothing has been affected or destroyed, and give an answer to whether the software has achieved the desired functionality, quality and stability etc.
Repeating all previous test cases in regression testing after each minor software revision or patch is often impossible due to the pressure of time and budget constraints, but omitting test cases used in regression testing is also risky. Studies indicate that 80% of the testing cost is regression testing and more than 50% of software maintenance cost is related to other testing processes. Some common challenges to regression testing include-
- Test case selection- It is hard to assess the impact of changes on existing code and to make a good selection as well as to prioritize test cases with respect to product risks and fault detection ability, and to be confident in not missing safety critical faults. Determining the required number of tests is also an issue, and it is hard to assess the test coverage.
- Test case design- Lack of time and resources for regression testing is also another issue in regression testing.
- Manual and Automated regression testing- Automating regression testing is time and resource consuming process, resulting in costs to the enterprise for implementing the automated tests that need to be considered, along with costs for maintaining the test suites and manual analysis of results.
- Design issues also play a crucial role in regression testing since a strong correlation exists between the effort needed for regression testing and software design. Design for testability, including modularization with well-defined and observable interfaces, helps to verify modules and their impact on the system. This could be addressed by including testability in design guidelines.
Regression testing increases in software projects as software becomes more and more complex with increasing emphasis on systematic reuse and shorter development cycles. Many of the challenges are not specific to regression testing but are general to all testing but have a significant impact on how effective the regression testing could become. Questions involving automated testing are of course particularly important for regression testing, as the same tests are repeated many times. Lastly, having a test-friendly design will go a long way in performing selective retesting processes like regression testing.