The image above is of General Motors managers keeping a record of the flow of materials needed for wartime production in 1941. All the updates were done manually. So, your struggle with managing regression testing using spreadsheets is not new; thanks to technology, more help is available.
In spite of the software quality assurance industry adopting automated testing tools in a big way, the majority of the tests are still run manually. This is mostly due to the need to optimize the test effort due to budget constraints, especially in smaller organizations.
Manual testing in an Agile environment is challenging; with changing business needs and shorter release cycles, maintaining regression testing suites, which will most likely comprise manual as well as automated tests, has become crucial.
Here are some of the challenges in managing regression testing without the use of high-priced software.
Challenges in managing regression testing suites
Regression testing determines if the existing features in the build still work well after a round of bug fixes or introduction of new features. At the end of regression testing, the expectation is to be able to understand the test effort and test coverage. In order to run regression testing, the QA team needs a solution to the following issues:
- Shared access to the regression testing suite is a necessity. Since businesses are now leaning towards Agile as the standard development methodology, the importance of real-time communication between team members cannot be stressed enough. To ensure that all the team members are in sync regarding test suite management, they must have shared access to it.
- Setting up business priority for regression testing. Priorities of test cases will most likely change in subsequent test runs. A test that has been passing in previous builds can be assigned low priority in the current build, while a test that has failed previously will be marked as high priority. This is secondary to setting the business priority for the test cases, which will ultimately define the composition of the critical path, besides regression testing.
- Test suite should be easy to understand for new team members. The regression testing suite should be self-explanatory, reducing the learning curve.
- Updating the regression testing suite and the critical path is an on-going effort. As more test cases are added with successive builds, the regression testing suite gets ‘heavier’. Managing such a large number of tests and ensuring that the critical path is updated simultaneously is crucial. Also, keeping the test results updated on a regular basis is a challenge, especially with distributed teams.
- Time estimates also have to be maintained for test case runs so that the time for single regression testing run can be determined. This is critical, since there is always a time constraint for regression testing, and emphasis on test planning is important.
- Support for multiple environments. A record of the number of executed, pending and skipped test cases has to be maintained meticulously for multiple environments. Add to this the time estimates for each category, and you have your hands full with maintaining your regression testing suite.
- Collation of results of manual and automated tests. Automated tests are a crucial part of regression testing. However, manual testing will always be integral to product development. There is a need to combine the results of both manual and automated tests in a single report for clearer understanding.
- Duplication of efforts between critical path and regression testing. Critical path is a subset of regression testing; however, they can be executed exclusive of each other. Still, some organizations maintain the critical path in a separate sheet; if the critical tests are not marked as such in the regression testing suite, it is possible that those tests will be duplicated during regression testing, causing confusion in the test results. Avoiding this duplication of test effort is a major challenge for the QA team.
Let’s see how cloud-based spreadsheets can overcome the challenges mentioned above.
Using cloud-based spreadsheets
Working in the cloud is all about collaboration. This single feature can help solve many of the concerns mentioned in this article.
- Shared access to the regression testing suite is easy. Through collaboration, cloud-based spreadsheets facilitate sharing of the regression testing suite among the team, along with the revision history. It still provides access control too; document owners can allow others to view or even edit the spreadsheet.
- Setting up business priority for regression testing is easy — You can easily create a copy of the current regression testing suite for the new build and then make the required changes there, such as updating the business priority of test cases.
- Updating the regression testing suite and the critical path is made easy via collaboration — Being cloud-based, it makes maintaining and updating the test suite easier - just the thing a distributed team needs to stay in sync!. Even if the priority for a test case is changed in the subsequent build, you can retain the last reason for failure if any, in the form of a note for tracking. You can even insert a link to the relevant ticket in Jira in the note, for that particular test case.
Updating a cloud-based spreadsheet is easy - Support for multiple environments. Though the test cases will be the same for different environments, the results will be different and these can be recorded in separate columns in the sheet. Time estimates for each environment can also be maintained separately.
- Collation of results of manual and automated tests. Collation of the test results is very much feasible by uploading them to the spreadsheets after the completion of the test run. These results can be combined through Google scripting to mark the relevant test case status. When the manual tests are run, the test results are recorded in their relevant cells.
- Helps avoid duplication of efforts between critical path and regression testing. To ensure that the critical path can be run exclusively of the regression testing suite, it should not be maintained as a separate sheet, but as a part of the regression testing suite only. The test cases that are critical should be marked as such in that spreadsheet only. This will help avoid duplication of test case runs and the critical path can be easily customized per build.
- Capacity to insert video links. You can insert links to videos of the test case runs, in the sheet itself. These video links can be accessed from other sheets that contain a particular cell reference. Regression testing is faster this way than test suite execution using a conventional spreadsheet.
- Facility to insert video links. You can insert links to videos of the test case runs, in the sheet itself. These video links can be accessed from other sheets that contain a particular cell reference. Regression testing is faster this way than test suite execution using a conventional spreadsheet.
- Provide dashboard views for regression testing metrics using pivot tables, formulas, etc. Using the formulas and pivot tables features in excel, we can provide more comprehensive metrics reports to the key stakeholders in real-time (thanks to cloud based excel solutions). This is an easier way for the resources and managers to consolidate information quickly and accurately. The respective charts also provide visual aspects of the regression testing metrics, facilitating decision making.
- Facility to insert multiple format files. As technology progresses, we are seeing multiple formats for visual representation of data. Cloud-based excel solutions provide multiple formats for files such as images, video, and sound. This supports the overall regression process with comprehensive information for issues found.
Alternative solutions
Using spreadsheets is just one of the many solutions available for managing your regression testing suite. There are other test management tools, such as the open source application Testlink, that offer cloud-based features and can help you organize and optimize your test effort.
Conclusion
You can opt for any of the test management tools available nowadays. However, the basic premise remains the same. Since regression testing is crucial for product development and it involves a large number of manual test cases, it becomes important that it be managed thoroughly - with precision and ease.