Have you ever reached release time for your product only to find out major bugs coming out of testing? Does this sound like what you face on a routine basis? Trying to drill into finding the root cause for this many times ended up inconclusive, there was no one practice or person to be held responsible at the end. This is where continuous testing comes in to save the day.
This is a huge topic and one can easily write a book on this. This post along with this video attempts to highlight probably the very basics I feel are most important.
The bug bottle neck
I call the last minute barrage of bugs ‘the bug bottleneck’. To some it seems the only obstacle coming in between releasing a new product version are these testers coming up with last minute bugs. Why were these issues not found earlier and had to be reported a day before release.
The bug bottleneck develops not over a few days, it’s a culmination of issues which could have been prevented from the very first sprint cycle. More complex and huge the product is, the more difficult it becomes to test it adequately causing more undetected bugs. Here is a typical sprint cycle with manual testing at the end:
Dev team implements the new feature for the sprint with some fixes. This introduces new bugs some being directly related to implementation of the new feature and some ripple effects due to the new feature / fixes. The testers should now ideally test all the new features with regression around each one of them in detail. However usually there is not enough time for them to test the feature AND do regression around them during a sprint, especially if this is a complex application. Bugs do slip from this stage and keep piling up. A few sprints down the line nearing release dates, there is an army of undetected bugs which start getting caught as soon as the regression cycle hits which is then too late.
What is continuous testing?
In our example, testers were not able to cover enough ground from the very first sprint cycle causing the melt down at the end. Adding more testers is not going to be a sustainable solution either, especially for large applications. The solution is automating as many tests possible and running them as part of the sprint process.
Continuous testing is defined in many ways. I understand it to be a process with testing ingrained into every step of the sprint including development, and keep evolving the tests based on feedback of what works and what does not. Testing is therefore a ‘continuous’ process and not just an activity done when code is pushed to QA.
It would start from running automation during development and testing. As soon as a developer checks in his / her code, tests for at least the changed areas of the application are executed giving the developer an immediate feedback if anything fundamental is not working. Similarly when testing kicks in, automation should run a regression on the application while manual testers can focus purely on understanding and testing the new features.
Why Continuous Testing
This is the next step in the software development processes evolution. The last big change was Agile which replaced waterfall. There were some forms of agile practices already in place like iterative models, however agile methodology was classified as a different species.
We have reached the next step of evolution here, and it’s for the same purpose agile came into being, reduce time to market and increase quality. Like the iterative model, the fundamentals to this next step of evolution were hidden in agile but not prominent and were being neglected, necessitating the next step. This next step is known by many names including continuous integration, continuous delivery, DevOps, continuous testing. All of them have different meaning and implementation, but work towards the same purpose, reduce time to market and increase product quality. To truly have a DevOps model, one would need continuous delivery, needing continuous testing needing continuous integration at the end creating one system interconnected for the same purpose.
Back to our example, with continuous testing using automation in place, our sprint would look something like this:
Issues being captured during development coupled with a higher ratio during testing cycle lead to a smaller number of bugs getting slipped passed our net. Come regression week (s), we will find far less issues during that time and hopefully no major ones.
Requirements for Continuous Testing
There can be a very long list here, the few I found important for one of the products I worked on are discussed briefly.
Increase test coverage
Unless we have a decent coverage of the application, the purpose of capturing bugs from code changes will not be accomplished. Today’s applications are far more complex which cannot be automated just from the UI. Careful planning and ingenuity is needed to have a decent coverage.
Make results meaningful
Tests a developer would like to see on adding a feature / updating code should be there. Yes, unit tests would be reflecting that, but also API and some UI level would be helpful too. Similarly for the tester, an automation batch run should be augmenting a tester’s wish list of areas to test. Therefore feedback and contribution from both these groups is vital.
Secondly the results should come in time. If we send the developer a report for 2 major bugs in a feature he / she implemented a week later, that’s of no use. The result should be there in hours while the implementation details are fresh in mind. Also for the tester, they should base their tests looking also at the automation report. For instance, “Areas A, B and C passed, I can test a slight variation there and that would be enough. D and E failed, probably there are more issues around this. “. Unless the report is delivered BEFORE the manual tester starts planning his test, it’s going to be late.
Test before check in
Before a piece of code is checked in, it should be tested through the automated scripts and added to the branch only when passing. This would be easier than done since it would require a lot of pre-requisites. Continuous integration portions implemented, automated tests running efficiently and processes institutionalizing the branching process. however once done, the benefit would be far beyond the cost.
That was the whole idea right? To test quickly with quality. If a tester is able to utilize the automation results as mentioned, it can help reduce the testing time per issue, allowing to test more areas and test quickly. This freed up time should be used to the advantage of shrinking the test cycle and increase time to market.
Word of caution here, by no measure this means to cut regression in half (or the team for that matter), rather increase the number of releases per year and have a quick turn around.
To get the desired results, all departments need to coordinate with one another sharing what worked and what can be improved. This would involve not only development and QA, but support, sales and other departments giving information on how the updates went with the customers and what next to be prioritized, what areas need more automated tests around, what should be included in manual tests and so on.
Continuous testing is going to be an ongoing journey, not a destination. For it to be continuous, it necessitates evolution and continuous improvement. Being an emerging concept and mostly early adopters in the arena having it fully implemented, I feel a lot of evolution is in store around implementing this idea and how to do it well. One thing is certain, teams, products and companies to adopt this early on will be the most successful.