Managing an automation project is overlooked at times, I feel perhaps, owing to the technicality of it, undermining some essential fundamentals is a mistake all to common. @Katrina shared a beautiful presentation titled Automation and Management at the #AutomationGuild conference 2017. There was a whole lot content, along with my reflection on the topic for one post. Follow up posts will cover the complete session. In this post, summarizing my leanings from her talk and my thoughts around deciding what to automate.
Drifting along without a concrete decision making criteria on what to automate will not render optimal utilization of automation efforts. Automating everything is not a good idea either, so how to “Deciding what to automate” that will fit the bill?
Understand your product’s strategy
How is the product being pitched in the market / the competitive advantage? What features are important to customers and the business, giving an idea of what’s important in the overall strategy.
Is the product in its growth stage? If so lots of changes are in store, for products in the maturity phase of the product’s life cycle, drastic changes might not be expected. What areas are planned for changing and what does the release plan look like.
The best phrase I learned here was “Testing is reputation assurance, to protect corporate image”. Automation can provide a big boost to build that credibility. To do that, find out what application features can be marked as high risk. She recommended a great article by @James Back on the topic Heuristic Risk-Based Testing to identify risk areas and make them a priority in automation.
Goals of your automation
The usual goals are increase test coverage and reduce time to market. However each product / company has their own unique challenges as well. Take them into consideration as well.
Few places where I kicked off automation, they had no formal test scenarios. Having formal test scenarios would have a great benefit to them, which we added as a goal in our automation effort.
Further goals should be quantifiable. The analogy given was ‘you should be able to add it to a check list and check it off once done’. Everyone mentions SMART goals, but creating one is not easy, not impossible either.
One of the best practices in automation has always been to make code reusable. Take that to higher layer of abstraction, at the requirements layer. Serenity, a tool which helps with that, I heard for the first time at the conference can help. Behavior Driven Development also allows repetition to some extend, however is not designed for that purpose.
Could be done without a tool or framework also, as done in one of the automation projects I worked on. Functional points were automated as independent units, making them reusable to construct larger scenarios. The thought process has to be there, you can then mold any tool / framework in use to achieve that.
When should a test be automated? I had not thought about this before. Katrina quoted a great piece of research from Brian Merik “When should a test be automated” published in 1998. Some ideas from back then are now mainstream, but his answer on ‘what to automate’ is neat. One of his points is “The cost of automating a test is best measured by the number of manual tests it prevents you from running and the bugs it will therefore cause you to miss”. most veterans of the industry might have thought of this earlier, but the words make it all clearer. If a certain feature is hardly used, for instance where manual testing for the feature is scheduled for just few hours per year, does not make sense to automate (that’s an extreme case, I know, using an outlier example to make a point!)
Also it is a good idea is to define how much time to spend automating (which is 99% of the time not going to be accurate). Having a ball park figure helps with prioritizing activities which are more important. Further, limits on execution time is a great requirement to have at the outset. Will prove to be considerable help when moving towards continuous integration
The only constant thing is change. Have an idea of how the above might change, that affects your decision making criteria. This is certainly hard to extrapolate; if broken down, one might have a good chance at figuring it out with reasonable probability.
Companies go through change due to external and internal environments. The external is further categorized according to amount of influence from different entities, these range from direct customers and suppliers to government laws and countries trade agreements. Companies operating in disrupting industries, or ones having a tricky market segment, are more prone to external environmental change. Organizations in stable markets with comfortable market segments, tend to be less affected by external factors. Internal environment I feel is relatively simple with few ingredients like, higher management’s mix, organizational structure, team’s skill set and future direction at the moment.
I have to admit though, C-level management is all about finding what’s changing and to navigate the ship accordingly, and it’s not an easy job. At the level of detail needed for automation, it can be managed.
When you crunch in the numbers, add folks from all teams to derive conclusions to get the right answer. Everyone has a different insight. No one team can have the whole picture. To get the best value for time spent, get opinions from across the board.
A great tip here was, “Allow yourself to be influenced by others”. Until others feel you can be influenced by their opinion, you might not get the best suggestions.
Team’s level of expertise
Although this should not be considered at the outset, but something to consider towards the end. I have seen this point coming way up the list, my rationale for bringing it down here is, as quoted from my favorite book “How google works”, “the internet has rendered resources and computation power almost limitless”. If you see a clear benefit in setting the scope as, for instance, UI, integration and unit tests with 80% code coverage, while the current QA team cannot handle that, it’s not hard to find resources to get it done.
When bringing current expertise level at the beginning, decisions made will have that bias in them unwillingly. However, if circumstance do not permit augmenting required expertise, then current skill level is a good idea to consider.
Anything else you would consider while deciding what to automate?