Automation Tool Selection Dilemma

By |2019-11-20T20:59:10+05:00June 29th, 2017|Tool selection|

After deciding what to automate, we move on to looking for what tools to use. With each passing day, this becomes a more complex decision to make. The application’s technology stack becomes more complex, the development process keeps evolving, client’s environment keeps changing and lastly a hell lot of vendors and automation options to make things more confusing (many options is also helpful, there is a bright side to everything!).

I think the tool selection process will keep on evolving, and would vary a lot according to the situation. Keeping the important aspects in mind, here are a few tips discussed by Katrina at the Automation Guild Conference 2017, along with my reflection on topic with some pointers from my talk at the TestTalks podcast.

Cost

THIS IS NOT IMPORTANT! Most of you were not expecting this I take it. but it’s true, unless it’s for some tools sitting on the outlier of the spectrum (with a 10 grand yearly license, which I might not recommend for myself anyway).

Now why is cost not important? Simply because for most tools the cost is equivalent to just one to a few months’ worth salary of the team. It does not make sense, for instance, to opt for Selenium and spend 6 months learning it, which could have taken a month or so with a vendor based tool. Apparently ‘free’ selenium, in this case, might cost far more than the vendor based tool. And this is just the upfront cost. Maintenance is a whole new issue.

The takeaway, please don’t start counting dollars right from the outset, the math does not add up for me at least (instead of dollars you can count stars !).

AUT

Specially in what technology is the front end developed in. Normally automation starts from the UI, you don’t want to try automating an Angular 2.0 front end with a primitive tool (It can be done, but so can every one of us land on the moon!). Also down the line hopefully the team will move on to integration / API level and Unit level tests, will the tool of choice support these / integrate with other tools handling it?

Most tools and frameworks can be scaled reasonably, it’s still a good idea to find out how much scaling would you need. Estimating number of scripts the team might create EVER, yes (I calculated 5000 scenarios / test cases for one project). It’s good to see if your tool can handle this at the outset, I have seen tools / frameworks crashing due to memory leaks and so on. With great ‘projects’ comes great ‘problems’ (Uncle Ben never thought of this one).

Support

A great phrase I learned from Greg Pascal I have to mention here, “Use the tool as intended, not as discovered”. As engineers I think that comes as a byproduct, mostly we are not overly fond of going through documentation for anything. We just fiddle, especially with software, and figure out what the application does. Perhaps some quick googling at the most, does not work everywhere. If there is no support for the tool, guess what, most probably it will be used as discovered, not as intended.

Vendor based tools have a great edge here. Some vendors have excellent support, great documentation for the tool, plus they are constantly in the process of upgrading their tool’s support for latest browser revisions and so on.

Look for a great community around the tool. A lot of problems get resolved that way, plus saves us from re-inventing the wheel for every solitary problem we face.

Another great point made by Katrina was to look for experts using that tool. If automation experts with great skill sets are working / have worked with the tool in question, might be a safe bet.

Longevity

For vendor based tools, are they in the industry for some time? More importantly are they upgrading themselves according to latest trends? Front end of Web applications is changing very fast. JavaScript based frameworks make life a lot easier for developers (quite hard for automation folks too!) forcing new applications to change. If a vendor cannot keep up with that, it’s going to be a bad choice.

For open source tools, there are a lot of frameworks out there, not all of them would keep up with the pace of change. Not a great idea to use just JUnit to automate a Sencha (Ext JS) based application. Not that it cannot be done, would take a lot more effort perhaps.

Integration

I never thought of this as a serious point to consider, since the last few weeks, now I feel otherwise. Firstly, being able to use the same IDE and language as development can prove to be a great plus. I know automation in C# is not ideal compared to Java / JavaScript, but might be worth considering specially if automation is a one / two-person team. Plus, it would work a long way towards developing trust and support from development.

Secondly, inevitably the development process of your company will change according to the new trends. If your automation framework cannot integrate with latest industry CI tools, virtualization offerings, development process tools, you might be looking for a big change down the line. I would recommend to seriously consider the CI flow for your application and make sure whatever you choose can be scaled up to be part of a continuous integration setup.

Features

In most cases, one can compensate for features lacking in a tool by using external libraries or improvising. Some aspects I would not try to compromise on. Features around reporting and log generation are extremely helpful. I think there is hardly any one reporting format that covers it all, therefore the tool’s reporting should be highly customizable. Great to have some things already like debug information and call stack, makes life easier.

Configuration with source control tools I found is also helpful. A team MUST use a source control, and if project specific files do not get merged easily, brings everyone a lot of pain.

Throw in other features you would love to see. for instance, I hate bad editors (IntelliJ is a benchmark for me, off course most vendor tools are not close).

Analyze

Crunch in the data from all the aspects above (without the tool’s cost, get over it already, that’s peanuts in the long run). Analyze in detail the top 3, best to have POC’s for all of them. Keep ‘What to automate‘ in mind and choose what suits that best.

A word of caution, do not confuse framework development with tool selection. No matter what tool(s) you use, developing a framework around it is a separate chapter. More on that some other time.