In the early months of learning automation, I realized having some sort of custom built code running on top of an automation tool is important. Over the years I have been learning the art of architecting the perfect automation framework design, I noticed the term automation framework does not mean the same thing to everyone, and hence this endeavor in clarifying my interpretation of the term.
What does framework mean
The coolest definition I found was from Wikipedia, the first section is worth reading, quoting just a few sentences to make a point. A framework is:
- ‘an abstraction providing generic functionality’,
- ‘Software frameworks may include
- support programs, compilers, code libraries, tool sets, and APIs
- that bring together all the different components
- to enable development of a project or system’.
Seems a framework is not just one tool, program or library. Rather a group of them joined together to create software. Therefore, for each software product / project a framework would include everything from libraries used to develop the code till peripheral tools such as compilers, source control and so on.
Automation framework used loosely
I’ve seen folks refer to a specific tool, library or a dependency as a framework. In the category of tools for automation, there is a long list of ‘types’ of these from:
- Vendor based
- UI automation ‘tools’ (UFT, TestComplete)
- Supporting tools like Soap UI / PostMan (API)
- Open source libraries / dependencies like Selenium, TestNG
- Open source (GUI) tools like Jenkins
- Code based tools like Maven, NPM
All these are often jumbled up into words like tools, libraries, dependencies and frameworks which are all used interchangeably. While the intent here is not to classify what jargon is suitable for what type, the use of framework for any one of them is not accurate IMHO. Of all the tools / libraries mentioned, none can ‘individually’ be called an automation framework.
An Automation Framework, a customized architecture
Most often, any off the shelf / generic library cannot be used without any tweaks / augmenting it’s functionality. To support automation of any AUT, we do need to add more functionality around the tool’s functions (watch this video in case you are not doing that right now) or use multiple tools / libraries to achieve a more complex task performing lots of steps with one call.
We create an abstract / high-level architecture which utilizes a lot of tools / libraries at the same time to facilitate creation and execution of automated checks. This ‘framework’ is managed by the automation engineer with a library of its own, utilizing other libraries / tools methods where needed. Our tests use these framework functions instead of the tools functions directly. For instance, the framework might recognize an object using one library, write a value using another and report the findings using a third tool.
More from Wikipedia on this, ‘A framework’s control flow is dictated by the framework itself, not the caller’. As in frameworks we design for our products, they usually dictate how the checks / tests will be written and the results generated.
Are there any Off the shelf Frameworks?
While a single tool cannot be called a framework, there are off the shelf frameworks we know of used in software development. The well-known MFC and .Net ‘application frameworks’ are an example. These are more than just a library. Along with giving a librar(ies) to use, they have design patterns of their own, enforce some programming practices wen using the framework,
The topic does get debatable at this point, but I like to distinguish an automation framework and application frameworks based on intent and usage of them. Consider comparing .Net with Selenium / Jasmine / Jenkins or any of the other famous tools / libraries used for automation. Is the usage similar in both cases? Dot Net is a world of its own which has its own design patterns recommended and integrates up and down the stack with other technologies. Dot Net is not just one library, but allows support for a whole technology stack summed up into one ‘framework’. The automation tools we just mentioned cater to specific needs only and are driven by the user instead of the tool / library driving the control flow.
Another way to look at it is frameworks are a higher level of abstraction integrating a lot of different dependencies / libraries providing most / all needs for a stable automation framework within one package. There are a lot of efforts being done to accomplish this in an off-the-shelf like products, but most are not ‘widely famous’ at this point.
An automation framework can be characterized by an architecture which can is utilizing multiple tools / libraries to perform various tasks needed to develop, run, report and maintain automation scripts. For the most part at the current stage of evolution in automation, I feel this would mostly be custom made architecture designed to meet automation needs of a specific product.
For further reading, this article explains the ‘pillars’ for automation framework design, principles to consider while designing a framework.