API Testing Archives - Quality Spectrum

Understanding HTTP messages

By Ali Khalid|2019-11-20T20:58:02+05:00November 14th, 2017|API Testing|

In our quest to understand API testing, the first step would be to understand the general protocol used by client and server side of an application to communicate, which in API automation we would try to automate.

This post is summary of the discussion from the video below.

Sending an email is like posting a letter

The goal here would be to translate the concepts with daily life examples to understand client server communication easily. Transmitting of data from client to server is pretty much like sending a letter from one postal address to another, only it takes days instead of milliseconds!

Sending letter vs sending email – Quality Spectrum

Details in the video

The steps in both are very much the same, except the way of doing them. The takeaway here is what the main requirements for sending a letter are the same for what is needed to send an email, sender and receiver details and transmission details needed.

Communication layers

In our example, when Bob sends his mail, the courier company would seal the letter in their envelope along with some of their transportation data like from / To addresses and a tracking number.

All digital communications follow a similar model called the OSI Layers (http://searchnetworking.techtarget.com/definition/OSI) model. The actual message to be sent from sender to receiver is encapsulated with headers and additional transmission information. There can be different message types and header configurations, all explaining how to send and display the message to the receiver.

Constructing an HTTP Request

There are few main areas of an HTTP request, we’ll look into each of them and find out how they help in forming a request.

‘Request Methods’ refers to the different type of HTTP messages used for different purposes. Mostly commonly used are GET and POST messages.

Headers include communication related data, session IDs, client type, body data type are some very common ones. In some cases (e.g. for GET messages) the request need to be made is part of the header. A client side requesting a specific record will add the request in the header of a GET message.

The ‘Body’ includes data to send the server side. A common example would be adding data to the database. In our example when Bob sends the email, the text of his email goes in the body section of the message. There are different formats in which this data can be encoded.

HTTP Request and Response

When Bob sent the email, an ‘HTTP request’ was sent. To confirm the transmission an ‘HTTP response’ will be sent back from the server to the client.

While performing API automation, the tests would construct an HTTP request and assert the corresponding response received. An understanding of the response is therefore critical. In some cases further requests might be needed to confirm if all the data expected is saved and retrieved properly.

What is API Testing

By Ali Khalid|2020-02-02T21:14:32+05:00July 13th, 2017|API Testing|

In the last post we discussed about UI automation and some of its disadvantages. Here we talk about what API level tests are and why to use them.

What are APIs

Application Programmable Interfaces (APIs) is an Interface to help a Program (client side programs like browsers, or any other program) access an Application. This way the main application is placed on a secure server where all the business logic works and connects with rest of the world using an API. To learn more about the basic concepts, this video will help.

APIs in the context of automation

Taking a step back from APIs, let’s illustrate here the different wheels turning behind the scene when a user, fills a form on a website for instance.

From the image above, when we perform UI automation, we are engaging the Application Under Test (AUT) from step 1 through step 8 all the way, hence another name for UI tests, end to end tests (end to end has a contextual meaning, more on that here). Alternatively, we can start our test from step 2 instead, and validate the response at step 7, which is generally called API / integration tests. We will be invoking every process along the control flow except steps 1 and 8. In other terms the ‘client-side’ scripting part is going to be ignored and rest of the application will be tested.

Automation pyramid

As we move from the database level to the browser level (step 4 to 1) automation is going to get more complex for automation and time consuming, with the advantage of testing more technology layers of the AUT. Similarly, in comparing automation from step 1 vs step 2, API tests are going to be less complex and easy to maintain, however we will forego testing certain portions of the AUT. This concept is captured in the phrase ‘Automation Pyramid’ illustrated below.

API level tests

Since we are not testing steps 1 & 8 we will simulate that functionality performed by our AUT to complete the flow of the application necessitating to understand how our client side scripts are generating messages (POST, GET etc.) sent to the server side. Also in the response, what values are we expecting to receive which would translate into the expected output on the browser’s interface.

Perhaps this might seem a bit tricky to work on, for most cases its way easier than it sounds. Developers simulate API calls and responses all the time to complete their development. A lot of efficient tools and methods are available to learn how the API is working without need for detailed documentation or insight from the product group (not to say you shouldn’t talk to the dev team, that would kill the whole purpose of testing. Rather it’s not that hard to understand an API from the outside to begin with).

Uses of API level tests

All the drawbacks we discussed in the previous post work to the advantage of API tests. These tests compared to UI tests take a lot less time to develop and maintain, more robust due to being less susceptible to change, execute way faster and allow the testers to cover a lot test coverage with the same amount of effort as well.

Business logic

Developers while building the application do not go through the whole control flow every time they test what they just coded. Instead the back-end developers working on the server side just test the API level messages being received and sent. Similarly, the front-end developers working on the client side (front end scripting) would also test their piece of code by simulating API messages. This makes testing both areas (front end and back end) less complex and quick.

What’s in all of this for the automation guy? We can break down a complex application’s automation into smaller and easier solutions. For applications which are huge, specially having a lot of business logic, it would be a nightmare to have decent code coverage from the UI level. API calls would make that job so much easier and effective.

Complex UI

Sometimes we run into problems on the UI level where certain elements on the web page are not recognized easily by the tool, or do not always perform the way it’s expected by the tool. These tests generally termed as ‘not automatable’ can be covered from API level tests since we don’t have to deal with the user interface level complexities.

Service virtualization

Many areas of the code need a very specific circumstance to execute which is sometimes very difficult from UI tests. Perhaps there are third party APIs involved, or some external hardware sending in data used by our application, or a piece of code which is scheduled for development in the future needs to be simulated for other teams to complete their feature set. We had a similar problem in one automation project where the server was supposed to receive input data from a hardware device which was still under development.

On the API level this becomes way lot easier. Mostly in such cases the input / output of the API is defined. Lots of tools are out there which can simulate the required calls or responses. It’s then just a matter of constructing the calls properly and validating the expected responses. You might need to create a test harness using other tools / frameworks, but it’s a very powerful concept.

Input for UI tests

Creating pre-test data and / or removing it at the end of the test can becomes a problem. API level tests can be very effective for test data management. Again one or more structured API requests would do the trick in no time. Similarly, for deleting data which is no longer needed should be equally simple (except where business logic is complex enough blocking the removal of a record by simply one call).

Would love to hear what other uses you can think of API tests.

UI Automation Vs API Automation

By Ali Khalid|2020-02-01T22:44:30+05:00July 6th, 2017|API Testing|

Over the past few months our automation team had the feeling we are not covering enough ground in our UI tests compared to the efforts being invested. That’s when we got serious about API testing. This post and consecutive ones build up the case for why to invest in API automation.

User Interface automation (aka Test Automation)

The term ‘test automation’ here is highly inaccurate. The term automation is incorrectly used to describe User Interface (UI) automation, which really is a subset of the many types of automation one can have. Back in the day this was the most prevalent type of automation software testers did. The community has evolved and now we have a few more to choose from. Case in point, automation is no longer synonymous to UI automation.

UI tests till date remain the most common type of automation test engineers love to program. The first word coming to mind when thinking of automation in software testing is Selenium and the likes. Before going into why invest in automation on the API level, let’s look as some drawbacks of this type of automation (despite its wide spread use).

Time intensive development

A UI tests mimics the end user. Therefore, these scripts perform all the tasks a person would do on the front end. That typically involves loading screens / pages, writing / reading a bunch of fields / elements and lots of waits here and there. Scripting all these actions, even with a highly reusable framework, takes some time. The result of the complete exercise is usually a bunch of data exchanges between the client and server. If we just create the data packet a client sends to the server (POST response), that would be comparatively be very few lines of code and very homogeneous too. On the unit level, things would get even easier.

This is the development part, maintaining UI tests is even more cumbersome. Since the code base is many times bigger than API level tests for instance, it becomes very expensive to maintain too. More lines of code and more complex – higher maintenance cost.

Fragility

Anyone in automation longer than a few months has tasted the fragility of UI test scripts. By following good development practices robustness of these scripts can be increased to a reasonable level, but UI tests are by nature are fragile. The UI elements can change quite rapidly, the browser version updates can have a change in behavior, response on different browsers is not the same and the list goes on.

Back in the day of WinRunner, UI automation was the mostly the only automation type most people used. So, there was no benchmark to compare UI tests against. The comparison was between good frameworks / tools and not so efficient ones. Now compared to other automation types in robustness, UI automation ranks at the bottom.

Slow execution

Big and complex UI automation batch runs take ridiculously long time to execute. Compared to a person doing that step manually they are fast no doubt, but again comparing to other automation types, these take a lot more time. To have a one to one comparison, we automated a test in our API suite and UI suite. The execution takes roughly around 7 minutes on the UI level, compared to just 12 seconds on the API level (that’s 35 times faster). A UI batch run taking 24 hours to run can complete probably in 40 minutes on API level.

The reason for this slowness is inherently the way UI tests are designed. For instance, after filling in 20 – 50 fields and many delays in between, the end result is usually a REST call to the server and getting a response back. If we construct the request and validate the response, it takes less than 2% time spent compared to filling the form and then validating it.

Reduced test coverage

A lot of business logic is hard to replicate from the front end. Some product features need very specific and detailed environmental settings to trigger that line of code on the business layer. UI automation very is inefficient for that. In many cases, it would be impractical especially where other hardware is involved, for instance, a car refueling hardware device sending information to a web server once the refuel is done. Instead you might want to just simulate the message on the API / Integration layer.

To perform even small tasks on the UI, it takes a lot of time compared to API level or unit level tests. This naturally has the effect of covering a lot less ground. For instance, a 500 man hour effort might cover 50 scenarios on the UI level, but on API level they might be able to bag 200 – 250 tests, depending on the application and both automation frameworks.

Lastly you can NEVER get a 100% test coverage from UI level. To reach there you would need API level and unit level tests to make sure every line of code is tested.

In the next post, we’ll briefly talk about what API level tests are and advantages of using them.