An important factor in automation framework design is managing your test data. Mismanaged test data for complex applications, for instance an ERP, could make life miserable. When I started automating an ERP product, my past experiences were inadequate to say the least. We were stuck in a rut not moving forward. Musarrat, a team member on the project came up with the ideas which helped us towards a sustainable solution. Thank you Musarrat.
Understand the product
Certain traits are common between application types, ERPs would have very complex test data, SaaS applications providing simple services, might have limited test data requirements and so on. However, it is paramount to understand
1. your application and the kind of data the tests might need.
2. Different methods through which the data can be created and removed (front end, web services, SQL scripts).
3. The amount of test data you would need per test, I automated products having 5 – 10 fields per test and 150 – 300 fields per test too. A large amount of data would have a VERY AMUSING impact on your tests.
We will not cover where to keep your test data in this post (XML, excel, csv, databases, JSON and so on). That is a separate pandora’s box I intend to deal with in a separate post.
The test data management process has two main areas, pre-script data treatment and post-script data treatment. In Paul’s words, creational strategy and clean up strategy. We’ll go through different methods under both and then pair ones that make more sense.
Test data can be created right before the test to execute, or in the beginning of the batch run, depending on execution schedules, the framework design and amount of test data.
Create records needed
The no brainer method is to create a specific record each time you run the test. Creation can be done through, again, Front end UI, web services or scripts adding rows to the database. The bright side here is it’s easy to debug and update. The down side, created data will have to be deleted at the end (again depends on the product and test, mostly might have to).
Some might not like the ‘delete at the end’ part, I don’t! Sometimes cleaning test data feels like getting that body fat off, every time we reach a great formula to solve the problem, it finds a new way to ‘stick around’.
The idea for random data is creating a new record every time we run the test. This way the record created last time does not affect our test. Since silver bullets do not exist, random data also has problems. Firstly, creating truly random data. This is more of a math problem, most algorithms tend to start repeating a pattern and the string generated is no longer as ‘random’. Secondly, if we keep adding body fat, eventually organs start to fail. Similarly, the test environment will over load reaching a point where it becomes unmanageable. We might have to get liposuction at some stage then!
For x number of tests, you would need x test data creation methods. Some tests can use data created by the previous test. For instance, a shopping cart application test case verifying a cart payment would need test data adding products to the cart first. Instead of adding data as a pre-requisite to this test, use data created by the test adding a product to the cart.
Main drawback here, if the previous test fails, the next test will fail due to the previous test which we do not want.
Add to AUT
Some tests do not perform transactions on any record, which means the data remain in the same state, before and after the test. Such tests can use data created in the application once, and no need to create or clean again.
This method has no real drawbacks that I came across (yes, sort of a silver bullet). However, you would not find many test scenarios where test data does not change during the test. Where this can work, the best strategy to use.
Depending on the creation strategy, you may or may not need a cleaning strategy. There are really not many methods here, but the one used can create an impact.
Delete from front end
At the test’s end, delete the record from the application as a user would. A no brainer method but, on applying brains turns out this can become the worst possible method.
For starters, sometimes creation is easy, the deletion business flow is way hard. Turns out the actual test was just 3 – 5 steps, that data takes 3 times more steps. Secondly, it is more prone to failure (which is why I hate this method the most). Probability of failure in automation is like playing those token games, mostly it’s not a question of IF you will lose, it’s WHEN you will lose! Let’s try to play longer with that one token right..
Delete from database
This is more stable method, might not run into many problems. However, what happens when the business flow changes and some other record also start to get affected? Most probably this change will just sneak in without the automation team knowing it changed (unless you have a well-knitted development process).
Delete using web service
Call a web service of the application to delete the data instead of doing it from the browser’s UI. This method can work very well. The problems with UI will not be there, also any change in the way records are treated for the specific test will also not affect the test either. Quick and robust.
Remember the ERP test data problem from the start, we tried all most of these methods, and nothing seemed to work for us. Unfortunately, this post is getting way to long. So we’ll talk about database restore, using Docker for managing data in a future post.