Among all the online conferences, Automation Guild is the best automation conference I happily attend, this year was a pleasure to speak at it again. If you are in automation, I think this is a must attend conference. In this post I\u2019ll give a brief overview of my talk at the conference.<\/p>\n

The subject of big data is exciting, but I\u2019ve felt there is a general lack of testing maturity in the space. I guess since the industry itself is comparatively new and is evolving. The walk was to share some basics about big data and how testing & automation works in this field.<\/p>\n<\/p>\n

About big data<\/h2>\n<\/p>\n

The evolution into big data has been fueled by technologies which have made processing lots of data at high speeds easy, and most importantly the ability to react to the insights very quickly. We discussed all these factors quickly, summarized in this image:<\/p>\n<\/div><\/span>

<\/h2>\n

What are big data projects all about<\/h2>\n<\/p>\n

The objective of big data projects is to gather insights \/ analytics to understand and solve problems. For that to happen, data from few or many sources may be needed to run analytics on. Now acquiring the data is usually not a big problem, to get it into a structure where it all makes sense collectively \u2013 is the challenge.<\/span><\/p>\n<\/p>\n

That\u2019s where the concept of a data pipeline comes in. The data is passed through different stages of \u2018transformation\u2019 \/ ETL (Extract, transform, load) to make it more usable for our problem at hand.<\/p>\n<\/div><\/span>

<\/h2>\n

Testing in Big data<\/h2>\n<\/p>\n

Like Web applications have some standard tests that happen, similarly in big data there are some tests which are common. However, they are nothing like the ones we do for web applications.<\/span><\/p>\n<\/p>\n

In data projects, all we are dealing with is \u2018data\u2019, data in and data out. The challenge is transforming the data as expected and building models which actually solve our problems. Therefore, most testing in this industry revolves around \u2018Data Quality\u2019.<\/p>\n<\/div><\/span>

Within the three stages of the data pipeline, there can be many ETL activities happening within. For each ETL, deciding what type of data quality checks are needed is important. In the talk we walked through a basic process of how to determine that.<\/p>\n<\/p>\n

Automating tests<\/h2>\n<\/p>\n

Because of the kind of tests we have in the big data space, automation also works quite differently. It\u2019s more about fetching sets of data and checking if the right logic \/ business rules was applied. To perform these activities, some data platforms provide the capability of doing that easily, if not the technologies used to build these ETL flows are also then used to test them.<\/p>\n

Talking about languages, Python is used widely because of it\u2019s data processing capability. These scripts are used within workflows to do the required validations. The most common validation is checking of all data has been copied from point A to B. Sometimes while moving data from one space to another, files or records get missed, maybe they get truncated or other reasons. This is just one of the 6 quality dimensions. <\/p>\n<\/p>\n

Data quality across the pipeline<\/h2>\n<\/p>\n

In the talk we walked through a sample pipeline and explained the kind of tests that can be done and how these tests would be executed. The image below summarizes all the checks discussed. The data pipeline was also expanded to show activities happening within the three stages and how they are tested.<\/p>\n<\/div><\/span>

<\/p>\n

If you were not part of the automation guild, you can still get access to it since all the talks and Q&A sessions are recorded. This talk would serve well for those willing to get into and expand within the big data field from a testing perspective.<\/p>\n<\/div>

<\/div><\/div><\/div><\/div><\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":3,"featured_media":15089,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"yoast_head":"\n