This year’s QA&TEST conference 2020 was another great addition. While planning the event there were discussions on if the event is supposed to be in person or online, turns out doing it online was definitely the right choice. I did miss the traditional Bilbao food from last year and an amazing experiencewhen I was there as a speaker, the online event this year was also well done too.
The AI track of the event had a talk by Parveen Kumar and a panel discussion with a quick 10-minute talk by all panelists followed by discussions on testing AI systems. Had a blast with the panelists, we all were from different backgrounds but surprisingly had very similar thoughts and challenges on talking about testing AI products. Thoroughly enjoyed the talks and presentations and gave me some new insights too which I want to share in this article.
Testing AI starts with data
Folks when thinking about testing AI systems start debating how to get into that black box neural network to figure out what’s going on and how to test it? The challenge off course, these are evolving programs which no one directly controls. In case of machine learning algorithms, the biggest factor is the data used to train the model – and IMHO that’s where testing should start from.
Before loads of data can be fed into the models, all that data needs to be gathered, cleansed and modelled. In my segment I talked about some fundamental concepts like data pipelines, data quality across these pipelines and some quality checks to look out for across different stages.
What’s a data pipeline?
All this usually starts with ‘Big’ data, which means the variety of data, speed of processing and size of data plays a big factor. From gathering this data till feeding it to an AI model, data passes through lots of stages. On a high level we classify them as:
Data ingestion – Getting data from different sources
Data lake & data warehouse – Creating data / domain models and normalized tables
Analytics – Creating semantic models and running analytics / or feed data into machine learning algorithms
As data is processed through this pipeline, the quality of data has to be measured to ensure as an end output we are able to provide clean data. One of the techniques to do this is data quality dimensions. There are 6 different attributes / characteristics (dimensions) which any data set should conform to. Measuring your data for these dimensions helps to analyze if the data is accurate and fir for purpose.
Stages across the data pipeline are curating data with different objectives, therefore the quality dimensions to look out for are also different. While this is a very detailed subject and I usually do a complete talk just on this, the illustrates below summarizes some examples:
The talks and questions during the panel discussion unearthed some interesting points which I feel might be very helpful for teams exploring to test AI systems.
For safety critical devices regulatory bodies provide practices, processes and guidelines governing how safety approvals will be given. With AI products the community is debating what is more practical and pragmatic approach to certify AI systems.
Due to evolving nature of AI products, it is possible the guidelines will be more process based rather than around the product’s functionality itself since those are going to be a moving target. It goes without saying this is a very complicated problem to solve and the stakes are high. Take an example of self-driving cars and it’s impact.
Continuous learning algorithms
In certain ML models, like deep learning, are mostly ever evolving. After the product is released, it still keeps learning and changing it’s algorithm. This poses a different set of challenges and the traditional test and release cycles are not enough. Observability and production tests become a must in such situations, which means testing becomes an ongoing activity happening in production.
Biases in AI products
AI models build their intelligence through machine learning by consuming large amounts of training data. The kind of data we provide is going to govern the kind of assumptions the model makes. In recent years a few incidences have surfaced where the AI model was biased to a certain group of people or other variables.
The challenge, many times we don’t even know if a bias exists. For instance, a leading tech company had an AI program to short list resumes. Later it was discovered the program assumed highly skilled people are usually male candidates. This was perhaps due to the training data it had, since most candidates who got job offers were men, it made that assumption. Even after knowing the problems it was very hard to fix it, and eventually they had to stop using the said AI program!
The most evident solution is first to figure out any biases that may exist before the training data is provided. The challenge is off course knowing about those biases. What can also help is giving a very wide range of data. Also train & test on every different data sets. This can highlight any incorrect assumptions and biases that might have been built / inferred from the training data set.
Standardizing building and testing AI systems
While regulators and the wider community is looking for ways to have some baseline practices, however the use cases of AI are very widespread and validating the underline algorithm is challenging, it’s going to be hard to reach some generic product-based validations.
Having said that, there are some techniques which can help look at these solutions from a different paradigm, which can be used as good techniques to identify potential risks in such systems. One such technique is STPA [link] which suggests an alternative top down approach of looking at holistic systems and focusing just on safety instead of focusing on the system’s functionality.
The field of AI is exciting and has lots of applications. By now we are already seeing many products started to use AI in some capacity. This is going to ever increase because of AI’s capability to process multi-dimension factors and process large amounts of data which can be hard for humans to do.
Apart from topics discussed above, the key challenge IMHO is going to be lack of skills. Looking after the quality aspect is going to be even more challenging, these systems needs engineers who have technical insights into how the underlying technology works plus have ‘testing Acumen’ to test well. While this has always been a problem, seems with AI systems & Big data projects this will be an even bigger one.
The subject of Big data fascinates people and businesses. While this might be just a buzz word, big data is immensely helpful to unearth important information and in the 21st century, information is power.
In this post, I’ll summarize my talk at the OnlineTestConf2019 titled “Why we call it big data and how to test it”. The talk was also recorded and can be watched on YouTube here.
An intro to big data
I start with talking about little bit history of Big data and what factors fueled growth and innovation in this industry.
Next we put ‘Big’ into perspective to help in understand the sheer size of data and the challenge it poses to process it.
Defining Big data
When would a project classify as big data? Is it only the size of data? This slide explains the different ways we tried to classify it and the most common method used.
The Hadoop platform
Hadoop is the most widely used big data platform which is also open source. I talk about it’s widely used MapReduce process and different products within like HDFS, HBase and HiveQL.
The Data Pipeline
All we are doing in a big data project is collect data from different sources, hash it up into meaningful big tables and generate insights from it. There are three main phases you might have in a big data project.
At the end we quickly skim through the different type of tests we perform across the pipeline. At each stage, depending on the type of activities being performed, the type of tests will be different.
Due to lack of time I couldn’t go into details of the Quality Dimensions and sample tests of these dimensions across the pipeline. My talk at the AutomationGuild 2020 explains that in more detail.
Katjya did a very good sketch summarizing the talk she mentioned in her tweet.
Among all the online conferences, Automation Guild is the best automation conference I happily attend, this year was a pleasure to speak at it again. If you are in automation, I think this is a must attend conference. In this post I’ll give a brief overview of my talk at the conference.
The subject of big data is exciting, but I’ve felt there is a general lack of testing maturity in the space. I guess since the industry itself is comparatively new and is evolving. The walk was to share some basics about big data and how testing & automation works in this field.
About big data
The evolution into big data has been fueled by technologies which have made processing lots of data at high speeds easy, and most importantly the ability to react to the insights very quickly. We discussed all these factors quickly, summarized in this image:
What are big data projects all about
The objective of big data projects is to gather insights / analytics to understand and solve problems. For that to happen, data from few or many sources may be needed to run analytics on. Now acquiring the data is usually not a big problem, to get it into a structure where it all makes sense collectively – is the challenge.
That’s where the concept of a data pipeline comes in. The data is passed through different stages of ‘transformation’ / ETL (Extract, transform, load) to make it more usable for our problem at hand.
Testing in Big data
Like Web applications have some standard tests that happen, similarly in big data there are some tests which are common. However, they are nothing like the ones we do for web applications.
In data projects, all we are dealing with is ‘data’, data in and data out. The challenge is transforming the data as expected and building models which actually solve our problems. Therefore, most testing in this industry revolves around ‘Data Quality’.
Within the three stages of the data pipeline, there can be many ETL activities happening within. For each ETL, deciding what type of data quality checks are needed is important. In the talk we walked through a basic process of how to determine that.
Because of the kind of tests we have in the big data space, automation also works quite differently. It’s more about fetching sets of data and checking if the right logic / business rules was applied. To perform these activities, some data platforms provide the capability of doing that easily, if not the technologies used to build these ETL flows are also then used to test them.
Talking about languages, Python is used widely because of it’s data processing capability. These scripts are used within workflows to do the required validations. The most common validation is checking of all data has been copied from point A to B. Sometimes while moving data from one space to another, files or records get missed, maybe they get truncated or other reasons. This is just one of the 6 quality dimensions.
Data quality across the pipeline
In the talk we walked through a sample pipeline and explained the kind of tests that can be done and how these tests would be executed. The image below summarizes all the checks discussed. The data pipeline was also expanded to show activities happening within the three stages and how they are tested.
If you were not part of the automation guild, you can still get access to it since all the talks and Q&A sessions are recorded. This talk would serve well for those willing to get into and expand within the big data field from a testing perspective.
Another year of great learnings has passed. I am thankful for having the opportunity to gain lots of new insights and experiences. Most of the year was spent on designing and supporting an IT transformation focusing to deliver at speed with quality. This initiative allowed me to be exposed to a lot of different challenges and meet wonderful people.
Stepping into 2020 I thought of writing top 3 learnings which I want to take into the next year and work more on.
Everyone has the same amount of time in a day, then how some people / teams accomplish much more than others? There has been considerable research on the subject, I feel the common theme in most results is successful people / teams don’t waste effort.
From reading and working in scrum teams, a very core focus in agile ways of working is to focus only on ‘what’s important’. Further to decide what’s important, ask the customer who’s paying for the effort. This singular focus on what’s important, what’s needed to meet the objective is a key factor in success.
Over the holidays some family time was spent talking about achieving goals. I shared an observation that every person has the same 24 hours in a day. Also, every person while they are awake are doing something, working on something, watching something, talking or whatever, just doing ‘something’. To achieve your goals, don’t do ‘something’, do ‘THE thing’ instead of low priority items.
I often give the example of a magnet. All solids have atoms exerting force in some direction, but the net result of all force is neutral. In magnets, like other solids atoms are exerting force too, only difference is they align in one direction. Collective focus in a singular direction makes the difference.
The problem with people / teams, we don’t always have a prioritized list of activities to do to reach our objective. Without this ‘prioritized backlog’, we react instead of being proactive and end up working on what’s not important and waste time for which the users / business is not ready to pay for.
For additional reading on alignment of purpose, especially for scum teams, I’d recommend “Scrum: The Art of Doing Twice the Work in Half the Time” by Jeff Sutherland.
Leadership is being Selfless
My philosophy for leadership has always been ‘leading by example’. In my quest to learn more about transformation, leadership kept coming up. Therefore, I turned to my favorite speaker on the subject: Simon Sinek.
From primitive times, leaders of the tribe were always given first choice of food and preferential treatment, and members of the tribe didn’t have a problem with that. That’s because they were getting something in return, the leader giving sacrifices which others were not ready to do. To protect from any external dangers and look after the tribe members.
The same unwritten rules still govern the modern world, but when leaders don’t give any protection in return, or don’t look after our interests, this unwritten contract is breached, that’s we have a problem. Leadership, by definition, is an act of being selfless and keeping the interests of the tribe / community / team ahead of your own personal interests.
In the capitalist world we live in, with extreme deprivation of moral code and ethics, it’s challenging to be truly selfless in a corporate environment. I feel we should at least try to be closest to this golden standard. One cannot be a true leader unless they have interests of others ahead of their own.
Additional reading: Leaders eat last by Simon Sinek.
DevOps – A mindset & a TON of enablers
I’ve always said the Agile manifesto had the core of DevOps in there, in theory DevOps does not suggest a new paradigm shift, it provides pointers around technology and tools to really become agile. Therefore, a big part of DevOps is the cultural mindset change.
A big part of that mindset change for me were the two points mentioned earlier, eliminate waste and leadership. To produce quickly, the whole team has to be single mindedly focused on one objective only and not waste time second or third priority items. Only then we can quickly move things from ‘work in progress’ to ‘done’.
Part of waste is teams waiting around for someone to take a decision and tell them what to do. Teams need to feel empowered, so they think for themselves, take decisions and make things happen. As they progress, they’ll learn and improve as they move along. This is where leadership comes in, if leaders selflessly try to make things easier for team members to do their job, they feel more empowered and motivated to go ahead.
With DevOps, things don’t just end with transformational talk, a lot of enablers need to be put in place to achieve this. Therefore,
DevOps is sexy – but not a piece of cake
Bare minimum to do it well, you need CI pipeline(s) with all hooks in place, code repos with proper branching / merging, spinning up of on-demand environments which are fit for purpose, a TON of automation running at different stages across the pipeline, mocks / stubs, test data creation / deletion, connectivity with other products where needed. So you see, doing all this is not sexy at all.
Oh, and lastly, if you expect DevOps is going to reduce your cost, in the long run it should yes, but for a year or two you’ll be spending 2X or 3X the amount to get all these enablers in place. It reduces cost by eliminating waste, reducing manual churn, people not working on low priority items (back to cultural change)
To sum this up, I wrote a short Poem a few weeks back:
Software over the wall ; results in a great fall.
Testers, developers and all ; will then not like management’s call.
Ship in collaboration, be quick and release small ; that’s a lesson for us all.
There were a lot of other things I experienced, these were the top 3. Looking forward to continue working on these in 2020 as well.
Since the start of my career as a tester, I’ve always been obsessed with testers delivering tangible value to the business. I realized ‘redefining software quality’ was my calling. There were quite a few powerful stories that lead to this realization, and one of the most important ones I presented at the QA&TEST 2019 conference.
This was a story about testers taking responsibility and recognizing how important it is to understand the underlying technology of the product. From this event emerged the three core values I feel which can reshape how we think about software quality.
This talk was awarded the best presenter award at the QA&TEST 2019 conference,
I thoroughly enjoyed the conference, the city and the hospitality. I talked about it in this articleand a short shout out for the organizers on a job well done.
Presentation key points
Out of scope
The products being very complex, it was impossible to test everything through black box testing. Areas which the testers were unable to understand how they work, or there was no way to test them, were not considered as the tester’s responsibility.
While quality is everyone’s responsibility, but testers do need to make sure all bases are covered. It isn’t necessary that everything must be tested by a tester, but a tester should make sure that ‘someone’ is testing all the areas that should be covered.
Code coverage is not the only measure to check, but certainly is one of the important ones. Unless we are sure all statements and branches of the code have been executed and tested, we cannot be sure of all the hidden risks. With just black box exploratory testing, it’s almost impossible to figure out the code coverage % even is, let alone having a decent code coverage.
In our story here that was the case, almost zero visibility into the code, some functionality had been going untested. Neither was there any effort in figuring out how to improve that coverage.
In pursuit to improve coverage we came with a method to do fault injection / mutation testing. Instead of controlling the input parameters from outside, alter the control flow within the program to reach the desired state.
Without knowing the technical details of how the product works, it’s very difficult to figure out the risk areas and to learn how to test them efficiently. A good understanding of product architecture and what areas are high risk is key.
Testers MUST put in effort to learn the technology. Without it, IMHO one cannot be a good tester.
Testing is more about thinking and collaborating than doing and checking. Learn to think how to test and learn / develop models on how to test. Unfortunately most testers intuitively learn to test well and don’t follow a systematic way to enhance that skill.
Redefining Software Quality
The most valuable learnings for me were the importance of testers to learn technology, enhanced methods to identify risk and ability to provide value in the context business.
Two weeks ago I went to the Test Leadership Congress conference in NYC, absolutely loved the content and speakers. It was at a very opportune time while we were working on IT transformation and myself looking after the processes and quality expectations in our new ways of working.
With the prevalent wisdom of fully automating ‘testing’ and less need for test leadership, the conference was loaded with evidence on the contrary. In the post-agile apocalypse, the system level thinking and the pivotal role of testers to question what and why we are building what is user story says and the current build demonstrates, is all getting lost in the mushroom cloud of buzz words and time to market.
There were a lot of amazing sessions including psychological techniques to figure out your own and your organizational values, using Improv to make a point and learn look at things from a holistic perspective, playing games designed to improve thinking and test design and a board game to understand the important aspects in continuous delivery. Most of the sessions were so great, it was hard to pick which one to go to.
The organizers especially Anna and Andrea did a great job in choosing the speakers to have a great mix and schedule the activities. Typically you would see topics around specific tools or technologies, this one was more about developing leadership which I feel is need of the hour.
It’s going to take me a while to digest and share my thoughts around the sessions. On top of my head here were the few learnings that really stuck with me:
System Theoretic Process Analysis – STPA.
Dr John P. Thomas’s keynote on systems approach to software testing was very refreshing. Having a similar background of working with safety critical systems, I have always felt the absence of certain things in the agile world like spending time in understanding and questioning the requirements, mapping the entire system and looking at how the control flow works. STPA was a process introduced to us by John which felt like a systematic process to codify how that system thinking is supposed to be done. Will share more in a separate post.
The Protreptic dialog.
Ive always struggled with finding out values of other people to act according to their and the organization’s commonly held values. I always try to do ‘only’ the right thing which now I feel might not always be the best way. In most cases should try to be resonant of the organizational values and try to start operating from there instead.
The protreptic dialog explanation and demonstration in which Ole and Anders did a great workshop which showed a way to understand other people’s perspectives and learn more about their values.
In the dialog, you ask the other person to describe an event and have follow up questions while being genuinely curious to understand what it means for them. Try to understand:
What do you sense about the person (hear in their voice, body language)
How do you think they feel (deduced from what you sensed above)
Ask the person to reflect on their feelings
Ask why this means so much to them, why does it matter to them.
If You Really Want to Make a Difference, Be the Flexibly Robust Leader
Ole S Rasmussen talk on Personal leadership was great and this stood out for me out the whole workshop. He explained what being robust and flexible meant and how to become a ‘flexible robust’ leader.
The technique to do a self assessment and find out what your personal values are and your own limits. In a short exercise we identified what does fragility mean for us and what corner is most significant for us in the philosophical triangle (shown below). It was a good demonstration to see how to journey inwards and understand your own self.
As a leader, understanding your own self is very helpful, gives an idea of your own limitations, strengths and weaknesses. For me the exercise revealed Fragility for me meant fearful, and from the triangle ‘attention to what matters most’ was most important. stitching them together we figured I may be fearful of not focusing on the right thing and waste my energy and my team’s on doing what does not matter the most.
The game of continuous delivery
Tanya’s session on continuous delivery was nice in which she explained the guiding principles for DevOps – CALMR. Working on SAFe framework I had been exposed to these principles, but what was new was the second half of the talk, a continuous delivery board game!
Looked like monopoly of DevOps. I’ve always been very passionate about having dry runs of complex problems to do some up-front thinking instead of just trying a whole bunch of failures and then finding what works. The board game instilled the importance of certain aspects in coding CI/CD which are otherwise hard to comprehend without practical experience.
For instance, how it’s important for features to be in production instead of having a whole heap of them at different stages in the pipeline, unless it’s out there, you cannot make money!
There were many other sessions which were great, I’ll add more details on them in separate posts. I learned not only in the sessions, but outside the workshops and presentations networking with others was a great opportunity to learn. Talking with leaders in other organizations and practitioners there was so much more we learned from each other.
‘No one’ has it all figured out
The biggest take away from these talks was: “EVERYONE IS STRUGGLING WITH CONTINUOUS TESTING”. One would assume the big names and thought leaders in our industry would have everything figured out, but we are all struggling, some are quite close, others might be far behind, but it seems no one has it nailed down with all their products working in a complete DevOps culture across the enterprise
those of us looking for greener grass on the other side, try to cultivate some green grass where you are, no one has the magic formula to continuous testing.
Fun with a Capital F
The games and round table session was amazing. We all had a blast designing test strategy for products of our choosing and then demonstrating them.
There was a separate games time in which Eddie and Jordan organized a lot of different games. I got pulled into playing story cubes, we all enjoyed it very much, it was so hilarious the kind of stories we came up with.
End of last day we had an after party, lots of great conversations and new friends. Had a wonderful time.
The most inspiring event
A fellow tester and I tried to meet during my last time visit to the US. This time he was determined and I’m glad we did it. He had to travel such a long distance in one day, just to learn and improve his skill set. I shred the details of the story in this post.
Met a lot of people I had been interacting with in the testing community on social media, Ajay being one of them. While traveling back from the conference Ajay and I happened to be on the same flight. While waiting at JFK we recorded a short video about how the conference went and what we liked about it.
Accompanying me on this trip was Head to QA at Emirates, to whom I had suggested the conference. She loved the setup too and while going back we recorded our thoughts, what we learned and some key take aways, you can watch in this video.
And enjoying the comfy Emirates A380 business class 🙂
Automation projects have a lot of problems which are common across the board. With the open source world of Selenium and others, there are many common areas where every new person struggles with. A group of people working in the industry for years tried to solve some of these challenges in their own way and named it ‘TestProject’.
What is it
In a nutshell I understood it to be kind of a cross breed between keyword driven and record play back but very different in terms of how it codifies the steps, how it interacts with the browser, saving and retrieving test data and so on. While this is an important feature of the ‘platform’, it’s not the main one from what I see. An execution and reporting engine is separate and different than traditional ones we see. Lastly the feature I liked the most, ‘Addons’ which is which is more like Chrome’s ‘Web Store’ where folks can build components and share them.
I feel the challenges automation faces today are very real and complex, and about to become extremely problematic which new technologies surfacing where we are not sure even how to test those in person! It might be impossible for any individual company / team to have breakthrough and will have to be a collaborative effort. The pace of change in technology, fragmented industries and changing market dynamics for software products all are very surreal threats which bog down evolutionary process of automation.
The part which resonated the most with me about TestProject was it’s a platform, not a tool. It allows you to have your individual code base and a very easy mechanism to share any libraries you develop which might help others through ‘Addons’. This should make it easy for teams to create solutions which can not only be shared across teams, but also across the complete platform.
While libraries are the most commonly shared ‘reusable’ resource, the platform allows to share elements (for object recognition), applications, data sources and parameters as well. Something I preach a lot under the ‘reusability’ pillars of framework design is to make not only your methods reusable, also leverage every other aspect of the framework.
With open source tools running scripts on multiple platforms (different browsers, different OS combinations), different execution environments can be difficult to manage. To make things easy, we usually have pre-defined test environments (machines, VMs, SaaS like Sauce Labs, Docker) and port our code in those prepared environments.
While this in theory sounds great, at times takes a lot of resources to setup new environments. Not only that, there is an upkeep cost associated with it. Automation is juggling between AUT versions, platform versions (browser, mobile OS), and the automation tool versions. Not to mention other tools you might have integrated for CI and others. At the end of the day all of this takes its toll and can mount into a sizeable effort if not managed properly.
In TestProject setting a new environment for any desired platform is easy, like how vendor-based tools do it. Install their executor and it takes care of the rest. The difference there is in the process of the setup. This being a SaaS product, can open, compile and run on any environment which makes it very easy.
This means you can flip the situation. Instead of making execution environments at the center of the equation, make your code base the center of the equation and you can easily change any environment into an execution environment for automation.
Eluded in the previous point, code access becomes very easy. Not just code access, but accessing the whole eco-system is very easy. Typically for someone wishing to investigate the automation code base, they’d have to log into an automation developers or execution environment. Trying to do it on ‘any’ environment is not possible and often needs some setup to be done first (IDE, Pre-reqs, SCM and so on) Since TestProject is a SaaS product, you just log in from anywhere and you can see everything. The code base, test suites, test execution results, all in one place accessible everywhere.
The keyword plus record and playback architecture is something the industry has struggled with for decades, and we still do. While almost every automation tool has a keyword driven or record / playback mechanism, which is mostly the focal selling point, there is a lot of snake oil in the industry around this.
The need for such a framework in an organization is a valid one, however creating one which can be plug and play for any product is impossible. In TestProject, there is an opportunity where experienced folks can ‘create’ components which core testers can use as building blocks. While this still might not be fool proof, I think it might be a step in the right direction and would help teams looking to create such a solution for their product.
It’s the start
While there are some fundamental shifts in the way TestProject is designed, it’s just the start. The success may hinge on how well it solves the fundamental issues it was designed to overcome along with adapting to the changing environment.
The #PSQC18 conference on April 07, 2018 was around the theme of ‘adapting to change’. There were around 11 talks on different topics from testing, automation, agile, security and data analytics and a segment ‘My two cents in two minutes’ where participants were invited to share the challenges faced in the testing career in two minutes.
Background to the talk
For past 11 years I have been trying to understand why testers are treated as second class citizens in the software industry, one of the changes we need to make is ‘Technological Excellence’, for more read here. In that spirit, this talk was around how to become a ‘Technical tester’, where a hypothetical fresh graduate shares his testing career adventures and lessons he learns along the way solving each challenge.
Testers must be technical
I argue testers must be technical through a story our hypothetical tester shares with us. A story of his first testing job where features were left untested due to lack of technical depth, causing huge repercussions for the company.
Understanding the Technology stack
Our tester’s next job is about testing a hybrid application, this time he shares his journey of learning about technology stacks and how learning that helps him in testing.
Aptitude for designing algorithms
The next challenge is to learn automation due to the amount of testing he has to do. He learns it’s not about tools or languages which makes a good automation engineer, but something even more fundamental.
Develop the right attitude
Over time once the peaceful workdays have now turned into stressful debug marathons. On the run again to learn how to deal with it, his master guru tester explains the dilemma of a programmer life and how to deal with trouble shooting.
The talk seemed to be well received, so I made a separate video for the YouTube channel where you can enjoy the insightful journey of our young tester :
TALK MAIN PARTS:
Introduction: 0:15 – 1:00
Perceived vs actual career path of a tester: 1:00 – 3:19
Learning the importance of being technical: 3:20 – 7:50
Maximum test coverage through learning the Technology stack: 7:50 – 22:05
The Keystone of learning automation: 22:05 – 31:55
The bane of programming and the holy grail: 31:55 – 38:10
Sharing his adventures with the Guru tester (Recap): 38:10 – 40:20
I had the opportunity to be part of the expert round table panel along with Angie Jonesand Oren Rubinwhich was a great experience. There were a lot of great questions, which showed how awesome the community was. Sharing some of those questions here and teasers from the answers.
How to catch bugs earlier
If you want to catch issues earlier in the cycle, then essentially you are talking about shifting left. Get tests executed earlier in the cycle to get early feedback. Divide the tests to execute at different stages of the development life cycle, some can be executed during development, during patch testing, during regression testing, basic tests after deployment on production and so on. Automation checks would go a long way in achieving this, but ultimately, it’s going to be testers who can really catch issues.
The purpose of automation is not to capture bugs, rather take over the mundane tasks which keep testers from spending their time ‘checking’ functionality. Automation is eventually not going to report a lot of bugs, it’s rather enabling testers to capture more bugs, a great point from Angie.
Which coding language is recommended
In my opinion, more important than selecting a language is developing the aptitude of designing algorithms. Learn the fundamentals of programming which are common across all languages. Once you can develop an algorithm to solve problems, language selection becomes less important. Plus, languages will keep on changing and you can never stick to one language forever. If you start with learning a language per se for instance Java, you might get tangled up in syntax issues and debugging problems than actually learning to code.
Retry failing tests
A solution to flaky tests is sometimes suggested as rerunning the flaky test. In multiple tries if it does not fail consistently, means there is no bug here rather just a script issue.
All of us at the panel thought retrying flaky tests is not a great idea. It adds lots of execution time and pushes away the concept of trying to avoid flakiness in the first place. Rather, avoid flakiness at all costs. There are a lot of ways to do that, use what works best for you. Apart from tips like using dynamic delays, using the AUT’s client side functions etc, try breaking the test into smaller tests. Longer scripts have a greater tendency to fail and cause problems.
The ultimate solution for flakiness is to have a great automation framework built to support the execution and handle unexpected states of the application.
How to elevate automation’s success to management
The goal of automation is to identify risks and inform of any potential issues going into the field. Unfortunately, many times we end up working on issues which might not directly translate to the value added towards this goal. It’s also hard to get matrices which reflect this, but we should not stop trying. The ‘Matric black hole’, a term from the book ‘deep work’, talks about this concept as well and about the complex nature of getting matrices on productivity which tend to be subjective and very hard to measure.
One of the values at Quality Spectrum is generating ‘business value’. This point eludes to that goal, to get tangible results proportionate to the effort done. It’s a long and complex road for which certainly we don’t have answers yet, but it is an important enough problem to solve. For starters, I feel measuring how much more test coverage we have is a better one. Not in the sense of number of tests executed, but how much of the code base have we tested. Tools from companies like SeaLightscan provide you with what areas of the code has been executed while you have been doing your tests and what areas remain to be tested. Another angle can be from the user perspective of coverage. This topic deserves a separate post, so I’ll just pause the discussion here.
A great point from Oren was the value of seeing passing test coverage. Getting the assurance of certain tests passing does add to the confidence in the build and allows testers to focus on other areas. Seeing a green mark on your automation results does have value and should not be discarded.
Management for the most part does not have a clear idea on how to measure value from automation and often are looking at the wrong metric. This comment from Angie was so true, I too never recall anywhere management having a clear vision of how to assess automation’s success, unless the person at the top has been doing automation themselves.
If you did not have the opportunity to join the guild this year, I’d recommend doing that. It gives a broad idea of what’s going on in the community, introduction to lots of new concepts and tools which one might not know if they were as important and where to start.
A lot of questions went unanswered on the first day’s round table I was at for which I am in the process of creating video replies for. The answers are uploaded to this playlist.
If you have any questions you’d like me to talk about, send them to firstname.lastname@example.org.