My Experience at the Test Leadership Congress

By | July 22nd, 2019|Uncategorized|

wo weeks ago I went to the Test Leadership Congress conference in NYC, absolutely loved the content and speakers. It was at a very opportune time while we were working on IT transformation and myself looking after the processes and quality expectations in our new ways of working.

With the prevalent wisdom of fully automating ‘testing’ and less need for test leadership, the conference was loaded with evidence on the contrary. In the post-agile apocalypse, the system level thinking and the pivotal role of testers to question what and why we are building what is user story says and the current build demonstrates, is all getting lost in the mushroom cloud of buzz words and time to market.


There were a lot of amazing sessions including psychological techniques to figure out your own and your organizational values, using Improv to make a point and learn look at things from a holistic perspective, playing games designed to improve thinking and test design and a board game to understand the important aspects in continuous delivery. Most of the sessions were so great, it was hard to pick which one to go to.

The organizers especially Anna and Andrea did a great job in choosing the speakers to have a great mix and schedule the activities. Typically you would see topics around specific tools or technologies, this one was more about developing leadership which I feel is need of the hour.

Key learnings

It’s going to take me a while to digest and share my thoughts around the sessions. On top of my head here were the few learnings that really stuck with me:

System Theoretic Process Analysis – STPA.

Dr John P. Thomas’s keynote on systems approach to software testing was very refreshing. Having a similar background of working with safety critical systems, I have always felt the absence of certain things in the agile world like spending time in understanding and questioning the requirements, mapping the entire system and looking at how the control flow works. STPA was a process introduced to us by John which felt like a systematic process to codify how that system thinking is supposed to be done. Will share more in a separate post.

The Protreptic dialog.

Ive always struggled with finding out values of other people to act according to their and the organization’s commonly held values. I always try to do ‘only’ the right thing which now I feel might not always be the best way. In most cases should try to be resonant of the organizational values and try to start operating from there instead.

The protreptic dialog explanation and demonstration in which Ole and Anders did a great workshop which showed a way to understand other people’s perspectives and learn more about their values.

In the dialog, you ask the other person to describe an event and have follow up questions while being genuinely curious to understand what it means for them. Try to understand:

  1. What do you sense about the person (hear in their voice, body language)
  2. How do you think they feel (deduced from what you sensed above)
  3. Ask the person to reflect on their feelings
  4. Ask why this means so much to them, why does it matter to them.

If You Really Want to Make a Difference, Be the Flexibly Robust Leader

Ole S Rasmussen talk on Personal leadership was great and this stood out for me out the whole workshop. He explained what being robust and flexible meant and how to become a ‘flexible robust’ leader.

The technique to do a self assessment and find out what your personal values are and your own limits. In a short exercise we identified what does fragility mean for us and what corner is most significant for us in the philosophical triangle (shown below). It was a good demonstration to see how to journey inwards and understand your own self.

As a leader, understanding your own self is very helpful, gives an idea of your own limitations, strengths and weaknesses. For me the exercise revealed Fragility for me meant fearful, and from the triangle ‘attention to what matters most’ was most important. stitching them together we figured I may be fearful of not focusing on the right thing and waste my energy and my team’s on doing what does not matter the most.

The game of continuous delivery

Tanya’s session on continuous delivery was nice in which she explained the guiding principles for DevOps – CALMR. Working on SAFe framework I had been exposed to these principles, but what was new was the second half of the talk, a continuous delivery board game!

Looked like monopoly of DevOps. I’ve always been very passionate about having dry runs of complex problems to do some up-front thinking instead of just trying a whole bunch of failures and then finding what works. The board game instilled the importance of certain aspects in coding CI/CD which are otherwise hard to comprehend without practical experience.

For instance, how it’s important for features to be in production instead of having a whole heap of them at different stages in the pipeline, unless it’s out there, you cannot make money!

There were many other sessions which were great, I’ll add more details on them in separate posts. I learned not only in the sessions, but outside the workshops and presentations networking with others was a great opportunity to learn. Talking with leaders in other organizations and practitioners there was so much more we learned from each other.

‘No one’ has it all figured out

The biggest take away from these talks was: “EVERYONE IS STRUGGLING WITH CONTINUOUS TESTING”. One would assume the big names and thought leaders in our industry would have everything figured out, but we are all struggling, some are quite close, others might be far behind, but it seems no one has it nailed down with all their products working in a complete DevOps culture across the enterprise

That means,

those of us looking for greener grass on the other side, try to cultivate some green grass where you are, no one has the magic formula to continuous testing.

Fun with a Capital F

The games and round table session was amazing. We all had a blast designing test strategy for products of our choosing and then demonstrating them.

There was a separate games time in which Eddie and Jordan organized a lot of different games. I got pulled into playing story cubes, we all enjoyed it very much, it was so hilarious the kind of stories we came up with.

End of last day we had an after party, lots of great conversations and new friends. Had a wonderful time.

The most inspiring event

A fellow tester and I tried to meet during my last time visit to the US. This time he was determined and I’m glad we did it. He had to travel such a long distance in one day, just to learn and improve his skill set. I shred the details of the story in this post.

Travel back

Met a lot of people I had been interacting with in the testing community on social media, Ajay being one of them. While traveling back from the conference Ajay and I happened to be on the same flight. While waiting at JFK we recorded a short video about how the conference went and what we liked about it.

Accompanying me on this trip was Head to QA at Emirates, to whom I had suggested the conference. She loved the setup too and while going back we recorded our thoughts, what we learned and some key take aways, you can watch in this video.

And enjoying the comfy Emirates A380 business class 🙂

Testing Heroes

By | September 30th, 2018|daily post, Uncategorized|

Drivers of the testing community ?

Are (IMHO) the testing heroes..

In my recent visit to Philadelphia I realized something new, importance of heroes

Every idea, change needs a change agent, a leader or group of leaders

Without those leaders the movement / change dies

Our testing community is no different, our thought leaders (heroes) have been one of the main cause of evolution in the field

Linked video in the comments has a few of my thoughts around why they are important..

TestProject – An Automation Platform

By | September 16th, 2018|Uncategorized|


Automation projects have a lot of problems which are common across the board. With the open source world of Selenium and others, there are many common areas where every new person struggles with. A group of people working in the industry for years tried to solve some of these challenges in their own way and named it ‘TestProject.

What is it

In a nutshell I understood it to be kind of a cross breed between keyword driven and record play back but very different in terms of how it codifies the steps, how it interacts with the browser, saving and retrieving test data and so on. While this is an important feature of the ‘platform’, it’s not the main one from what I see. An execution and reporting engine is separate and different than traditional ones we see. Lastly the feature I liked the most, ‘Addons’ which is which is more like Chrome’s ‘Web Store’ where folks can build components and share them.



I feel the challenges automation faces today are very real and complex, and about to become extremely problematic which new technologies surfacing where we are not sure even how to test those in person! It might be impossible for any individual company / team to have breakthrough and will have to be a collaborative effort. The pace of change in technology, fragmented industries and changing market dynamics for software products all are very surreal threats which bog down evolutionary process of automation.

The part which resonated the most with me about TestProject was it’s a platform, not a tool. It allows you to have your individual code base and a very easy mechanism to share any libraries you develop which might help others through ‘Addons’. This should make it easy for teams to create solutions which can not only be shared across teams, but also across the complete platform.

While libraries are the most commonly shared ‘reusable’ resource, the platform allows to share elements (for object recognition), applications, data sources and parameters as well. Something I preach a lot under the ‘reusability’ pillars of framework design is to make not only your methods reusable, also leverage every other aspect of the framework.



With open source tools running scripts on multiple platforms (different browsers, different OS combinations), different execution environments can be difficult to manage. To make things easy, we usually have pre-defined test environments (machines, VMs, SaaS like Sauce Labs, Docker) and port our code in those prepared environments.

While this in theory sounds great, at times takes a lot of resources to setup new environments. Not only that, there is an upkeep cost associated with it. Automation is juggling between AUT versions, platform versions (browser, mobile OS), and the automation tool versions. Not to mention other tools you might have integrated for CI and others. At the end of the day all of this takes its toll and can mount into a sizeable effort if not managed properly.

In TestProject setting a new environment for any desired platform is easy, like how vendor-based tools do it. Install their executor and it takes care of the rest. The difference there is in the process of the setup. This being a SaaS product, can open, compile and run on any environment which makes it very easy.

This means you can flip the situation. Instead of making execution environments at the center of the equation, make your code base the center of the equation and you can easily change any environment into an execution environment for automation.



Eluded in the previous point, code access becomes very easy. Not just code access, but accessing the whole eco-system is very easy. Typically for someone wishing to investigate the automation code base, they’d have to log into an automation developers or execution environment. Trying to do it on ‘any’ environment is not possible and often needs some setup to be done first (IDE, Pre-reqs, SCM and so on) Since TestProject is a SaaS product, you just log in from anywhere and you can see everything. The code base, test suites, test execution results, all in one place accessible everywhere.



The keyword plus record and playback architecture is something the industry has struggled with for decades, and we still do. While almost every automation tool has a keyword driven or record / playback mechanism, which is mostly the focal selling point, there is a lot of snake oil in the industry around this.

The need for such a framework in an organization is a valid one, however creating one which can be plug and play for any product is impossible. In TestProject, there is an opportunity where experienced folks can ‘create’ components which core testers can use as building blocks. While this still might not be fool proof, I think it might be a step in the right direction and would help teams looking to create such a solution for their product.


It’s the start

While there are some fundamental shifts in the way TestProject is designed, it’s just the start. The success may hinge on how well it solves the fundamental issues it was designed to overcome along with adapting to the changing environment.

PSQC 2018 Conference Talk

By | April 15th, 2018|Uncategorized|

The #PSQC18 conference on April 07, 2018 was around the theme of ‘adapting to change’. There were around 11 talks on different topics from testing, automation, agile, security and data analytics and a segment ‘My two cents in two minutes’ where participants were invited to share the challenges faced in the testing career in two minutes.

Background to the talk

For past 11 years I have been trying to understand why testers are treated as second class citizens in the software industry, one of the changes we need to make is ‘Technological Excellence’, for more read here. In that spirit, this talk was around how to become a ‘Technical tester’, where a hypothetical fresh graduate shares his testing career adventures and lessons he learns along the way solving each challenge.

Testers must be technical

I argue testers must be technical through a story our hypothetical tester shares with us. A story of his first testing job where features were left untested due to lack of technical depth, causing huge repercussions for the company.

Understanding the Technology stack

Our tester’s next job is about testing a hybrid application, this time he shares his journey of learning about technology stacks and how learning that helps him in testing.

Aptitude for designing algorithms

The next challenge is to learn automation due to the amount of testing he has to do. He learns it’s not about tools or languages which makes a good automation engineer, but something even more fundamental.

Develop the right attitude

Over time once the peaceful workdays have now turned into stressful debug marathons. On the run again to learn how to deal with it, his master guru tester explains the dilemma of a programmer life and how to deal with trouble shooting.

The talk seemed to be well received, so I made a separate video for the YouTube channel where you can enjoy the insightful journey of our young tester :


  • Introduction: 0:15 – 1:00
  • Perceived vs actual career path of a tester: 1:00 – 3:19
  • Learning the importance of being technical: 3:20 – 7:50
  • Maximum test coverage through learning the Technology stack: 7:50 – 22:05
  • The Keystone of learning automation: 22:05 – 31:55
  • The bane of programming and the holy grail: 31:55 – 38:10
  • Sharing his adventures with the Guru tester (Recap): 38:10 – 40:20


The presentation slides can be downloaded here.

Automation Guild 2018 Round Table Q&A

By | January 26th, 2018|Uncategorized|

The Automation Guild 2018 conference this year was a nice event, people from around the world coming together to learn and improve their automation efforts.

I had the opportunity to be part of the expert round table panel along with Angie Jones and Oren Rubin which was a great experience. There were a lot of great questions, which showed how awesome the community was. Sharing some of those questions here and teasers from the answers.


How to catch bugs earlier

If you want to catch issues earlier in the cycle, then essentially you are talking about shifting left. Get tests executed earlier in the cycle to get early feedback. Divide the tests to execute at different stages of the development life cycle, some can be executed during development, during patch testing, during regression testing, basic tests after deployment on production and so on. Automation checks would go a long way in achieving this, but ultimately, it’s going to be testers who can really catch issues.

The purpose of automation is not to capture bugs, rather take over the mundane tasks which keep testers from spending their time ‘checking’ functionality. Automation is eventually not going to report a lot of bugs, it’s rather enabling testers to capture more bugs, a great point from Angie.


Which coding language is recommended

In my opinion, more important than selecting a language is developing the aptitude of designing algorithms. Learn the fundamentals of programming which are common across all languages. Once you can develop an algorithm to solve problems, language selection becomes less important. Plus, languages will keep on changing and you can never stick to one language forever. If you start with learning a language per se for instance Java, you might get tangled up in syntax issues and debugging problems than actually learning to code.

A great point from Oren was about JavaScript, it’s a very easy language to begin with and also is very popular (in fact most popular) these days. More importantly for automation engineers I think it makes more sense since UI automation is usually a great part and using JavaScript allows synergies with using client side scripts and browser’s native functions as well. Usage of promises is also a very powerful concept mentioned by Oren which can certainly help with dynamic delays.


Retry failing tests

A solution to flaky tests is sometimes suggested as rerunning the flaky test. In multiple tries if it does not fail consistently, means there is no bug here rather just a script issue.

All of us at the panel thought retrying flaky tests is not a great idea. It adds lots of execution time and pushes away the concept of trying to avoid flakiness in the first place. Rather, avoid flakiness at all costs. There are a lot of ways to do that, use what works best for you. Apart from tips like using dynamic delays, using the AUT’s client side functions etc, try breaking the test into smaller tests. Longer scripts have a greater tendency to fail and cause problems.

The ultimate solution for flakiness is to have a great automation framework built to support the execution and handle unexpected states of the application.


How to elevate automation’s success to management

The goal of automation is to identify risks and inform of any potential issues going into the field. Unfortunately, many times we end up working on issues which might not directly translate to the value added towards this goal. It’s also hard to get matrices which reflect this, but we should not stop trying. The ‘Matric black hole’, a term from the book ‘deep work’, talks about this concept as well and about the complex nature of getting matrices on productivity which tend to be subjective and very hard to measure.

One of the values at Quality Spectrum is generating ‘business value’. This point eludes to that goal, to get tangible results proportionate to the effort done. It’s a long and complex road for which certainly we don’t have answers yet, but it is an important enough problem to solve. For starters, I feel measuring how much more test coverage we have is a better one. Not in the sense of number of tests executed, but how much of the code base have we tested. Tools from companies like SeaLights can provide you with what areas of the code has been executed while you have been doing your tests and what areas remain to be tested. Another angle can be from the user perspective of coverage. This topic deserves a separate post, so I’ll just pause the discussion here.

A great point from Oren was the value of seeing passing test coverage. Getting the assurance of certain tests passing does add to the confidence in the build and allows testers to focus on other areas. Seeing a green mark on your automation results does have value and should not be discarded.

Management for the most part does not have a clear idea on how to measure value from automation and often are looking at the wrong metric. This comment from Angie was so true, I too never recall anywhere management having a clear vision of how to assess automation’s success, unless the person at the top has been doing automation themselves.


If you did not have the opportunity to join the guild this year, I’d recommend doing that. It gives a broad idea of what’s going on in the community, introduction to lots of new concepts and tools which one might not know if they were as important and where to start.


A lot of questions went unanswered on the first day’s round table I was at for which I am in the process of creating video replies for. The answers are uploaded to this playlist.


If you have any questions you’d like me to talk about, send them to questions@quality-spectrum.com.


By | June 14th, 2017|Uncategorized|

The software quality assurance industry in general has been dragging along rest of the software industry. The formation of quality spectrum is to research and enhance testing practices. This book is a step in that direction, creating effective and efficient automation initiatives enhancing testing efforts for software product.

Why another automation blog?

There are many great ones out there, some we love to read and comment on. I find navigating these blogs really cumbersome. The relationships between posts are hard to find out. That’s where the idea of a “Blog Book” came in, a term we at Quality Spectrum have coined. Instead of just a blog, all posts collectively form a book like structure. Making navigation and getting a general sense of the topics discussed easier for the reader to digest.

Secondly, we intend to make this a community of people pooling in, fast forwarding the speed of ‘automation acumen’.

Contributing to the blog

Share your thoughts by commenting. We’d love to know your opinion on the topics. Eventually we hope thought leaders in our industry will help us evolve the automation spectrum and help us collectively create ‘the next generation tester’.