alikhalid

About Ali Khalid

This author has not yet filled in any details.
So far Ali Khalid has created 291 blog entries.

QA&TEST 2019 – Pro Tester using Fault Injection

By | December 13th, 2019|Uncategorized|

Since the start of my career as a tester, I’ve always been obsessed with testers delivering tangible value to the business. I realized ‘redefining software quality’ was my calling. There were quite a few powerful stories that lead to this realization, and one of the most important ones I presented at the QA&TEST 2019 conference.

This was a story about testers taking responsibility and recognizing how important it is to understand the underlying technology of the product. From this event emerged the three core values I feel which can reshape how we think about software quality.

This talk was awarded the best presenter award at the QA&TEST 2019 conference,

 

Award ceremony video here.

I thoroughly enjoyed the conference, the city and the hospitality. I talked about it in this article and a short shout out for the organizers on a job well done.

Presentation key points

Out of scope

The products being very complex, it was impossible to test everything through black box testing. Areas which the testers were unable to understand how they work, or there was no way to test them, were not considered as the tester’s responsibility.

While quality is everyone’s responsibility, but testers do need to make sure all bases are covered. It isn’t necessary that everything must be tested by a tester, but a tester should make sure that ‘someone’ is testing all the areas that should be covered.

 

Code coverage

Code coverage is not the only measure to check, but certainly is one of the important ones. Unless we are sure all statements and branches of the code have been executed and tested, we cannot be sure of all the hidden risks. With just black box exploratory testing, it’s almost impossible to figure out the code coverage % even is, let alone having a decent code coverage.

In our story here that was the case, almost zero visibility into the code, some functionality had been going untested. Neither was there any effort in figuring out how to improve that coverage.

 

Fault injection

In pursuit to improve coverage we came with a method to do fault injection / mutation testing. Instead of controlling the input parameters from outside, alter the control flow within the program to reach the desired state.

Technological excellence

Without knowing the technical details of how the product works, it’s very difficult to figure out the risk areas and to learn how to test them efficiently. A good understanding of product architecture and what areas are high risk is key.

Testers MUST put in effort to learn the technology. Without it, IMHO one cannot be a good tester.

Testing Acumen

Testing is more about thinking and collaborating than doing and checking. Learn to think how to test and learn / develop models on how to test. Unfortunately most testers intuitively learn to test well and don’t follow a systematic way to enhance that skill.

Redefining Software Quality

The most valuable learnings for me were the importance of testers to learn technology, enhanced methods to identify risk and ability to provide value in the context business.

Definition of flaky tests

By | October 23rd, 2019|daily post|

Anyone who has written automation scripts has faced flakiness, and we all hate it to our bone.

But everyone defines a flaky test a bit differently..

For me, a flaky test definition could be “a test running multiple times under the same conditions gives different results”

How would you define flakiness?

#QsDaily #automation #flakytests

Automation without test strategy

By | October 19th, 2019|daily post|

Mostly when companies talk about transforming, automation come up as the most hot topic.

While that certainly is important, without your quality strategy & foundations it’s not going to work.

Immediately people start running towards buzz words and new shiny tools.

Knowing what quality means for your product, what to test and how to test should precede any automation effort.

If you used so ship garbage at a slow pace, with new shiny tools and terms it’s going to garbage coming out double time..

#QsDaily #Automation #Testing

People over process

By | October 16th, 2019|daily post|

Ever been in a working environment with a lot of processes?

While I agree ‘some’ of them are needed, but mostly they become shackles and hamper progress.

It’s a fine line sometimes to make a process which is effective and efficient at the same time,

In case of doubt, my personal preference is always to make it more leaner.

After all, that’s what the manifesto says, people over process.

Many organizations begin with the assumption employees are unable to make right decisions

Therefore, there they need ‘processes’ to ensure they operate as expected.

The problem: it’s impossible to come up with a ‘perfect’ process,

So instead of a rigid process, have general guidelines and have faith the people will make the right decision.

#QsDaily #processes #WaysOfWorking

Automation is meant to help testers

By | October 15th, 2019|daily post|

I hate when testers doing exploratory testing don’t look at automation run results…

This mentality of us vs them among Automation engineers vs Exploratory testers has to go

And I wouldn’t blame testers only for this, when management feels automation will help them get rid of manual testers, It’s only natural for testers to feel insecure

For the 100th time AUTOMATION IS MEANT TO HELP TESTING!!!!

A common question I ask teams – If I stop automation runs today, will that change the time you’ll take for exploratory testing?

If the answer is no, i.e. with or without automation our testing time remains the same, then WHY IN THE WORLD ARE WE DOING AUTOMATION!!!

#QsDaily #CommonSense #Automation #Testing #regression

Big data and data quality

By | October 14th, 2019|daily post|

Testing in big data area has typical challenges,

A big factor is quality of data ingested.

The analysis results has a heavy dependency on the ‘quality’ of data ingested (obviously),

What often happens is inconsistency across the data from down-stream sources, missed records, missing data within records etc and other data quality issues.

Unless these issues are flagged at the lower levels, problems creep up and start reflecting in the analytics results.

While having automated data checks on massive data stores might not be an easy job, it certainly is worth implementing.

Automation Regression Percentage

By | October 12th, 2019|daily post|

Have you ever calculated Automation Regression Percentage?

I think it’s one of the most misused metric. Mostly used to show ‘cost savings’ on pretty graphs.

Here’s how to use the metric get some actual value:

#QsDaily #Automation #Regression #Testing #AutomationRegression

Give me a fixed date to automate

By | October 9th, 2019|daily post|

A client asked me to ‘Quote a budget and timeline for completely ‘automating testing’ while giving no context of the product.

I refused to work with them (obviously), here’s why:

Even if disregard the ‘automating testing’ ask, which is a BIG ask..

Getting a budget and time estimation of a complete project might be a common ask before any project sign off.

But I consider this the biggest anti pattern to agile.

Decades of practical experience has showed us we are horrible at estimating.

On the other hand I understand we need to budget and calculate what return we might get.

What might be more practical is to have a ball park range for budget and work we can deliver in first three months.

You can extrapolate and calculate the total estimated cost and use for long term budget planning, but remember it’s always going to be wrong.

We get so hung up with the detailed plans & good old

gantt charts (which were invented in WWI) and forget the ground reality.

#QsDaily #projectestimation #agile

Fault injection post Agile

By | October 7th, 2019|daily post|

Fault injection / mutation testing is among the things I miss from the old waterfall days.

More on fault injection in my talk at QA&TEST conference this month end.

In waterfall days it was okay to spend time on improving testing and coming up with better testing techniques.

After the agile apocalypse, many teams found excuses to get away with perfecting their craft under the inaccurate pretense of urgency.

In my understanding, agile never meant not to get half cooked stuff out of the door, in fact quite the opposite, important activities should be part of the Definition of Done.

The only difference was to break a big project into very small deliverable.

So essentially all the good practices from before should continue, just implement in bite sized pieces.

Fault injection is one of those practices.

#QsDaily #Agile #Testing #MutationTesting

Fault Injection at QA&TEST Conference

By | October 2nd, 2019|daily post|

I am furious on hearing a tester saying ‘this is out of scope’.

I’ll be talking about one way to solve this at QA&TEST this month (https://www.qatest.org/ali-khalid/?lang=en).

Most of the time I’ve seen testers say this because they don’t know how to test that feature.

Instead of trying to go under the hood and understand the architecture and code base, they just raise their hands.

What really should be done is understand how that piece of code can be tested.

Yes developers should be doing unit tests, but even then tests on module / component level should be figured out

In the world of embedded systems and especially safety critical devices, letting a functionality go untested is not in the cards.

Using fault injection / mutation testing is one technique which I’ll be discussing at the conference.