alikhalid

About Ali Khalid

This author has not yet filled in any details.
So far Ali Khalid has created 356 blog entries.

Data quality in big data

By |2020-06-15T19:15:08+05:00June 15th, 2020|daily post|

One of the biggest challenge in #BigData projects is ensuring the ‘quality’ of data

Debugging ‘anomalies’ across #DataPipelines can be a nightmare. Certain teams end up spending more time debugging than actually coding new features.

That’s because usually there are no automated data quality checks in place to catch issues, and they have to trace back anomalies across huge amounts of data sifting through complex ETL processes.

With other development projects, the behavior is more predictive because inputs into the system are homogeneous.

For #BigData projects, there is no guarantee on the data we ingest & the how it get’s processed will always be accurate – the input is NOT homogeneous.

The solution to this: have #automated data quality checks running in #production across the data pipeline

#RedefiningSoftwareQuality #BigData #Testing #Automation

Waterfall vs Agile Project Management

By |2020-06-11T19:13:13+05:00June 11th, 2020|daily post|

Project management – waterfall vs agile:

– Project phase 1: 50% over budget & 6 months late
vs
– In X budget & Y time we’ve managed to prove benefit hypothesis works / doesn’t work

Prime difference is:

In waterfall we benchmark spend against ‘planned spend’ (which is ALWAYS WRONG)

In Agile we benchmark spend against ‘value created’

Also – Can you sniff the BIG WALL / Them vs Us in waterfall

And agile is more about we as a group managed to achieve this

#RedefiningSoftwareQuality #AgileTransformation #Agile

Git course in automation training

By |2020-06-07T19:59:47+05:00June 7th, 2020|daily post|

Automation training projects should have a course on Git.

Mostly engineers don’t prioritize to learn how to effectively collaborate on source control management tools.

This causes lack of support & maintenance on the code base.

Get in the habit of collaborating more on code (code reviews, adding comments, contributing).

Otherwise we don’t document design decisions, or its in a document buried somewhere and never used because hard to find when needed.

Technical debt

By |2020-06-07T20:05:51+05:00June 6th, 2020|daily post|

“We don’t want to invest in tech debt right now, need to churn out more features”

That’s like saying, we’re not going to clean the house for a month, it’s just a waste of time..

(FYI, not investing in automation is accumulating tech debt)

Data pipeline quality stages

By |2020-06-07T20:07:17+05:00June 5th, 2020|daily post|

Three common stages of a data pipeline:

– Ingestion into data lake. Curating data.
– Pushing into data warehouse, model data.
– Pull data from warehouse and build analytical models.

Can differ according to use case.

Data quality checks should happen before entering and within each stage.

Automate BDD scenarios on the service layer

By |2020-06-04T19:33:31+05:00June 4th, 2020|daily post|

Should a BDD test scenario only be automated on the UI?

Commonly the answer is yes, I have a different opinion.

Let’s say feature is to show total cost of items in a shopping cart along with the shopping cart icon.

Implementation changes will be at the front end & service layer.

I’d personally want to have unit tests for new methods, component tests on the end point and:

Service integration tests for the BDD scenario – Adding / removing products to the cart and seeing if the cart total gets changed – or whatever the scenario may be

No harm in adding an extra check to an existing UI test, but I’d refrain from having a whole new UI script just for this, rather have an exclusive one on the API.

Thoughts?

Combining Web & Mobile framework

By |2020-05-23T19:23:32+05:00May 23rd, 2020|daily post|

A recent question I got asked:

How to combine Web & mobile UI automation frameworks together?

Now there will be no one size fits all answer here, but here’s what my experience has been:

1. Most teams try to run every test through selenium, and then the same through mobile (Appium, XCUITest etc..)

And that is the biggest problem – don’t have a lot of tests running there, most of you testing should be completed on your services level

2. Depending on your product, there might not be a lot of synergies between the tests you have on web & mobile, keep an eye out

3. Keep your frameworks very modularized, every component can be reusable

4. The utils layer can be shared across both frameworks, can have it as a separate project, added to the web & UI ones

5. Similarly test data / configs should be in sharable format. My favorite is JSON structures & config files

6. Having both web & mobile tests within the same project might not be a bad idea either.

FYI – Google maintains one code repo for all products. But be careful, if you don;t structure your product well you’ll run into trouble – Make sure everything is loosely coupled

These are few on top of my head, anything else you can think of?

#RedefiningSoftwareQuality #TestAutomation #MobileAutomation #WebAutomation

Definition of Shift Left

By |2020-05-22T19:40:06+05:00May 22nd, 2020|daily post|

What does shift left really mean?

Here’s my understanding and what it means to implement it:

Any activity to help us prevent or identify issues BEFORE someone spends dedicated time testing the change – Shift identification of issues & risks as early as possible.

I’d implement this by doing the following:

1. Have three amigo sessions before coding

2. Practice TDD & have decent unit test coverage

3. The developer and any other person on the team together test the change on the developers machine

4. Have quality gates on the pull request pipeline including static analysis, unit tests & component test

By the time engineers testing the feature start exploring in detail, most of the issues should have been fixed already.

#RedefiningSoftwareQuality #ShiftLeft #ContinuousTesting #DevOps

Digital workplace

By |2020-05-21T19:45:53+05:00May 21st, 2020|daily post|

An important part of transformations is building technical enablers.

Initiatives around Digital Workspace were been worked on with a lot of vigor at Emirates Group IT for some time

As a result of all that effort we were able to transition smoothly to working from home across the landscape

I had always taken this for granted, being IT professionals lot of these things are common place us

However the magnitude and scale at which large scale enterprises operates, ability to pull this off on such a short notice is an achievement.

Realized this today in a CTO listening session where Microsoft’s head UAE region was sharing his experiences post COVID19.

Bottom line, if you want to truly transform your team, you’ll have to build the technical enablers they need.

#Transformation #DigitalWorkplace

Competence & clarity

By |2020-05-18T19:36:49+05:00May 18th, 2020|daily post|

Building self autonomous teams is a key change in transformations,

And a key element in there is how to delegate decision making?

Typically enterprises have large groups of people with layers upon layers of decision makers

Decisions and actual hands on work are done in two different groups

As mush possible, decision making has to be delegated, and here are the two key components:

1. Competency. Develop the teams technical skills to have the background to make those decisions.

2. Clarity. Access to information and ability to process, to allow teams to have all the background information they need to make decisions.

Both are easier said than done, but without them the transformation will fail.

#RedefiningSoftwareQuality #QualityTransformation #Transformation #HighPerformingTeams