One of the biggest challenge in #BigData<\/strong> projects is ensuring the ‘quality’ of data<\/p>\nDebugging ‘anomalies’ across #DataPipelines<\/strong> can be a nightmare. Certain teams end up spending more time debugging than actually coding new features.<\/p>\nThat’s because usually there are no automated data quality checks in place to catch issues, and they have to trace back anomalies across huge amounts of data sifting through complex ETL processes.<\/p>\n
With other development projects, the behavior is more predictive because inputs into the system are homogeneous.<\/p>\n
For #BigData<\/strong> projects, there is no guarantee on the data we ingest & the how it get’s processed will always be accurate – the input is NOT homogeneous.<\/p>\nThe solution to this: have #automated<\/strong> data quality checks running in #production<\/strong> across the data pipeline<\/p>\n#RedefiningSoftwareQuality<\/strong> #BigData<\/strong> #Testing<\/strong> #Automation<\/strong><\/p>\n<\/div><\/div><\/div><\/div><\/div><\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[52],"tags":[],"yoast_head":"\n
Data quality in big data - Quality Spectrum<\/title>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/quality-spectrum.com\/data-quality-in-big-data\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Data quality in big data - Quality Spectrum\" \/>\n<meta property=\"og:url\" content=\"http:\/\/quality-spectrum.com\/data-quality-in-big-data\/\" \/>\n<meta property=\"og:site_name\" content=\"Quality Spectrum\" \/>\n<meta property=\"article:published_time\" content=\"2020-06-15T14:15:08+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@aali_khalid\" \/>\n<meta name=\"twitter:site\" content=\"@aali_khalid\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"http:\/\/quality-spectrum.com\/#organization\",\"name\":\"Quality Spectrum\",\"url\":\"http:\/\/quality-spectrum.com\/\",\"sameAs\":[\"https:\/\/www.linkedin.com\/in\/alikhalid\/\",\"https:\/\/www.youtube.com\/c\/QualitySpectrum\",\"https:\/\/twitter.com\/aali_khalid\"],\"logo\":{\"@type\":\"ImageObject\",\"@id\":\"http:\/\/quality-spectrum.com\/#logo\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/quality-spectrum.com\/wp-content\/uploads\/2019\/11\/QS-logo-mobile-e1574510459832.png\",\"width\":40,\"height\":40,\"caption\":\"Quality Spectrum\"},\"image\":{\"@id\":\"http:\/\/quality-spectrum.com\/#logo\"}},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/quality-spectrum.com\/#website\",\"url\":\"http:\/\/quality-spectrum.com\/\",\"name\":\"Quality Spectrum\",\"description\":\"Redefining software quality\",\"publisher\":{\"@id\":\"http:\/\/quality-spectrum.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"http:\/\/quality-spectrum.com\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"http:\/\/quality-spectrum.com\/data-quality-in-big-data\/#webpage\",\"url\":\"http:\/\/quality-spectrum.com\/data-quality-in-big-data\/\",\"name\":\"Data quality in big data - Quality Spectrum\",\"isPartOf\":{\"@id\":\"http:\/\/quality-spectrum.com\/#website\"},\"datePublished\":\"2020-06-15T14:15:08+00:00\",\"dateModified\":\"2020-06-15T14:15:08+00:00\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/quality-spectrum.com\/data-quality-in-big-data\/\"]}]},{\"@type\":\"Article\",\"@id\":\"http:\/\/quality-spectrum.com\/data-quality-in-big-data\/#article\",\"isPartOf\":{\"@id\":\"http:\/\/quality-spectrum.com\/data-quality-in-big-data\/#webpage\"},\"author\":{\"@id\":\"http:\/\/quality-spectrum.com\/#\/schema\/person\/4805a00d7139e111ea9430e17cc8f28c\"},\"headline\":\"Data quality in big data\",\"datePublished\":\"2020-06-15T14:15:08+00:00\",\"dateModified\":\"2020-06-15T14:15:08+00:00\",\"mainEntityOfPage\":{\"@id\":\"http:\/\/quality-spectrum.com\/data-quality-in-big-data\/#webpage\"},\"commentCount\":0,\"publisher\":{\"@id\":\"http:\/\/quality-spectrum.com\/#organization\"},\"articleSection\":\"daily post\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"http:\/\/quality-spectrum.com\/data-quality-in-big-data\/#respond\"]}]},{\"@type\":[\"Person\"],\"@id\":\"http:\/\/quality-spectrum.com\/#\/schema\/person\/4805a00d7139e111ea9430e17cc8f28c\",\"name\":\"Ali Khalid\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"http:\/\/quality-spectrum.com\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/1.gravatar.com\/avatar\/70cbf539f218f275a77959dd2e56bddb?s=96&d=mm&r=g\",\"caption\":\"Ali Khalid\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/posts\/15285"}],"collection":[{"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/comments?post=15285"}],"version-history":[{"count":1,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/posts\/15285\/revisions"}],"predecessor-version":[{"id":15286,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/posts\/15285\/revisions\/15286"}],"wp:attachment":[{"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/media?parent=15285"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/categories?post=15285"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/tags?post=15285"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}