Introduction to data pipelines in 2 minutes:<\/p>\n
For analytics, AI products lots of data is needed – commonly dubbed as Big data<\/p>\n
To get the data needed, and in a usable format – we need to pass it through a lot of different data processing stages, a collection of which can be called data pipeline.<\/p>\n
Data pipelines have 3 main stages:
\n– Ingestion: Gather data from various sources in different formats
\n– Data hub & warehouse – Cleanse, model data at different stages
\n– Analytics – Run analytics or use specific data sets for machine learning algorithms<\/p>\n
Join me at TestBash New Zealand Online 2020 where I’ll talk about a lot more around testing in big data projects!<\/p>\n<\/div>![](\"http:\/\/quality-spectrum.com\/wp-content\/uploads\/2020\/10\/QsDataPipelines-e1603809721800.png\" "\\"QsDataPipelines\\"")
<\/span><\/div>
<\/div>
<\/div>
<\/div><\/div><\/div><\/div><\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[52],"tags":[],"yoast_head":"\n
2 minute intro to Data pipelines - Quality Spectrum<\/title>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/quality-spectrum.com\/2-minute-intro-to-data-pipelines\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"2 minute intro to Data pipelines - Quality Spectrum\" \/>\n<meta property=\"og:url\" content=\"http:\/\/quality-spectrum.com\/2-minute-intro-to-data-pipelines\/\" \/>\n<meta property=\"og:site_name\" content=\"Quality Spectrum\" \/>\n<meta property=\"article:published_time\" content=\"2020-10-27T14:42:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-10-28T15:11:32+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@aali_khalid\" \/>\n<meta name=\"twitter:site\" content=\"@aali_khalid\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"http:\/\/quality-spectrum.com\/#organization\",\"name\":\"Quality Spectrum\",\"url\":\"http:\/\/quality-spectrum.com\/\",\"sameAs\":[\"https:\/\/www.linkedin.com\/in\/alikhalid\/\",\"https:\/\/www.youtube.com\/c\/QualitySpectrum\",\"https:\/\/twitter.com\/aali_khalid\"],\"logo\":{\"@type\":\"ImageObject\",\"@id\":\"http:\/\/quality-spectrum.com\/#logo\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/quality-spectrum.com\/wp-content\/uploads\/2019\/11\/QS-logo-mobile-e1574510459832.png\",\"width\":40,\"height\":40,\"caption\":\"Quality Spectrum\"},\"image\":{\"@id\":\"http:\/\/quality-spectrum.com\/#logo\"}},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/quality-spectrum.com\/#website\",\"url\":\"http:\/\/quality-spectrum.com\/\",\"name\":\"Quality Spectrum\",\"description\":\"Redefining software quality\",\"publisher\":{\"@id\":\"http:\/\/quality-spectrum.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"http:\/\/quality-spectrum.com\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"http:\/\/quality-spectrum.com\/2-minute-intro-to-data-pipelines\/#webpage\",\"url\":\"http:\/\/quality-spectrum.com\/2-minute-intro-to-data-pipelines\/\",\"name\":\"2 minute intro to Data pipelines - Quality Spectrum\",\"isPartOf\":{\"@id\":\"http:\/\/quality-spectrum.com\/#website\"},\"datePublished\":\"2020-10-27T14:42:17+00:00\",\"dateModified\":\"2020-10-28T15:11:32+00:00\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/quality-spectrum.com\/2-minute-intro-to-data-pipelines\/\"]}]},{\"@type\":\"Article\",\"@id\":\"http:\/\/quality-spectrum.com\/2-minute-intro-to-data-pipelines\/#article\",\"isPartOf\":{\"@id\":\"http:\/\/quality-spectrum.com\/2-minute-intro-to-data-pipelines\/#webpage\"},\"author\":{\"@id\":\"http:\/\/quality-spectrum.com\/#\/schema\/person\/4805a00d7139e111ea9430e17cc8f28c\"},\"headline\":\"2 minute intro to Data pipelines\",\"datePublished\":\"2020-10-27T14:42:17+00:00\",\"dateModified\":\"2020-10-28T15:11:32+00:00\",\"mainEntityOfPage\":{\"@id\":\"http:\/\/quality-spectrum.com\/2-minute-intro-to-data-pipelines\/#webpage\"},\"commentCount\":0,\"publisher\":{\"@id\":\"http:\/\/quality-spectrum.com\/#organization\"},\"articleSection\":\"daily post\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"http:\/\/quality-spectrum.com\/2-minute-intro-to-data-pipelines\/#respond\"]}]},{\"@type\":[\"Person\"],\"@id\":\"http:\/\/quality-spectrum.com\/#\/schema\/person\/4805a00d7139e111ea9430e17cc8f28c\",\"name\":\"Ali Khalid\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"http:\/\/quality-spectrum.com\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/1.gravatar.com\/avatar\/70cbf539f218f275a77959dd2e56bddb?s=96&d=mm&r=g\",\"caption\":\"Ali Khalid\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/posts\/15655"}],"collection":[{"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/comments?post=15655"}],"version-history":[{"count":5,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/posts\/15655\/revisions"}],"predecessor-version":[{"id":15665,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/posts\/15655\/revisions\/15665"}],"wp:attachment":[{"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/media?parent=15655"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/categories?post=15655"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/quality-spectrum.com\/wp-json\/wp\/v2\/tags?post=15655"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}