Implementing Robust Data Quality Solution

DATA ENGINEERING

BACKGROUND

Today the success of every enterprise is directly linked to the data that it consumes for making its business decisions. No data is perfect and usage of poor quality data can lead to inaccurate and untimely decisions resulting in increased operational costs, loss of customers and negative brand image. Recent Gartner research has found that organizations believe poor data quality to be responsible for an average of $15 million losses per year. Enterprise data architectures are moving from an Extract-Transform-Load (ETL) to Extract-Load-Transform (ELT) strategy. This will result into accumulation of more and more data and at a much higher speed than ever before.

KEY DESIGN CHALLENGES:

• Quantity of data being collected for historical analysis can easily range from terabytes to petabytes.
• Speed of incoming data can be much higher due to real time reporting and analytic needs.
• Structure of incoming data can be in a variety of formats like CSV, XML, JSON etc.
• Configuration of data quality rules can take enormous time due to a high number of incoming data elements.
• Integration with downstream engines for process handshaking and communication.

SOLUTION

Traditional Approach: A typical approach to build a data quality solution involves collecting requirements from business users and then handing it over to technology team which in turn implements using database procedures or any scripting language. But to cater this kind of complexity at scale, the traditional approach wont suffice and can lead into longer implementation cycles and sub-optimal routines taking longer execution times. It may also have limitations in terms of handling non-standard data structures.

Building Robust Data Quality Solution involves handling these challenges in a cost effective and an efficient manner. Here are few things we suggest to keep in mind while designing.

• Build a framework that is driven by metadata rather than programming or scripting of rules.
• Automatically detect data elements and provide out of box rule configuration.
• Support for traditional systems like RDBMS and distributed file systems like Hadoop and others for large scale processing.
• Leverage technologies like Spark and Kafka to support high speed real time data processing.
• APIs for downstream integration for handshaking and communication.
• Reporting and Dashboarding capabilities for real time monitoring, tracking and auditing.