What are some common data quality checks and validations to implement in your pipelines?

1 Data Completeness

Data completeness refers to the extent to which the data covers the expected scope and volume. For instance, if you are ingesting data from a source system, it is important to ensure that all records and fields have been received with no missing or duplicate values. Data completeness checks can be used to verify that data has not been corrupted, truncated, or filtered during the extraction, transformation, or loading process. Such checks involve counting the number of records and fields in both the source and target systems to ensure consistency, checking for null, empty, or default values in fields that should have valid data, and checking for duplicates or outliers which could indicate data duplication or loss.

4 Data Timeliness

Data timeliness refers to the extent to which the data is available and accessible within the expected time frame. For instance, when delivering data to a dashboard, you want to ensure that the data is updated and refreshed at the required frequency and latency. To this end, data timeliness checks can help you monitor and optimize your data pipeline performance, as well as meet your service level agreements. Examples of such checks include tracking the data ingestion, processing, and delivery time stamps in comparison with the expected schedule and deadline; measuring the data pipeline throughput, latency, and availability in comparison with the expected benchmarks and thresholds; and identifying and resolving any data pipeline bottlenecks, failures, or delays while notifying stakeholders if necessary.

5 Data Relevancy

Data relevancy refers to how well the data meets the needs and expectations of the data consumers and users. For instance, if you are providing data to a machine learning model, you want to make sure that the data is relevant, representative, and sufficient for the model training and evaluation. To assess and improve your data value and usability, you can perform data relevancy checks. These checks involve evaluating the data quality dimensions and metrics based on the data use cases and requirements, profiling and analyzing the data distribution, patterns, and trends in order to identify any gaps, biases, or noise, as well as collecting and incorporating feedback from the data consumers and users.

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?