How can you ensure data interoperability and compatibility?

Establish a clear taxonomy and glossary of terms appropriate for your business and industry. Consider adopting established taxonomies like the Banking Industry Architecture Network (BIAN), Australian Securities and Investments Commission (ASIC), or Financial Information eXchange (FIX) in the financial industry.

The data standards are often consumed by developers, product managers and other stake holders in the business. The very first step I recommend any data leaders to do is UNDERSTAND the business before defining the standards. The second thing is consider using a tool ( OpenMetadata, AVRO etc. ) for storing data standards, this can make life simpler moving ahead because you shoot two birds with one stone; 1. lay down path for automation 2. define standards.

Agree on the data quality standards. e.g. A given field should never be null. Identify all the sources to make a permanent fix. It may be a change in the UI screen, deriving from other fields etc. Data orgs should invest in classifying data sources into system of records and derived ones . Within each business unit that generates data, there will be an application which is considered the source. All other files/streams/tables, that fully or partially use this data are deriving from this source. Having this distinction will help maintain the quality of this data source and there by quality cascades downstream. Trying to fix the downstream while the upstream is polluted is a losing game.

Think of data standards as the superheroes of the data world, setting the rules for data to play nicely. It's like giving each piece of data a superhero suit with clear instructions on how to behave. When data follows these standards, it's like our heroes working together seamlessly, making data integration and analysis a breeze – no capes needed!

Data modelling is the most important aspect of working with data. Recently, it has been hidden away in discussions of new and shiny tools. I would suggest going for Data Vault 2 or Anchor if you are migrating to a new DWH or just building it from scratch, since these two allow to build it iteratively. Furthermore, with these, you can effortlessly mutate your marts in accordance with ever-changing business demands. Other modelling options work too, they would just require more premeditation and careful planning. What you don't want, however, is to do ad-hoc modelling, because there will be a time when you will regret it - sooner, rather than later. Properly modelled and documented data is hard, but worth it for the whole organization.

Remember that data models will be used beyond your data systems; in business logic, in API, in data pipeline. Collaborate with business experts and don't make your columns sounds wildly different from the business definition. Be careful with vague words like account, transaction, check point which can have very different meaning in changing context. DBT, dbdocs ... these tools can be very effective that can help you define common data models while making it visually appealing and easy to consume by both machines and human.

Common data models are like the universal translators of the data world. They ensure that data speaks the same language, making it easier for different systems to understand and work with it. It's like having a magical dictionary that everyone in the data universe can consult, ensuring smooth communication and collaboration among data sources and transformations.

Data Vault makes you to build the model iteratively and also with more M&A being carried out , this would come in handy . Also Iteration is something which i like. Data Vault gives more flexibility

Data pipelines are best when they are idempotent - switch the underlying technology, and they should still run, do the needed transformations and load data wherever it needs to go. In the olden days, a lot of it was done through stored procedures - poorly documented, barely tested, written in pl\sql. The whole logic was placed on the database, which scaled terribly and had a whole lot of other associated issues. Don't do it. Pipelines can be written in Scala, Java, Rust, Python, SQL - essentially, whatever you want. Scala is on its way out, Java is fast, Python is friendly. SQL, though, will always be there. More important than language is to keep pipelines documented tidy,DRY, SWE have already travelled this road: learn from their labor

Data pipelines are like the traffic controllers of the data highway. They keep data flowing smoothly, guiding it from various sources to destinations. Think of them as digital maestros orchestrating a symphony of data, ensuring it arrives on time and in the right format. With data pipelines, you turn the chaos of data diversity into a harmonious melody of interoperability and compatibility.

3 important aspect of data pipeline one should thing about on top of what is already mentioned. 1) keep it loosely coupled - why ? you don't want your downstream issues to affect business logic and you also want to swap technologies without touching your business logic. How ? use Kafka or other technologies that provides fast write with no data loss guarantee 2) no data loss guarantee - no leaks. Use message queues 3) extensible and reusable - The pipeline should be extensible to add more data fields without huge change and ready to add more data stores if requries ( i.e. time series database or other fast datastores for caching purposes

Data pipelines, in essence, act as the arteries of a data ecosystem, ensuring a smooth and efficient flow. It's not just about transporting data but about doing it smartly - by filtering out noise, ensuring the fidelity of data, and transforming it into actionable insights. It's also noteworthy that the design and maintenance of these pipelines are as crucial as their implementation. A poorly designed pipeline can lead to data bottlenecks, inaccuracies, or even losses. Furthermore, in today's rapidly evolving digital landscape, the flexibility of these pipelines to adapt to new data sources or changes in existing sources is invaluable.

Pipelines need orchestration, hence, the orchestrator: a tool that will run your pipelines, retry the failed ones, notify you when something went wrong, keep logs of how the jobs performed and so on. Airflow has the most know-hows accumulated and a big community support. Dagster is on the rise, wherein vertices in DAGs are data assets, not operations - making it perfect candidate for DE tasks. In the most basic form, that could be enough. However, oftentimes you may need an execution engine to do the hard job: Spark cluster, duckdb, Polars, Ballista, dbt, Kafka. Selection would depend on whether you have a Lake, Lakehouse or DWH and what language you prefer. Tools are a plenty: don't be bogged down by sheer amount of choices!

Data tools are like Swiss Army knives for data engineers. They come packed with gadgets to slice, dice, and manipulate data with finesse. It's like having a toolbox filled with magic wands that transform data into a standardized, compatible format. With these tools, data engineering becomes a thrilling journey where you're the wizard, conjuring up data spells to ensure interoperability and compatibility in your data realm.

Data formt : Avro, protobub and JSON with each of it's straight on flexibility, support for tooling, schema enforcement at the source etc. E.G. AVRO and ProtoBuf allows you to use Kafka Connect which makes it much easier and straight forward to use exactly once semantic for data pipeline which is huge but it enforces strict schema enforcement at the source which could be challenge in some cases. AVRO has worked well in past for me as schema standard but ProtoBuf and OSS tools like OpenMetadata has improved it lately. For near real time streaming Kafka has been a defacto standard and others like RedPanda and Pulsar have adopted it as well.

The right data tools not only streamline processes but also serve as a bridge between diverse data sets, ensuring seamless communication and integration. One thing worth noting is that as the landscape of data engineering evolves, so does the ecosystem of data tools. Therefore, it's vital for organizations to stay updated with the latest advancements and adopt tools that best suit their specific needs. Moreover, while tools can be incredibly powerful, the human element – in terms of understanding the intricacies of the data and ensuring correct implementation – remains irreplaceable. Combining the strengths of advanced data tools with skilled professionals can significantly boost an organization's data interoperability efforts.

There are often too many moving parts in data pipeline but the most basic one to monitor is 1) don't lose your data ( your design and tooling of choice must guarantee 'at least once' and if possible 'exactly once' semantic to ensure no data loss. The standard is to follow classic RED metric ( Request Rate, Error Rate and Duration ) but if you're using a message queue with right architecture, message backlog is a must for monitoring. Also ensuring schema driven design with centralized schema can ensure data interoperability issues if you're starting from clean slate.

A reliable method to verify data accuracy is by feeding any aggregated data back into the originating application maybe for another use case (like reporting). This allows the source application's owner, who has the deepest understanding of the data, to easily detect any anomalies or inconsistencies.

Testing and monitoring data is the vigilant guardian of data quality and performance. It's the compass that keeps your data journey on course. By systematically checking and tracking data against standards and expectations, you ensure that your data remains a reliable and interoperable asset, driving confidence in your data-driven endeavors.

Testing is paramount if you want stakeholders to have any sort of trust in the data. The most robust ML model, the shiniest visual report, the most perfectly written pdf mean nothing if built on the dirty data. Garbage in - garbage out. Test can be purely technical - that data exists, all required attributes are there and so on. Arguably, the most important ones are the business checks - if data adheres to the business expectations, it gets validated with business owners, DE often just implements what was said. Ideally, after each transformation step in a pipeline, you'd want to check if results are what you were expecting. Data Observability rarely checks for specifics, rather, it shows a holistic overview and anomalies, if there's any.

Specifically related to DQ, technical folks won't have the domain knowledge of the data. Hence, they are not capable of writing comprehensive quality checks - only technical ones. Communication with business is essential, because they know what data actually means and what attributes are important, they have expectations that are for you to fulfil. Collaboration with data analysts is also important, because business utilizes data through them. DE feeds the data to DA, DS and, essentially, everyone else. Don't close yourself out and do data work for yourself, do it for people who will use it. This will help to define all the definitions and setting of your pipeline in such a way to benefit the whole organization.

There are typically 3 groups; 1) data producers ( application teams, product managers ) 2) data governors ( data team ) 3) data users ( everyone else but domain specific users marketing, accounting etc) Data team should define standards and operate on loosely defined rule that is progressively enforced by tooling. The others should collaborate on domain specific data when introduced or refined.

Simply talking to stakeholders isn't enough. As data leaders, we need to talk about the data lifecycle early on, even before we discuss the technical details.

Collaboration and communication are like the secret sauce in the data engineering recipe. They bring all the ingredients together, making the data dish not just tasty but also perfectly balanced. It's like hosting a data dinner party where everyone shares their favorite recipes (insights) and makes sure the meal (data) is a delightful experience. When you communicate openly and collaborate effectively, you create a harmonious data environment, ensuring that everyone speaks the same data language and dines on compatibility and trust.

To achieve data interoperability and compatibility: 1. Semantic Data Modeling: Employ semantic data models and ontologies to define the meaning and relationships between data elements. Semantic models enhance understanding and compatibility by establishing a common vocabulary. 2. Data Exchange Agreements: When sharing data with external partners, establish clear data exchange agreements that define data formats, standards, and protocols to ensure compatibility between organizations. 3. Data Governance and Stewardship: Establish data governance practices and assign data stewards responsible for overseeing data standards, quality, and compatibility. Effective change management ensures that updates do not disrupt data interoperability.

While this article covers some of the critical aspects, some other aspect I have experienced and recommend are: 1. Scalability: Its important to recognize that implemented solutions not only meet current data demands but also adaptable to future growth, providing a scalable foundation. 2. Versioning: Implement versioning for data structures and APIs to manage changes effectively. Helps in maintaining backward compatibility and smooth transitions during updates. 3. Documentation: Not having good documentation can be a real headache! I worked with a company whose data team struggled because there wasn't info on how their systems worked. Connecting old tools to new ones ended up being a costly mess in terms of time and money.

The points raised touch on key areas, but we frequently overlook: 1. Semantic Interoperability: It's essential to recognize that the meaning of certain 'keys' might differ across various contexts or systems. This discrepancy frequently arises when systems aren't originally designed with data engineering principles in mind. 2. Data-Centric Approach: Data engineering should be central to design and architecture, not an afterthought. 3. Simplicity and Accessibility: Ensure that data processes are straightforward enough for even non-technical team members to grasp. By understanding the origin and derivation of aggregated data, compatibility can be more swiftly and sustainably improved.