What emerging data pipeline design patterns and technologies should you know?

When designing the architecture, ask yourself- 1. How end users gonna use it ? Is it expected to be Multi-tenancy 2. Is near real-time , daily, weekly, monthly or yearly updations required ? 3. Granularity of data to be fetched, processed and stored. 4. Estimated volume of data in 1 iteration 5. What are Constants and variables for preparing the skeleton ? 6. Cost of Maintenance 7. Data Quality checks, Pipeline health checks , Notification alert system and Auditing

In the modern cloud world, we say the data can be brought to the cloud as it's generated/ready in the source system within the next few seconds/minutes. Yes, We have to solve the puzzle based on the following. 1. How are we gonna use it? 2.How much are we ready to pay for it? 3.How are these data will change/hold at the origin? 4.How do we handle the late arrival of the data sources in our logic? 5.How do we plan the production loads and the operation team? 6.How do we handle the deletion records at the source?

While Lambda's fusion of batch and stream layers offers versatility, its applicability hinges on your data's granularity, update frequency, and volume. Kappa, with its real-time promise, necessitates rigorous data quality checks, especially when addressing issues like late data arrival or source deletions. Ultimately, aligning the chosen architecture with both the data's temporal characteristics and the specific processing requirements ensures a robust and efficient pipeline.

Great coverage by fellow experts on patterns. In terms of technologies, seeing a lot of Databricks + Snowflake out there all of a sudden. Companies are clearly investing in Data Lakes and Cloud Analytics platforms as a bet for the future.

Both are highly relevant approaches to enable enterprises transform from application-centric to data-centric architectures, although the differences and pros/cons are a lot more nuanced. As a data architect, it's worth asking two questions: 1) How much does it cost to move data thru the pipelines? Sometimes moving processing to data, e.g. consuming data products where they reside (as in a data mesh), is much cheaper. 2) How much does it cost to operate and govern a data fabric / common data layer? Level of efforts for maintaining common data models, metadata standards, and APIs may grow exponentially for a large organization. In deciding which approach to take (and they are by no means mutually exclusive), org structure is a key factor.

Apache Airflow is so much more than just a data pipeline tool. It's a highly flexible and customizable orchestration platform buoyed by the structural core and foundation it's built on: Python. Multi-platform and multi-technology orchestration is something every IT organization can take advantage of. How did I help get Airflow into my organization? I demonstrated things that Airflow can help accomplish beyond just data flows and pipelines. I was able to integrate custom Operators and Sensors that allowed us to orchestrate between systems and applications that didn't have any formal Airflow integration. Not a Python shop? Neither were we.

We could also think of Dagster as light-weight orchestrator, easy to plugin and maintain. It treats data as an asset, and jobs maintain single straightforward way of tracking dependencies between the the assets. So when debugging, reviewing changes, or backfilling, we can have deep, asset-level lineage(data) that goes all the way to the source.

Delta Lake and Apache Iceberg are two of the most used open table formats which kickstarted the concept of having a data lakehouse. Open table formats are the component in a Data Lakehouse that structures your files in a certain way that allows them to be directly consumed in the data lake itself, saving the cost of having the data stored in a data warehouse. For instance, Delta Lake creates delta log JSON files that contain the latest atomic version of the table. The log entry also contains records of which data is appended, updated, or removed. Delta Lake also creates data files and checkpoints according to the logs formed. This allows users to read from the data lake while maintaining ACID-compliant principles of data warehouses.

Delta lake and iceberg are going to dominate the data engineering space. While delta lake is still popular and vastly used but iceberg in other hand has been going under lot of improvements and upgrades. Delta lake along with databricks is very much popular and has created developers life very easy and more agile. It has given developers to leverage their both SQL and programming language experience on large scale data sets for both real time and batch analytical process. Iceberg on the other hand has come a long way and big organisations using it in their intensive data pipelines. It has a vibrant community of active users and contributors. Pipeline using iceberg are more reliable and less resistant to change.

Apache Hudi is also an option apart from the delta and iceberg which can provide the similar capabilities with a great developer community.

Implementing MLOps strategies transformed our workflow. Previously, Data Scientists created models locally, packaged them with dependencies, and handed them to Data Engineers for infrastructure integration. Now, with tools like MLFlow or Vertex AI Pipelines, Data Scientists can code on a customizable machine, create containers, define pipelines, and trigger them via Airflow. This self-service approach accelerates production and frees up Data Engineers' time, enabling faster idea testing.

GraphQL was originally intended to make accessing JSON-type databases easier, as you can define the associated schemas with TypeScript that allow for both querying and (when allowed) mutating content. This approach has since spread to other kinds of database and datastores, including knowledge graphs. In this case, the schema language (increasingly based on the semantic language SHACL) can then generate both the query and mutator components dynamically, and can map content to internal RDF representations. As large language models continue to evolved, it's very likely that GraphQL will play a part in binding between deep learning solutions and other data sources.

Data Engineers working on real-time analytics and event-driven architectures will need to know that Apache Kafka is a valuable tool for these scenarios. Kafka is often used as a messaging system that follows the publish-subscribe pattern. Producers publish messages to Kafka topics, and consumers subscribe to topics to receive and process those messages. This pattern enables decoupling of producers and consumers, allowing for scalability and flexibility in data processing. Kafka can be used in conjunction with stream processing frameworks like Apache Flink or Apache Kafka Streams to perform real-time data transformations, enrichments, and aggregations. This enables data engineers to clean and prepare data as it flows through the pipeline.