Apache Hudi

Apache Flink + Apache Hudi 🚀 Apache Flink provides features such as event-time processing, exactly-one semantics & diverse windowing mechanisms that makes it an excellent choice for streaming workloads. Flink when paired with lakehouse formats like Apache Hudi enables building low-latency data platforms by consuming data from various sources such as RDBMS, Kafka (DB-> Debezium CDC), etc. Beyond just table formats, Hudi offers robust table & platform services, enhancing Flink to support a real-time lakehouse architecture. Hudi was built was on the primitives of streaming workloads, which makes it a natural choice for these sort of use cases. Let's take a look at some of the common use cases for (Hudi + Flink) and how Hudi's unique capabilities adds value. ✅ Streaming Ingestion with Changelog: Use Flink’s CDC connectors or Kafka message queues to capture changes (inserts, updates, and deletes) from source databases and persist them in Hudi tables, enabling real-time streaming ingestion. ✅ Incremental ETL Pipeline: Combine Flink’s dynamic tables with Hudi’s capabilities for sequence preservation, row-level updates & file-sizing (compact) to build incremental ETL pipelines, allowing efficient processing of only changed data. ✅ Incremental Materialized View: Ingest and compute data using Flink, then materialize the final results in Hudi tables. Post that, you can query with other engines in your architecture. I linked a talk from this year Current (Confluent on how you can apply these use cases and learn about the internals of the Flink-Hudi integration. #dataengineering #softwareengineering

Have you checked out the January month Hudi newsletter? From community meetups, amazing blogs to project updates, learn what's happening in the Hudi community. Thanks to all the contributors! Link: https://lnkd.in/dEdu6cTg #dataengineering #softwareengineering

🚨 Upcoming: Episode 6 of "Lakehouse Chronicles with Apache Hudi" Join us for this episode where we will have Sagar Lakshmipathy, go over Change Data Capture (CDC) and its integration with data lakehouse, their benefits, implementation methods, key technologies and tools involved, best practices, and how to choose the right tools for your needs. Date/Time: 🗓️ February 20th (Thursday) at 9 AM Pacific Time. Link: 👉 https://lnkd.in/dpAa8xq5 #dataengineering #lakehouse

𝗨𝗻𝗹𝗼𝗰𝗸𝗶𝗻𝗴 𝗜𝗻𝗰𝗿𝗲𝗺𝗲𝗻𝘁𝗮𝗹 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗼𝗻 𝗗𝗮𝘁𝗮 𝗟𝗮𝗸𝗲𝘀 𝘄𝗶𝘁𝗵 𝗔𝗽𝗮𝗰𝗵𝗲 𝗛𝘂𝗱𝗶 The rise of data lakes has revolutionized how organizations store and manage vast amounts of data. However, traditional data lakes often struggle with real-time analytics and incremental data processing. Apache Hudi emerges as a powerful solution, enabling efficient incremental processing on data lakes while ensuring data freshness and reliability. 𝗧𝗵𝗲 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 𝘄𝗶𝘁𝗵 𝗧𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗗𝗮𝘁𝗮 𝗟𝗮𝗸𝗲𝘀 Traditional data lakes operate on batch processing principles, which means updates, deletes, and incremental changes require expensive full-table scans. This results in inefficiencies in data freshness, processing costs, and latency, making it challenging to support real-time use cases. 𝗘𝗻𝘁𝗲𝗿 𝗔𝗽𝗮𝗰𝗵𝗲 𝗛𝘂𝗱𝗶 Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data lake framework that brings database-like capabilities to data lakes. It enables efficient upserts, deletes, and incremental processing, helping organizations unlock real-time analytics at scale. 𝗞𝗲𝘆 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀 𝗼𝗳 𝗔𝗽𝗮𝗰𝗵𝗲 𝗛𝘂𝗱𝗶 𝗳𝗼𝗿 𝗜𝗻𝗰𝗿𝗲𝗺𝗲𝗻𝘁𝗮𝗹 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗖𝗵𝗮𝗻𝗴𝗲 𝗖𝗮𝗽𝘁𝘂𝗿𝗲 𝘄𝗶𝘁𝗵 𝗜𝗻𝗰𝗿𝗲𝗺𝗲𝗻𝘁𝗮𝗹 𝗤𝘂𝗲𝗿𝗶𝗲𝘀: Unlike traditional batch-based ETL jobs, Hudi enables querying only the changed data since the last checkpoint. This significantly improves efficiency and reduces processing overhead. 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗨𝗽𝘀𝗲𝗿𝘁𝘀 𝗮𝗻𝗱 𝗗𝗲𝗹𝗲𝘁𝗲𝘀: Hudi supports record-level updates and deletes, ensuring data consistency while maintaining performance. 𝗧𝗶𝗺𝗲 𝗧𝗿𝗮𝘃𝗲𝗹 𝗮𝗻𝗱 𝗦𝗻𝗮𝗽𝘀𝗵𝗼𝘁 𝗜𝘀𝗼𝗹𝗮𝘁𝗶𝗼𝗻: With Hudi’s timeline-based capabilities, users can perform historical data queries and rollback changes when needed. 𝗦𝗲𝗮𝗺𝗹𝗲𝘀𝘀 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻: Hudi works with popular big data processing engines like Apache Spark, Presto, Hive, and Flink, making it easy to integrate into existing data architectures. 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲𝘀 𝗳𝗼𝗿 𝗔𝗽𝗮𝗰𝗵𝗲 𝗛𝘂𝗱𝗶 𝗦𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 𝗗𝗮𝘁𝗮 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀: Companies dealing with high-velocity data can use Hudi to process real-time updates efficiently. 𝗗𝗮𝘁𝗮 𝗪𝗮𝗿𝗲𝗵𝗼𝘂𝘀𝗶𝗻𝗴 𝗮𝗻𝗱 𝗕𝗜: Hudi simplifies data ingestion into data lakes and enables low-latency analytics. 𝗙𝗶𝗻𝗮𝗹 𝗧𝗵𝗼𝘂𝗴𝗵𝘁𝘀 Apache Hudi is redefining how organizations process and manage data within modern data lakes. By enabling efficient incremental processing, upserts, and real-time analytics, Hudi empowers businesses to achieve better data freshness and cost efficiency. 🔗 Read the full article on Apache Hudi’s official blog: https://lnkd.in/g2FCJQhn #ApacheHudi #BigData #DataEngineering #DataLakes #RealTimeAnalytics #CloudComputing #ETL #MachineLearning #DataPipeline #OpenSource #DataScience

Want to stay in the loop on the latest Hudi developments? Join the "Apache Hudi Developer Sync Call" to stay updated on the latest developments in the community. This monthly call is the place to hear about upcoming releases, RFC discussions, and best practices while collaborating with fellow contributors. Whether you're looking to contribute to the project - through code, documentation, or blogs or simply want to follow key updates, this is the perfect opportunity to engage with the Hudi community. For details, visit the Hudi Developer Sync page here 👉 https://lnkd.in/dKhVNDSi 🗓 Next Call: 19th Feb 2025 🔗 Join via Zoom: https://lnkd.in/d6Jwtwsp #dataengineering #softwareengineering

What is Incremental Processing in a Lakehouse? How orgs like Uber benefit from it? Incremental processing is a technique that processes data in small increments rather than in one large batch. This approach is particularly useful in systems where data is continuously updated, allowing for more frequent and manageable updates. Here are 2 examples of how 'incremental processing' with Apache Hudi has helped Uber see large performance gains 🎯 decrease the pipeline run time by 50% 🎯 decrease SLA by 60%. 1. Uber Trips - Driver and Courier Earnings: - Late-arriving updates, such as "driver tips", require reprocessing large data partitions, causing inefficiency & delayed earnings updates. - This process demands substantial computational resources & time, often resulting in outdated or inaccurate earnings information for drivers. ✅ Incremental updates enable real-time data accuracy by processing 'only' the changes. ✅ This approach minimizes the need for reprocessing entire partitions, ensuring timely and precise earnings updates without excessive resource consumption. 2. Uber Eats - Frequent Menu Updates for Uber Eats Merchants - Uber Eats merchants frequently update menus that needs merging the entire day's changes. - This leads to long processing times, increased computational overhead, and higher risks of processing failures, ultimately delaying the availability of updated menu information to customers. ✅ Incremental processing reduces the time & computational effort required to incorporate frequent menu changes. ✅ This method allows for quicker updates, shorter SLAs, and lower failure rates, improving the overall user experience for both merchants & customers. The primitives in Hudi with the concepts around - timeline along with features like indexing, log merges, incremental queries makes it critical for this style of processing data. Read more in the blog from Uber Eng. #dataengineering #softwareengineering

Indexes are core to the design of Hudi. It powers faster writes & queries! Check out this blog by Sanjeet Shukla that goes over how Apache Hudi utilizes various types of indexes to enhance the efficiency of data updates and reads. Highlights: - Indexes in Hudi optimize data operations by efficiently locating data, reducing the amount processed during updates and reads. - Different indexes like Bloom Filter, HBase, and simple indexes are tailored for varying data volumes, update frequencies, and access patterns. - clarifies the roles of indexes in Copy-On-Write (COW) and Merge-On-Read (MOR) tables. Link 👉 https://lnkd.in/dsdwXQB6 #lakehouse #dataengineering #softwareengineering

Change Data Capture (CDC) is a technique used to identify & capture data changes, ensuring that the data remains fresh and consistent across various system. Combining CDC with data lakehouses can significantly simplify data management by addressing several challenges commonly faced by ETL pipelines delivering data from transactional databases to analytical databases. These include maintaining data freshness, ensuring consistency, and improving efficiency in data handling. In this episode of Lakehouse Chronicles, Sagar will go over the integration between data lakes and CDC, their benefits, implementation methods, key technologies and tools involved, best practices, and how to choose the right tools for your needs. Join us!

Bloom Filter in Parquet & Lakehouse Table Formats. Parquet filter pushdown is a performance optimization method that prunes irrelevant data from a #parquet file to reduce the amount of data scanned by a query engine. You can skip row groups typically using 2 ways: - column min/max statistics - dictionary filter ✅ Column statistics include minimum and maximum values that allow for range-based filtering. ✅ Dictionaries provide more specific filtering, enabling readers to exclude values that fall within the min and max range but are not listed in the dictionary. Problem with dictionary is that it can consume more space with higher cardinality columns. This results in columns with large cardinalities and widely separated min and max values lacking effective support for predicate pushdown. This is where the 3rd approach comes. ✅ A bloom filter is a probabilistic data structure that allows you to identify whether an item belong to a data set or not. ✅ It either outputs: "definitely not present" or "maybe present" for every data search. By using Bloom filters, you can efficiently skip over large portions of the Parquet file that are irrelevant to your query, reducing the amount of data that needs to be read and processed. In lakehouse platforms like Apache Hudi users can utilize the native Parquet bloom filters, provided their compute engine supports Apache Parquet 1.12.0 or higher. I also read an amazing blog by the Influx Data Team on Parquet's bloom filter implementation. Link in comments. #dataengineering #softwareengineering

Data Skipping + Index = Faster Queries 🚀 Data skipping is one of the common techniques used with large volumes of data to achieve better query performances. The whole idea is simple - read as less data as possible! What that means is that as a compute engine paired with a lakehouse platform like Apache Hudi it should read only the data files that are needed to satisfy the query from the storage. Data skipping not only reduces the volume of data that needs to be scanned & processed, but it can also lead to substantial improvements in execution time. Of course this is made possible because of the metadata information provided by file formats like #Parquet. Basically, each Parquet file contains the min/max values of each column along with other useful info such as: num of NULL values. These min/max values are 'column statistics'. Now, although, we could directly leverage these stats for data skipping, this could affect the query performance because the engine still has to go through each file to read the footer. ❌ This process can be very time taking specifically with large data volumes. What's Hudi's approach? ✅ Hudi adds a next level of pruning in this case. It basically takes all these column statistics & collates the info in the form of an INDEX. ✅ Indexes such as this (& more) are incorporated into Hudi's internal metadata table so engines can just get the files where the data is stored. Therefore, instead of reading individual Parquet footers, compute engines can directly go to the metadata table's index & fetch the required files. Way faster. For e.g. in this image, there are 2 parquet files that has well-defined value ranges for each of its columns. - File1.parquet contains 'Salary' ranges in $10000-40000 - File2.parquet contains 'Salary' ranges in $45000-90000 Now, if we run a query to fetch all the records where Salary=30000, the engine (Spark) can fetch the records from only File1 as it has values within the range & skip the other one. Imagine doing this at scale (with large number of files)! #dataengineering #softwareengineering