Why Delta Lake Is The Most Widely Used Lakehouse Format In The World?

Introduction to lakehouse architecture

The lakehouse architecture combines the best elements of data lakes and data warehouses — the data management and performance typically found in data warehouses with the low-cost, flexible object stores offered by data lakes. This unified platform simplifies your data architecture by eliminating the data silos that traditionally separate analytics, real-time analytics, data science, and machine learning. Merging data lakes and data warehouses into a single system means that data teams can move faster as they are able to use data without needing to access multiple systems.

Lakehouse architecture is increasingly being embraced as the data architecture for the future. According to Bill Inmon, widely considered the father of the data warehouse, the lakehouse presents an opportunity similar to the early years of the data warehouse market. The lakehouse’s unique ability to combine the data science focus of the data lake with the analytics power of the data warehouse — in an open environment — will unlock incredible value for organizations.

Lakehouses Need a Storage Format

Lakehouses are enabled by a new system design: implementing similar data structures and data management features to those in a data warehouse directly on top of low cost cloud storage in open formats. They are what you would get if you had to redesign data warehouses in the modern world, now that cheap and highly reliable storage (in the form of object stores) are available. As a result, lakehouses need a storage format and have the following key features:

• Transactional Guarantees Support: In an enterprise lakehouse, many data pipelines will often be reading and writing data concurrently. Support for ACID transactions ensures consistency as multiple parties concurrently read or write data, typically using SQL. Enterprise lakehouses need a storage format that acts as a single source of truth where transaction log and checkpoint files are stored in table’s log folder, not external catalogs. Storing all the metadata for a table in these ordered files enables ACID transactions and simplifies portability and scalability.
• Schema Evolution/Enforcement: The Lakehouse should have a way to prevent bad data from causing data corruption. This is enabled by support for schema enforcement and evolution, supporting DW schema architectures such as star/snowflake-schemas. The system should be able to reason about data integrity, and it should have robust governance and auditing mechanisms.
• Openness: The storage formats lakehouses use are open and standardized, such as Parquet, and they provide an API so a variety of tools and engines, including machine learning and Python/R libraries, can efficiently access the data directly.

Read the full research paper on the inner workings of the Lakehouse. This paper argues that the data warehouse architecture as we know it today will wither in the coming years and be replaced by a new architectural pattern, the Lakehouse, which will (i) be based on open direct-access data formats, such as Apache Parquet, (ii) have first-class support for machine learning and data science, and (iii) offer state-of-the-art performance.

Considerations when Choosing a Storage Format for Lakehouse Architecture

A fundamental requirement of your data lakehouse is the need to bring reliability to your data. Customers have shared that they typically evaluate storage formats based on the ability to share data “openly” across platforms, performance, production readiness, and openness. Customers value the maturity of the storage format to shave off development time.

What makes Delta Lake the format of choice

Today, Delta Lake is the most comprehensive Lakehouse format used by over 7,000 organizations, processing exabytes of data per day. Delta Lake is open, performant, production-ready, and enables open data sharing. All data in Delta Lake is stored in open Apache Parquet format, allowing data to be read by any compatible reader. APIs are open and compatible with Apache Spark. Delta Lake comes with standalone readers/writers that lets any Python, Ruby, or Rust client write data directly to Delta Lake without requiring any big data engine such as Apache Spark™. Delta Lake enables organizations to build Data Lakehouses, which enable data warehousing and machine learning directly on the data lake.

Below are a few key reasons why customers choose Delta Lake:

Most Widely Used Storage Layer

Today, Delta Lake is the most widely used storage layer in the world, with over 9 million monthly downloads; growing by 10x in monthly downloads in just one year.

“Graph showing immense growth in monthly downloads over the past year”

Enables Open Secure Data Sharing

With Delta Sharing, it is easy for anyone to easily share data and read data shared from other Delta tables in a secure way. Databricks released Delta Sharing in 2021 to give the data community an option to break free of vendor lock-in. As data sharing became more popular, many customers expressed frustrations of even more data silos (now even outside the organization) due to proprietary data format and proprietary compute required to read it.

Delta Sharing is the industry’s first open protocol for secure data sharing, making it simple to share data with other organizations regardless of where the data lives or what platform the data provider and recipient are using. Users can directly connect to the shared data through pandas, Tableau, or dozens of other systems that implement the open protocol, significantly accelerating time-to-value with no vendor lock-in. The recipients have direct, secure access to the shared data on cloud object stores like Amazon S3, Azure Data Lake Storage and Google Cloud Storage; there is no Databricks compute charge on the recipient side.

Metadata Colocated with Data

Delta Lake metadata is entirely colocated with data. Delta was architected solely on cloud storage it makes it much more portable and flexible than other formats

Enables Open Ecosystem

Delta Lake boasts the richest ecosystem of direct open-source connectors, including Apache Flink, Presto, and Trino, giving you the ability to read and write to Delta Lake directly from the most popular engines without Apache Spark. Thanks to the Delta Lake contributors from Scribd and Back Market, you can also use Delta Rust — a foundational Delta Lake library in Rust that enables Python, Rust, and Ruby developers to read and write Delta without any big data framework.

Growth of Community

Today, the Delta Lake project is thriving with over 190 contributors across more than 70 organizations, nearly two-thirds of whom are from outside Databricks contributors from leading companies like Apple, IBM, Microsoft, Disney, Amazon, and eBay, just to name a few. In fact, we’ve seen a 633% increase in contributor strength (as defined by the Linux Foundation) over the past three years. It’s this level of support that is the heart and strength of this open source project. One of the most exciting news from Data+AI Summit 2022 is Databricks is open sourcing all of Delta Lake.

“Graph showing consistent growth of contributor numbers to the project”

Source: Linux Foundation Contributor Strength: The growth in the aggregated count of unique contributors analyzed during the last three years. A contributor is anyone who is associated with the project by means of any code activity (commits/PRs/changesets) or helping to find and resolve bugs.

Databricks’ Solid Track Record of Contributions to Open Source Community and to Open Standards

From the beginning, Databricks has been committed to open standards and the open source community. Databricks has created and contributed to some of the most impactful innovations in modern open source data technology. Open data lakehouses are quickly becoming the standard for how the most innovative companies handle their data and AI. Delta Lake, MLflow and Spark are all core to this architectural transformation. Databricks engineers are the original creators of some of the world’s most popular open source data technologies — projects include Apache Spark, Delta lake, MLflow, Redash, and Delta Sharing.

Apache Spark is a unified engine for executing data engineering, data science and ML workloads. Delta Lake lets you build a lakehouse architecture on top of storage systems such as AWS S3, ADLS, GCS and HDFS. MLflow manages the ML lifecycle, including experimentation, reproducibility, deployment and a central model registry. Redash enables anyone to leverage SQL to explore, query, visualize, and share data from both big and small data sources. Delta Sharingis the industry’s first open protocol for secure data sharing, making it simple to share data with other organizations. Databricks supports additional popular open source technologies like TensorFlow, PyTorch, Keras, RStudio, scikit-learn, XGBoost, and Terraform. At the Data + AI Summit 2022, the largest gathering of the open source data and AI community, Databricks announced that the company will contribute all features and enhancements it has made to Delta Lake to the Linux Foundation and open source all Delta Lake APIs as part of the Delta Lake 2.0 release.

Organizations Who Chose Delta Lake

Today, Databricks has over 5,000 customers. The majority of them have built lakehouses using Databricks. More than 40% of the Fortune 500 use Databricks. Databricks mission is to help data teams solve the world’s toughest problems.

• Customer Blog by Claimsforce: Lakehouse — The journey unifying Data Lake and Data Warehouse
• Customer Led Talks at Data+AI Summit 2022: Accidentally Building a Petabyte-Scale Cybersecurity Data Mesh in Azure With Delta Lake at HSBC
• Customer Session: Why did we migrate to a Data Lakehouse on Delta Lake for T-Mobile
• YouTube: Massive Data Processing in Adobe Using Delta Lake
• YouTube: A High Performance Mutable Engagement Activity Delta Lake
• Customer Blog by Banco Hipotecario: Personalizing the banking experience with data and ML
• Customer Blog by Storebrand: Delivering analytics and ML innovation at speed and scale

Organizations that have contributed to Delta Lake

Together we have made Delta Lake the most widely used lakehouse format in the world. (Source)

Capabilities to Convert Parquet or Iceberg Tables to Delta Lake

Databricks provides a single command to convert Parquet or Iceberg tables to Delta Lake and unlock the full functionality of the lakehouse. The CONVERT TO DELTA SQL command performs a one-time conversion for Parquet and Iceberg tables to Delta Lake tables. You can use Databricks clone functionality to incrementally convert data from Parquet or Iceberg data sources to managed or external Delta tables. Unity Catalog supports the CONVERT TO DELTA SQL command for Parquet and Iceberg tables stored in external locations managed by Unity Catalog.

Summary

A fundamental requirement of your data lakehouse is the need to bring reliability to your data — one that is open, simple, production-ready, and platform agnostic, like Delta Lake. Delta Lake is open, performant, production-ready, and enables open data sharing. Delta Lake enables organizations to build Data Lakehouses, which enable data warehousing and machine learning directly on the data lake. Today, Delta Lake is the most comprehensive Lakehouse format used by over 7,000 organizations, processing exabytes of data per day.

Additional Blogs and Keynotes

• Blog: Announcing Delta 2.0: Bringing Everything to Open Source — The Databricks Blog
• Data+AI Summit 2022 Day 1 keynote: Day 1 Morning Keynote | Data + AI Summit 2022 — YouTube
• Delta 2.0 technical blog: Delta 2.0 — The Foundation of your Data Lakehouse is Open | Delta Lake