Introducing Databricks Lake-Flow: A No-Code, Next-Generation Intelligent Solution for Data Engineering

The Databricks team has announced LakeFlow, a unified data platform poised to propel Databricks ahead of its rivals. This innovative platform significantly reduces the complexity of data pipeline development and lowers the learning curve. Much like Azure Data Factory and Informatica, the industry is expected to swiftly adopt Databricks LakeFlow for its intuitive drag-and-drop interface and built-in connectors for various data sources.

Databricks LakeFlow is a new solution that has everything you need to build and run data pipelines. It includes new connectors that work with popular databases like MySQL, Postgres, SQL Server, and Oracle, as well as enterprise applications like Salesforce, Microsoft Dynamics, NetSuite, Workday, ServiceNow, ERP and Google Analytics. These connectors are built-in and highly scalable, making it easier to connect and manage your data.

With Databricks LakeFlow, you can now organize and track workflows, and seamlessly deploy them into production using CI/CD. It's integrated into the Data Intelligence Platform, offering serverless computing and unified governance through the Unity Catalog.

LakeFlow serves as a comprehensive solution for data engineering, handling ingestion, transformation, and orchestration in a unified manner.

Difficulties in Creating and Managing Dependable Data Pipelines

Data engineering—collecting and preparing fresh, high-quality, and reliable data—is crucial for making data and AI accessible across your business. However, it's complex and involves integrating many different tools.

Firstly, data teams must gather data from various systems, each with its own formats and access methods. This involves creating and maintaining connectors for databases and business applications, and keeping up with changes in their APIs is a significant task. Next, data needs to be prepared in both batch and streaming formats, which requires writing and managing complex logic for processing data incrementally and triggering actions. When delays or failures occur, it can lead to alerts, unhappy users, and disruptions that impact business operations and revenue. Finally, teams must deploy these pipelines using CI/CD and ensure data quality and lineage, often requiring additional tools like Prometheus or Grafana.

To simplify this process, we developed LakeFlow—a unified solution for data ingestion, transformation, and orchestration powered by data intelligence. Its key components include LakeFlow Connect for data ingestion, LakeFlow Pipelines for data transformation, and LakeFlow Jobs for orchestration.

LakeFlow Connect: Simple and scalable data ingestion 

LakeFlow Connect simplifies data import with a click from databases like MySQL, Postgres, SQL Server, and Oracle, and business apps like Salesforce, Microsoft Dynamics, NetSuite, Workday, ServiceNow, and Google Analytics. It can also handle unstructured data like PDFs and Excel files from sources like SharePoint.

Additionally, it expands our existing connectors for cloud storage (like S3, ADLS Gen2, and GCS) and messaging systems (such as Kafka, Kinesis, Event Hub, and Pub/Sub), and integrates with partner solutions such as Fivetran, Qlik, and Informatica.

Customers using LakeFlow Connect at Databricks discover that its straightforward data ingestion boosts productivity, enabling quicker transitions from data to valuable insights. For instance, Insulet, a company manufacturing the Omnipod wearable insulin management system, leverages the Salesforce ingestion connector in LakeFlow Connect. This allows them to effortlessly bring customer feedback data into their Databricks-based data solution.

LakeFlow Pipelines: Efficient declarative data pipelines

LakeFlow Pipelines simplify the creation and management of effective batch and streaming data pipelines. Using the Delta Live Tables framework, they reduce complexity by allowing you to focus on writing SQL and Python for business logic. Databricks takes care of automating data orchestration, incremental processing, and scaling compute resources as needed.

Additionally, LakeFlow Pipelines include built-in features for monitoring data quality. Its Real Time Mode ensures you can consistently deliver time-sensitive datasets with low latency, all without requiring changes to your code.

LakeFlow Jobs: Reliable orchestration for every workload

LakeFlow Jobs efficiently handles and monitors production tasks with reliability. Leveraging Databricks Workflows, it orchestrates a wide range of workloads, from data ingestion and pipelines to notebooks, SQL queries, machine learning training, model deployment, and inference. Data teams benefit from features like triggers, branching, and looping to manage complex data delivery needs effectively.

Moreover, LakeFlow Jobs automates and simplifies the process of monitoring data health and delivery. It adopts a data-centric approach to health monitoring, providing comprehensive lineage that traces relationships across data ingestion, transformations, tables, and dashboards. Additionally, it tracks data freshness and quality, enabling data teams to set up monitors effortlessly through Lakehouse Monitoring.

Efficiently bringing all your data into the Data Intelligence Platform is just the beginning of unlocking its value and driving innovation. You can orchestrate advanced workflows for analytics and AI, and perform incremental transformations downstream. Use Mosaic AI to build and deploy ML and Gen-AI applications, or analyze and visualize your data to uncover actionable insights using Databricks SQL.

#databricks #analytics #dataengineering #bigdata #cloud #AI #GenAI #KeepLearning

Note - This blog post has been written based on new releases and demo presented by Databricks team in June event.