Native BigQuery support for Apache Spark alongside SQL. Learn more.

Spark on Google Cloud

Industry’s first autoscaling serverless Spark, integrated with the best of Google-native and open source tools. Develop and run Spark where you need it across all use cases, including ETL, data science, and exploration.

Try serverless Spark

Benefits

Increase developer productivity and get faster data insights

Operational simplicity through serverless Spark

Write Spark applications and pipelines that autoscale without any manual infrastructure provisioning or tuning.

Seamless Spark for all data users

Spark is integrated with BigQuery, Vertex AI, and Dataplex, so you can write and run it from these interfaces in two clicks, without custom integrations, for ETL, data exploration, analysis, and ML.

Flexibility of consumption

One size does not fit all. You can choose between serverless, Kubernetes clusters, and compute clusters for your Spark applications.

Key features

Run Spark jobs that autoscale, from the interface of your choice, in two clicks

BigQuery external procedures for Apache Spark

Unified SQL and Spark experience: Create and run Apache Spark code that is written in Python directly from BigQuery. You can then run and schedule these stored procedures in BigQuery using a Google Standard SQL query, similar to running SQL stored procedures.

Serverless Spark

Developers can spend all their time on code and logic, and use their chosen interface to submit Spark jobs which auto-provision and auto-scale. Read the documentation for serverless Spark.

Spark through Vertex AI

Spark for data science in one click: Data scientists can use Spark for development from Vertex AI Workbench seamlessly, with built-in security. Spark is integrated with Vertex AI's MLOps features, where users can execute Spark code through notebook executors that are integrated with Vertex AI Pipelines.

Spark through Dataplex

Run auto-scaling Spark on data across Google Cloud from a single interface that has one-click access to SparkSQL, Notebooks, or PySpark. Also offers easy collaboration with the ability to save, share, search notebooks and scripts alongside data, and built-in governance across data lakes.

Flexible consumption options

In addition to serverless Spark for no-ops deployment, customers standardizing on Kubernetes for infrastructure management can run Spark on Google Kubernetes Engine to improve resource utilization and simplify infrastructure management. Customers looking for Hadoop-style infrastructure management can run Spark on Compute Engine.

Ready to get started? Contact us

Partners