Introducing Batch, a new managed service for scheduling batch jobs at any scale

Today, we’re excited to announce the Preview release of our fully managed service, Batch, which provides easy access to Google Cloud’s computing power and scale. 

Batch processing is as old as computing itself, with the term 'batch' dating back to the punchcards used by early mainframes. Batch uses resources very efficiently and remains the preferred way of running jobs that don’t need much human interaction. Enterprise workloads very often include some batch processing elements. Batch jobs are especially prevalent in areas such as research, simulation, genomics, visual effects, fintech, manufacturing and EDA.

As enterprise customers turn to cloud to meet the need for more compute resources or to easily access the latest processors or GPUs, they bring their batch workloads with them. Batch is a general-purpose batch job service and the latest in a long list of products we’ve created over the years that process jobs to help enterprises migrate their workloads to the cloud. These services include Cloud Life Sciences (formerly Google Genomics), Dataflow, and Cloud Run Jobs.

What is Batch?

The Batch service handles several essential tasks. It manages the job queue, provisions and autoscales resources, runs jobs, executes subtasks, and deals with common errors — all automatically. You can access the service through the API, the gcloud command line tool, workflow engines, or via the easy to use UI in Cloud Console. In short, Batch allows developers, admins, scientists, researchers, and anyone else interested in batch computing to focus on their applications and results, handling everything in between. 

Here are just a few examples of what Batch can do:

• Run batch jobs as a service. Batch supports throughput-oriented, HPC, AI/ML, and data processing jobs.

• Provision compute resources. Batch supports all CPU machine families including the newly released T2A Arm instances

• Use accelerator-optimized resources. In collaboration with NVIDIA, Batch supports the use of NVIDIA GPUs when running demanding batch workloads such as ML training, HPC, and graphics simulation.

• Support common job types including arrays of jobs and multi-node MPI jobs utilizing task parallelization. 

• Handle any executable. Bring your scripts or containerized workload.

• Provide flexible provisioning models, including support for Spot VMs, which offer up to 91% savings versus regular compute instances, and custom machine types.

• Simplify native integrations with Google Cloud services as well as popular workflow engines and tools such as Nextflow. The dsub command line tool will be supported imminently.
  

What customers and partners are saying

"At Locomation, we are leading the way in autonomous trucking technology by making driving safer and more efficient. Google Cloud Batch made it easy for our engineers to quickly secure the compute resources needed to analyze large amounts of data. With Batch, we are able to reduce the time to unlock insights from large datasets by 80%, enabling us to meet our business objectives faster." - David Hendi, Lead, Tools and Infrastructure at Locomation

"Batch provides an elegant but powerful API with a straightforward execution model. The integration makes data handling a breeze. These and other technical advantages baked into Google Batch will directly benefit pipeline efficiency, throughput, and reliability. A benefit of working with Google through the pre-release is that we’ve had ample time to focus on testing. Validating the integration at scale with dozens of production pipelines has helped us quickly get out of the gate with a high-quality integration."—Paolo Di Tommaso, Nextflow lead engineer and Seqera Labs co-founder

"At Roivant Discovery, we leverage massive computational simulations to drive the discovery of novel drugs across small molecule modalities. Our scientists are required to operate at the cutting edge of quantum physics with speed and agility, at scale. Google Batch's unprecedented simplicity and flexibility will ensure we're always focused on innovation."—Matt Maisak, COO at Roivant Discovery

Using Batch

Because it’s offered as a service, you don’t need to install or configure Batch in order to use it. But there are a few fundamental concepts you should be aware of. 

• Job - One execution of a piece of run-to-completion computation work.Jobs can be as simple as a single shell script or more complex multipart computations that can be run either in parallel or sequentially. 

• Tasks - The computations that make up a job. Tasks run on Compute Engine instances.

• Array Job - Multiple tasks in a job that simultaneously execute the same executable (script or container)

• Resources - What you need to run the job. Common resources include Compute Engine instances (VM instances that are created by Batch within your project to execute the tasks) and storage, such as Cloud Storage, NFS mounts, or additional disks that are used to read and write input and output.

With these concepts under our belts, let’s take a look at how to submit a simple job from the Google Cloud console (for more complex jobs, you should use the Batch API or the command line). This job will output the text ‘helloworld’ and the task index within the job before sleeping for 60 seconds. The script will execute 100 times to account for the number of tasks and will run at most 20 tasks in parallel.

The hello-world job created above was populated in the Job List page and the status progressed from Queued to Scheduled to Succeeded.

You can drill down into the job details to display additional information as shown below.

Getting started

If you’re looking for an easier way to handle your batch processing jobs, Batch provides the simplicity to go from an executable to results within a few steps. Batch is currently available in four Google Cloud regions: Iowa (us-central1), South Carolina (us-east1), Oregon (us-west1), and Finland (europe-north1). There is no additional charge for using the Batch service. You only pay for the resources used to run your batch job. Head to the Cloud Console to start deploying your batch jobs today.

To learn more about Batch, check out the product page and these resources below to help you onboard:

• Hands-on experience with Batch - Walk through the media transcoding tutorial, which leverages Batch to transcode H.264 video files to VP9.

• GitHub samples - Browse a curated list of Batch how-tos and samples 

• Learn about how to use Batch via the newly released Google HPC Toolkit

• If you have questions, reach out to gcp-batch-preview@google.com or explore the Batch category within the Cloud Forums to engage with fellow community members.

Available now in GA (updated as of Oct 11, 2022)