How do I run a serverless batch job in Google Cloud

Question

I have a batch job that takes a couple of hours to run. How can I run this in a serverless way on Google Cloud?

AppEngine, Cloud Functions, and Cloud Run are limited to 10-15 minutes. I don't want to rewrite my code in Apache Beam.

Is there an equivalent to AWS Batch on Google Cloud?

Answer 1

Note: Cloud Run and Cloud Functions can now last up to 60 minutes. The answer below remains a viable approach if you have a multi-hour job.

Vertex AI Training is serverless and long-lived. Wrap your batch processing code in a Docker container, push to gcr.io and then do:

gcloud ai custom-jobs create \
  --region=LOCATION \
  --display-name=JOB_NAME \
  --worker-pool-spec=machine-type=MACHINE_TYPE,replica-count=REPLICA_COUNT,executor-image-uri=EXECUTOR_IMAGE_URI,local-package-path=WORKING_DIRECTORY,script=SCRIPT_PATH

You can run any arbitrary Docker container — it doesn’t have to be a machine learning job. For details, see:

https://cloud.google.com/vertex-ai/docs/training/create-custom-job#create_custom_job-gcloud

Answer 2

Google Cloud does not offer a comparable product to AWS Batch (see https://cloud.google.com/docs/compare/aws/#service_comparisons).

Instead you'll need to use Cloud Tasks or Pub/Sub to delegate the work to another product, such as Compute Engine, but this lacks the ability to do this in a "serverless" way.

Answer 3

This answer to a How to make GCE instance stop when its deployed container finishes? will work for you as well:

In short:

• First dockerize your batch process.
• Then, create an instance:
  • Using a container-optmized image
  • And using a Startup script that pulls your docker image, runs it, and shutdown the machine at the end.