Serverless for Apache Spark

Google Cloud Serverless for Apache Spark lets you run Spark workloads without requiring you to provision and manage your own Spark cluster.

About

The Serverless for Apache Spark source allows Toolbox to interact with Spark batches hosted on Google Cloud Serverless for Apache Spark.

Available Tools

serverless-spark-list-batches List and filter Serverless Spark batches.
serverless-spark-get-batch Get a Serverless Spark batch.
serverless-spark-cancel-batch Cancel a running Serverless Spark batch operation.

Requirements

IAM Permissions

Serverless for Apache Spark uses Identity and Access Management (IAM) to control user and group access to serverless Spark resources like batches and sessions.

Toolbox will use your Application Default Credentials (ADC) to authorize and authenticate when interacting with Google Cloud Serverless for Apache Spark. When using this method, you need to ensure the IAM identity associated with your ADC has the correct permissions for the actions you intend to perform. Common roles include roles/dataproc.serverlessEditor (which includes permissions to run batches) or roles/dataproc.serverlessViewer. Follow this guide to set up your ADC.

Example

sources:
  my-serverless-spark-source:
    kind: serverless-spark
    project: my-project-id
    location: us-central1

Reference

field	type	required	description
kind	string	true	Must be “serverless-spark”.
project	string	true	ID of the GCP project with Serverless for Apache Spark resources.
location	string	true	Location containing Serverless for Apache Spark resources.

Last modified November 7, 2025: chore(main): release 0.19.1 (#1901) (cd8d68d)