Serverless for Apache Spark Source
2 minute read
About
The Serverless for Apache Spark source allows Toolbox to interact with Spark batches hosted on Google Cloud Serverless for Apache Spark.
Available Tools
Serverless for Apache Spark Source Tools
| Tool Name | Description |
|---|---|
| serverless-spark-get-batch Tool | A "serverless-spark-get-batch" tool gets a single Spark batch from the source. |
| serverless-spark-list-batches Tool | A "serverless-spark-list-batches" tool returns a list of Spark batches from the source. |
| serverless-spark-cancel-batch Tool | A "serverless-spark-cancel-batch" tool cancels a running Spark batch operation. |
| serverless-spark-create-pyspark-batch Tool | A "serverless-spark-create-pyspark-batch" tool submits a Spark batch to run asynchronously. |
| serverless-spark-create-spark-batch Tool | A "serverless-spark-create-spark-batch" tool submits a Spark batch to run asynchronously. |
Requirements
IAM Permissions
Serverless for Apache Spark uses Identity and Access Management (IAM) to control user and group access to serverless Spark resources like batches and sessions.
Toolbox will use your Application Default Credentials
(ADC) to authorize and
authenticate when interacting with Google Cloud Serverless for Apache Spark.
When using this method, you need to ensure the IAM identity associated with your
ADC has the correct
permissions
for the actions you intend to perform. Common roles include
roles/dataproc.serverlessEditor (which includes permissions to run batches) or
roles/dataproc.serverlessViewer. Follow this
guide to
set up your ADC.
Example
kind: sources
name: my-serverless-spark-source
type: serverless-spark
project: my-project-id
location: us-central1
Reference
| field | type | required | description |
|---|---|---|---|
| type | string | true | Must be “serverless-spark”. |
| project | string | true | ID of the GCP project with Serverless for Apache Spark resources. |
| location | string | true | Location containing Serverless for Apache Spark resources. |
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.