René's URL Explorer Experiment


Title: Workflow using Cloud Run functions  |  Managed Service for Apache Spark  |  Google Cloud Documentation

Open Graph Title: Workflow using Cloud Run functions  |  Managed Service for Apache Spark  |  Google Cloud Documentation

Opengraph URL: https://docs.cloud.google.com/managed-spark/docs/tutorials/workflow-function

direct link

Domain: docs.cloud.google.com

Hey, it has json ld scripts: 
  {
    "@context": "https://schema.org",
    "@type": "Article",
    
    "headline": "Workflow using Cloud Run functions"
  }

  {
    "@context": "https://schema.org",
    "@type": "BreadcrumbList",
    "itemListElement": [{
      "@type": "ListItem",
      "position": 1,
      "name": "Managed Service for Apache Spark",
      "item": "https://docs.cloud.google.com/managed-spark/docs"
    },{
      "@type": "ListItem",
      "position": 2,
      "name": "Workflow using Cloud Run functions",
      "item": "https://docs.cloud.google.com/managed-spark/docs/tutorials/workflow-function"
    }]
  }
google-signin-client-id721724668570-nbkv1cfusk7kk4eni4pjvepaus73b13t.apps.googleusercontent.com
google-signin-scopeprofile email https://www.googleapis.com/auth/developerprofiles https://www.googleapis.com/auth/developerprofiles.award https://www.googleapis.com/auth/devprofiles.full_control.firstparty
og:site_nameGoogle Cloud Documentation
og:typewebsite
theme-color#1a73e8
NoneIE=Edge
og:imagehttps://docs.cloud.google.com/_static/cloud/images/social-icon-google-cloud-1200-630.png
og:image:width1200
og:image:height630
og:localeen
twitter:cardsummary_large_image

Links:

Skip to main content https://docs.cloud.google.com/managed-spark/docs/tutorials/workflow-function#main-content
https://docs.cloud.google.com/
Documentation https://docs.cloud.google.com/docs
Get Started with Google Cloud https://docs.cloud.google.com/docs/get-started
Product List https://docs.cloud.google.com/docs/product-list
Cloud Customer Care https://docs.cloud.google.com/support/docs
Agent Platform https://docs.cloud.google.com/products/gemini-enterprise-agent-platform
Apigee API Management https://docs.cloud.google.com/apigee
BigQuery https://docs.cloud.google.com/bigquery
Compute Engine https://docs.cloud.google.com/products/compute
Cloud CDN https://docs.cloud.google.com/cdn
Cloud Run https://docs.cloud.google.com/run
Cloud Storage https://docs.cloud.google.com/storage
Cloud SQL https://docs.cloud.google.com/sql
Gemini Enterprise https://docs.cloud.google.com/gemini/enterprise/docs
Google Kubernetes Engine https://docs.cloud.google.com/kubernetes-engine
Looker https://docs.cloud.google.com/looker
Access and resources management https://docs.cloud.google.com/docs/access-resources
Costs and usage management https://docs.cloud.google.com/docs/costs-usage
Infrastructure as code https://docs.cloud.google.com/docs/iac
SDK, languages, frameworks, and tools https://docs.cloud.google.com/docs/devtools
AI and ML https://docs.cloud.google.com/docs/ai-ml
Application development https://docs.cloud.google.com/docs/application-development
Application hosting https://docs.cloud.google.com/docs/application-hosting
Compute https://docs.cloud.google.com/docs/compute-area
Data analytics and pipelines https://docs.cloud.google.com/docs/data
Databases https://docs.cloud.google.com/docs/databases
Distributed, hybrid, and multicloud https://docs.cloud.google.com/docs/dhm-cloud
Industry solutions https://docs.cloud.google.com/docs/industry
Migration https://docs.cloud.google.com/docs/migration
Networking https://docs.cloud.google.com/docs/networking
Observability and monitoring https://docs.cloud.google.com/docs/observability
Security https://docs.cloud.google.com/docs/security
Storage https://docs.cloud.google.com/docs/storage
Console https://console.cloud.google.com/
https://docs.cloud.google.com/managed-spark/docs
Managed Service for Apache Spark https://docs.cloud.google.com/managed-spark/docs
Start freehttps://console.cloud.google.com/freetrial
Overview https://docs.cloud.google.com/managed-spark/docs
Guides https://docs.cloud.google.com/managed-spark/docs/overview/key-concepts
Reference https://docs.cloud.google.com/managed-spark/docs/reference
Samples https://docs.cloud.google.com/managed-spark/docs/samples
Resources https://docs.cloud.google.com/managed-spark/docs/resources
https://docs.cloud.google.com/
Documentation https://docs.cloud.google.com/docs
Overview https://docs.cloud.google.com/managed-spark/docs
Guides https://docs.cloud.google.com/managed-spark/docs/overview/key-concepts
Reference https://docs.cloud.google.com/managed-spark/docs/reference
Samples https://docs.cloud.google.com/managed-spark/docs/samples
Resources https://docs.cloud.google.com/managed-spark/docs/resources
Console https://console.cloud.google.com/
Key Conceptshttps://docs.cloud.google.com/managed-spark/docs/overview/key-concepts
Overviewhttps://docs.cloud.google.com/managed-spark/docs/serverless-overview
Managed Service for Apache Spark serverless tiershttps://docs.cloud.google.com/managed-spark/docs/tiers
Managed Service for Apache Spark on clustershttps://docs.cloud.google.com/managed-spark/docs/concepts/clusters-overview
Compare serverless and cluster deploymentshttps://docs.cloud.google.com/managed-spark/docs/concepts/serverless-spark-compare
Managed Service for Apache Spark on GKEhttps://docs.cloud.google.com/managed-spark/docs/concepts/gke-overview
Create a serverless Spark batch workloadhttps://docs.cloud.google.com/managed-spark/docs/quickstarts/spark-batch
Create a clusterhttps://docs.cloud.google.com/managed-spark/docs/quickstarts/create-cluster
Submit a Spark job to a clusterhttps://docs.cloud.google.com/managed-spark/docs/guides/submit-job
Use Gemini to develop Spark applicationshttps://docs.cloud.google.com/managed-spark/docs/guides/gemini-spark
Analyze public datasets with Sparkhttps://docs.cloud.google.com/managed-spark/docs/tutorials/monte-carlo-methods-with-hadoop-spark
Sentiment analysis with Spark MLlibhttps://docs.cloud.google.com/managed-spark/docs/tutorials/bigquery-sparkml
Run a Spark job on Kuberneteshttps://docs.cloud.google.com/managed-spark/docs/guides/dpgke/quickstarts/gke-quickstart-create-cluster
Use custom containershttps://docs.cloud.google.com/managed-spark/docs/guides/custom-containers
Use GPUshttps://docs.cloud.google.com/managed-spark/docs/guides/gpus-serverless
Use Dynamic Workload Schedulerhttps://docs.cloud.google.com/managed-spark/docs/guides/dws-serverless
Network configurationhttps://docs.cloud.google.com/managed-spark/docs/concepts/network-serverless
Overviewhttps://docs.cloud.google.com/managed-spark/docs/concepts/versions/serverless-versions
Spark runtime version 3.0https://docs.cloud.google.com/managed-spark/docs/concepts/versions/spark-runtime-3.0
Spark runtime version 2.3https://docs.cloud.google.com/managed-spark/docs/concepts/versions/spark-runtime-2.3
Spark runtime version 2.2https://docs.cloud.google.com/managed-spark/docs/concepts/versions/spark-runtime-2.2
Spark runtime version 1.2https://docs.cloud.google.com/managed-spark/docs/concepts/versions/spark-runtime-1.2
Service accountshttps://docs.cloud.google.com/managed-spark/docs/concepts/service-accounts-serverless
Spark propertieshttps://docs.cloud.google.com/managed-spark/docs/concepts/spark-properties-serverless
Staging buckethttps://docs.cloud.google.com/managed-spark/docs/concepts/staging-bucket-serverless
Create a serverless Spark batch workloadhttps://docs.cloud.google.com/managed-spark/docs/guides/spark-batch
Create serverless interactive sessions and runtime profileshttps://docs.cloud.google.com/managed-spark/docs/guides/create-serverless-sessions-templates
Use the serverless Spark Connect clienthttps://docs.cloud.google.com/managed-spark/docs/guides/spark-connect-client
Use JupyterLab for serverless batch and notebook sessionshttps://docs.cloud.google.com/managed-spark/docs/guides/jupyterlab-sessions-serverless
Visualize Spark DataFrames in notebookshttps://docs.cloud.google.com/managed-spark/docs/guides/visualize-dataframes-notebooks
Overviewhttps://docs.cloud.google.com/managed-spark/docs/templates/templates
Cloud Spanner to Cloud Storagehttps://docs.cloud.google.com/managed-spark/docs/templates/spanner-to-storage
Cloud Storage to BigQueryhttps://docs.cloud.google.com/managed-spark/docs/templates/storage-to-bigquery
Cloud Storage to Cloud Spannerhttps://docs.cloud.google.com/managed-spark/docs/templates/storage-to-spanner
Cloud Storage to Cloud Storagehttps://docs.cloud.google.com/managed-spark/docs/templates/storage-to-storage
Cloud Storage to JDBChttps://docs.cloud.google.com/managed-spark/docs/templates/storage-to-jdbc
Hive to BigQueryhttps://docs.cloud.google.com/managed-spark/docs/templates/hive-to-bigquery
Hive to Cloud Storagehttps://docs.cloud.google.com/managed-spark/docs/templates/hive-to-storage
JDBC to BigQueryhttps://docs.cloud.google.com/managed-spark/docs/templates/jdbc-to-bigquery
JDBC to Cloud Spannerhttps://docs.cloud.google.com/managed-spark/docs/templates/jdbc-to-spanner
JDBC to Cloud Storagehttps://docs.cloud.google.com/managed-spark/docs/templates/jdbc-to-storage
JDBC to JDBChttps://docs.cloud.google.com/managed-spark/docs/templates/jdbc-to-jdbc
Pub/Sub to Cloud Storagehttps://docs.cloud.google.com/managed-spark/docs/templates/pubsub-to-storage
Create a lakehouse with Sparkhttps://docs.cloud.google.com/managed-spark/docs/guides/create-lakehouse
Access Workday data with Sparkhttps://docs.cloud.google.com/managed-spark/docs/guides/access-workday-data-spark
Query federated catalogs with Sparkhttps://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs
Create an Apache Iceberg table with metadata in Lakehouse runtime cataloghttps://docs.cloud.google.com/managed-spark/docs/guides/spark-workloads-with-bigquery-metastore
Transform data and write to Apache Iceberghttps://docs.cloud.google.com/managed-spark/docs/tutorials/spark-to-iceberg
Overviewhttps://docs.cloud.google.com/managed-spark/docs/guides/spark-bigquery-connector
Query BigQuery tableshttps://docs.cloud.google.com/managed-spark/docs/guides/query-bigquery-tables
Run PySpark code in BigQuery Studio notebookshttps://docs.cloud.google.com/managed-spark/docs/use-spark
Autoscale workload resourceshttps://docs.cloud.google.com/managed-spark/docs/concepts/autoscaling-serverless
Autotune Spark workloadshttps://docs.cloud.google.com/managed-spark/docs/concepts/autotuning
Accelerate batch workloads and sessions with Lightning Enginehttps://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine-serverless
Run the Native Query Execution qualification toolhttps://docs.cloud.google.com/managed-spark/docs/guides/run-nqe-tool
Serverless Spark solution acceleratorshttps://github.com/GoogleCloudPlatform/serverless-spark-workshop
Manage Spark dependencieshttps://docs.cloud.google.com/managed-spark/docs/guides/manage-spark-dependencies
Customize Spark environmenthttps://docs.cloud.google.com/managed-spark/docs/guides/docker-yarn
Enable concurrent writeshttps://docs.cloud.google.com/managed-spark/docs/guides/fileoutput-committer
Enhance Spark performancehttps://docs.cloud.google.com/managed-spark/docs/guides/performance-enhancements
Tune Sparkhttps://docs.cloud.google.com/managed-spark/docs/support/spark-job-tuning
Use Lightning Enginehttps://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine
Use the consolehttps://docs.cloud.google.com/managed-spark/docs/quickstarts/create-cluster-console
Use the command linehttps://docs.cloud.google.com/managed-spark/docs/quickstarts/create-cluster-gcloud
Create a clusterhttps://docs.cloud.google.com/managed-spark/docs/quickstarts/create-cluster-template
Run a Spark jobhttps://docs.cloud.google.com/managed-spark/docs/quickstarts/submit-spark-job-template
Update a clusterhttps://docs.cloud.google.com/managed-spark/docs/quickstarts/update-cluster-template
Delete a clusterhttps://docs.cloud.google.com/managed-spark/docs/quickstarts/quickstart-explorer-delete
Use client librarieshttps://docs.cloud.google.com/managed-spark/docs/quickstarts/create-cluster-client-libraries
Run Hadoop jobshttps://docs.cloud.google.com/managed-spark/docs/tutorials/bigquery-connector-mapreduce-example
Write and run Spark Scala jobshttps://docs.cloud.google.com/managed-spark/docs/tutorials/spark-scala
Run Trinohttps://docs.cloud.google.com/managed-spark/docs/tutorials/trino-dataproc
Run Flinkhttps://docs.cloud.google.com/managed-spark/docs/concepts/components/flink
Run Hivehttps://docs.cloud.google.com/managed-spark/docs/tutorials/kafka-tutorial
Run Pighttps://docs.cloud.google.com/managed-spark/docs/concepts/components/pig
Run HBasehttps://docs.cloud.google.com/managed-spark/docs/tutorials/spark-hbase
Overviewhttps://docs.cloud.google.com/managed-spark/docs/tutorials/python-configuration
Migrate and set up Conda channelshttps://docs.cloud.google.com/managed-spark/docs/guides/conda-channel-migration
Use Cloud Client Libraries for Pythonhttps://docs.cloud.google.com/managed-spark/docs/tutorials/python-library-example
Overviewhttps://docs.cloud.google.com/managed-spark/docs/tutorials/bigquery-connector-spark-example
BigQuery connector code sampleshttps://docs.cloud.google.com/managed-spark/docs/examples/bigquery-example
Use the Cloud Storage connectorhttps://docs.cloud.google.com/managed-spark/docs/tutorials/gcs-connector-spark-tutorial
Use the Spark Spanner connectorhttps://docs.cloud.google.com/managed-spark/docs/tutorials/spanner-connector-spark-example
Explore and extract datahttps://docs.cloud.google.com/managed-spark/docs/guides/explore-data
Transform datahttps://docs.cloud.google.com/managed-spark/docs/guides/transform-data
Load data into BigQueryhttps://docs.cloud.google.com/managed-spark/docs/guides/load-data
Create a lakehouse with Sparkhttps://docs.cloud.google.com/managed-spark/docs/guides/create-lakehouse
Query federated catalogs with Sparkhttps://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs
Configure metastoreshttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/secure-hive-metastore
Create an Apache Iceberg table with metadata in BigLake metastorehttps://docs.cloud.google.com/managed-spark/docs/guides/iceberg-metadata-biglake-metastore
Use Iceberghttps://docs.cloud.google.com/managed-spark/docs/concepts/components/iceberg
Use Deltahttps://docs.cloud.google.com/managed-spark/docs/concepts/components/delta
Use Hudihttps://docs.cloud.google.com/managed-spark/docs/concepts/components/hudi
Overviewhttps://docs.cloud.google.com/managed-spark/docs/concepts/notebooks
Run a Jupyter notebook on a clusterhttps://docs.cloud.google.com/managed-spark/docs/tutorials/jupyter-notebook
Run a genomics analysis on a notebookhttps://docs.cloud.google.com/managed-spark/docs/tutorials/genomics-single-cell
Use the JupyterLab extension to develop serverless Spark workloadshttps://docs.cloud.google.com/managed-spark/docs/guides/jupyterlab-sessions-serverless
Use the Component Gatewayhttps://docs.cloud.google.com/managed-spark/docs/concepts/accessing/gateways
Connect to data sourceshttps://docs.cloud.google.com/managed-spark/docs/guides/connect-to-data-sources
Connect to Cloud Storagehttps://docs.cloud.google.com/managed-spark/docs/concepts/connectors/cloud-storage
Connect to BigQueryhttps://docs.cloud.google.com/managed-spark/docs/concepts/connectors/bigquery
Connect Hive to BigQueryhttps://docs.cloud.google.com/managed-spark/docs/concepts/connectors/hive-bigquery
Connect to Bigtablehttps://docs.cloud.google.com/managed-spark/docs/concepts/connectors/cloud-bigtable
Connect to Pub/Sub Litehttps://docs.cloud.google.com/managed-spark/docs/concepts/connectors/pubsub-lite
Fleet managementhttps://docs.cloud.google.com/managed-spark/docs/guides/fleet-management
Enable Spark data lineagehttps://docs.cloud.google.com/managed-spark/docs/guides/spark-lineage
Enable Hive data lineagehttps://docs.cloud.google.com/managed-spark/docs/guides/hive-lineage
Overviewhttps://docs.cloud.google.com/managed-spark/docs/concepts/components/overview
Delta Lakehttps://docs.cloud.google.com/managed-spark/docs/concepts/components/delta
Dockerhttps://docs.cloud.google.com/managed-spark/docs/concepts/components/docker
Flinkhttps://docs.cloud.google.com/managed-spark/docs/concepts/components/flink
HBasehttps://docs.cloud.google.com/managed-spark/docs/concepts/components/hbase
Hive WebHCathttps://docs.cloud.google.com/managed-spark/docs/concepts/components/hivewebhcat
Hudihttps://docs.cloud.google.com/managed-spark/docs/concepts/components/hudi
Iceberghttps://docs.cloud.google.com/managed-spark/docs/concepts/components/iceberg
Jupyterhttps://docs.cloud.google.com/managed-spark/docs/concepts/components/jupyter
Pighttps://docs.cloud.google.com/managed-spark/docs/concepts/components/pig
Prestohttps://docs.cloud.google.com/managed-spark/docs/concepts/components/presto
Install Rangerhttps://docs.cloud.google.com/managed-spark/docs/concepts/components/ranger
Use Ranger with Kerberoshttps://docs.cloud.google.com/managed-spark/docs/concepts/components/ranger-w-kerberos
Use Ranger with caching and downscopinghttps://docs.cloud.google.com/managed-spark/docs/concepts/components/ranger-caching-downscoping
Back up and restore a Ranger schemahttps://docs.cloud.google.com/managed-spark/docs/concepts/components/backup-ranger-schema
Solrhttps://docs.cloud.google.com/managed-spark/docs/concepts/components/solr
Trinohttps://docs.cloud.google.com/managed-spark/docs/concepts/components/trino
Zeppelinhttps://docs.cloud.google.com/managed-spark/docs/concepts/components/zeppelin
Zookeeperhttps://docs.cloud.google.com/managed-spark/docs/concepts/components/zookeeper
Overviewhttps://docs.cloud.google.com/managed-spark/docs/concepts/versioning/overview
Serviceshttps://docs.cloud.google.com/managed-spark/docs/concepts/services
Overviewhttps://docs.cloud.google.com/managed-spark/docs/concepts/versioning/image-version-lists
3.0.x release image versionshttps://docs.cloud.google.com/managed-spark/docs/concepts/versioning/image-release-3.0
2.3.x release image versionshttps://docs.cloud.google.com/managed-spark/docs/concepts/versioning/image-release-2.3
2.2.x release image versionshttps://docs.cloud.google.com/managed-spark/docs/concepts/versioning/image-release-2.2
2.1.x release image versionshttps://docs.cloud.google.com/managed-spark/docs/concepts/versioning/image-release-2.1
Machine typeshttps://docs.cloud.google.com/managed-spark/docs/concepts/compute/supported-machine-types
GPUshttps://docs.cloud.google.com/managed-spark/docs/concepts/compute/gpus
Minimum CPU platformhttps://docs.cloud.google.com/managed-spark/docs/concepts/compute/min-cpu
Flexible VMshttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/flexible-vms
Secondary workershttps://docs.cloud.google.com/managed-spark/docs/concepts/compute/secondary-vms
Local solid state driveshttps://docs.cloud.google.com/managed-spark/docs/concepts/compute/local-ssds
Boot diskshttps://docs.cloud.google.com/managed-spark/docs/concepts/compute/persistent-disks
Attached hyperdiskshttps://docs.cloud.google.com/managed-spark/docs/concepts/compute/attached-disks
Overviewhttps://docs.cloud.google.com/managed-spark/docs/guides/create-cluster
Overviewhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network
Use secure tagshttps://docs.cloud.google.com/managed-spark/docs/guides/use-secure-tags
Configure Private Service Connecthttps://docs.cloud.google.com/managed-spark/docs/networking-private-service-connect
Select a cluster regionhttps://docs.cloud.google.com/managed-spark/docs/concepts/cluster-region
Enable auto zone placementhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/auto-zone
Create a multi-zone clusterhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/multi-zone
Schedule cluster deletionhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/scheduled-deletion
Use initialization actionshttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/init-actions
Set cluster metadatahttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/metadata
Set cluster propertieshttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/cluster-properties
Enable cluster web interfaceshttps://docs.cloud.google.com/managed-spark/docs/concepts/accessing/cluster-web-interfaces
Create a high availability clusterhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/high-availability
Create a high obtainability clusterhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/high-obtainability
Create a node group clusterhttps://docs.cloud.google.com/managed-spark/docs/guides/node-groups/driver-node-groups
Create a partial clusterhttps://docs.cloud.google.com/managed-spark/docs/guides/create-partial-cluster
Create a zero-scale clusterhttps://docs.cloud.google.com/managed-spark/docs/guides/create-zero-scale-cluster
Create a single-node clusterhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/single-node-clusters
Create sole-tenant clusterhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/sole-tenant-nodes
Create a custom imagehttps://docs.cloud.google.com/managed-spark/docs/guides/images
Set labels for filteringhttps://docs.cloud.google.com/managed-spark/docs/guides/creating-managing-labels
Scale clusters manuallyhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/scaling-clusters
Autoscale clustershttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/autoscaling
Start and stop a clusterhttps://docs.cloud.google.com/managed-spark/docs/guides/start-stop
Schedule cluster stophttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/scheduled-stop
Recreate a clusterhttps://docs.cloud.google.com/managed-spark/docs/guides/recreate-cluster
Rotate clustershttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/cluster-rotation
Update and delete a clusterhttps://docs.cloud.google.com/managed-spark/docs/guides/manage-cluster
Use SSH to connect to a clusterhttps://docs.cloud.google.com/managed-spark/docs/concepts/accessing/ssh
Overviewhttps://docs.cloud.google.com/managed-spark/docs/guides/best-practices
Maximize cluster and job flexibility and efficiencyhttps://docs.cloud.google.com/managed-spark/docs/guides/maximize-cluster-job-flexibility
Hadoop data storagehttps://docs.cloud.google.com/managed-spark/docs/concepts/hdfs
Select storage typehttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/staging-bucket
Cache cluster datahttps://docs.cloud.google.com/managed-spark/docs/concepts/cluster-caching
Offload shuffle datahttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/enhanced-flexibility-mode
View cluster logshttps://docs.cloud.google.com/managed-spark/docs/guides/logging#cluster-logs
Life of a jobhttps://docs.cloud.google.com/managed-spark/docs/concepts/jobs/life-of-a-job
Restart jobshttps://docs.cloud.google.com/managed-spark/docs/concepts/jobs/restartable-jobs
Overviewhttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/overview
Parameterizationhttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/workflow-parameters
Use YAML fileshttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/use-yamls
Use cluster selectorshttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/cluster-selectors
Use inline workflowshttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/inline-workflows
GitHub templateshttps://docs.cloud.google.com/managed-spark/docs/guides/templates
Workflow scheduling solutionshttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/workflow-schedule-solutions
Use workflow templateshttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/use-workflows
Use Cloud Composerhttps://docs.cloud.google.com/managed-spark/docs/tutorials/workflow-composer
Use Cloud Functionshttps://docs.cloud.google.com/managed-spark/docs/tutorials/workflow-function
Use Cloud Schedulerhttps://docs.cloud.google.com/managed-spark/docs/tutorials/workflow-scheduler
Create node poolshttps://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-nodepools
Recreate a GKE virtual clusterhttps://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-recreate-cluster
Create a GKE custom custom container imagehttps://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-custom-images
Scale GKE clustershttps://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-scale-cluster
Delete a GKE virtualclusterhttps://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-delete-cluster
Release versionshttps://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-versions
Visualize Spark DataFrames in notebookshttps://docs.cloud.google.com/managed-spark/docs/guides/visualize-dataframes-notebooks
Permissions and roleshttps://docs.cloud.google.com/managed-spark/docs/concepts/iam-serverless
Personas and serverless IAM roleshttps://docs.cloud.google.com/managed-spark/docs/concepts/personas-and-serverless-iam-roles
Use CMEK with Serverless for Apache Sparkhttps://docs.cloud.google.com/managed-spark/docs/guides/cmek-serverless
Default security measureshttps://docs.cloud.google.com/managed-spark/docs/concepts/security-serverless
Use Secret Managerhttps://docs.cloud.google.com/managed-spark/docs/guides/hadoop-google-secret-manager-credential-provider-serverless
Create custom constraintshttps://docs.cloud.google.com/managed-spark/docs/guides/custom-constraints-serverless
Security best practiceshttps://docs.cloud.google.com/managed-spark/docs/concepts/security-best-practices
Authenticate to Managed Service for Apache Sparkhttps://docs.cloud.google.com/managed-spark/docs/authentication
Authenticate personal clustershttps://docs.cloud.google.com/managed-spark/docs/concepts/iam/personal-auth
Workforce identity federationhttps://docs.cloud.google.com/managed-spark/docs/concepts/accessing/workforce-identity-federation
Permissions and roleshttps://docs.cloud.google.com/managed-spark/docs/concepts/iam/iam
Managed Service for Apache Spark principalshttps://docs.cloud.google.com/managed-spark/docs/concepts/iam/principals
Granular IAMhttps://docs.cloud.google.com/managed-spark/docs/concepts/iam/granular-iam
Assign roles for Kuberneteshttps://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-iam
Service accountshttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/service-accounts
Secure multi-tenancy using Kerberoshttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/security
Secure multi-tenancy using service accountshttps://docs.cloud.google.com/managed-spark/docs/concepts/iam/sa-multi-tenancy
Encrypt memoryhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/confidential-compute
Manage data encryption keyshttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/customer-managed-encryption
Enable Ranger authorization servicehttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/ranger-plugin
Use the Secret Manager credential providerhttps://docs.cloud.google.com/managed-spark/docs/guides/hadoop-google-secret-manager-credential-provider
Create and secure a Hive metastore clusterhttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/secure-hive-metastore
Create custom constraintshttps://docs.cloud.google.com/managed-spark/docs/concepts/custom-constraints
Assured Workloadshttps://docs.cloud.google.com/assured-workloads/docs
FedRAMP Compliancehttps://docs.cloud.google.com/managed-spark/docs/concepts/fedramp-compliance
Use Kerberoshttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/security
VPC Service Controlshttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network#dataproc-vpc-sc-networks
Set Workforce accesshttps://docs.cloud.google.com/managed-spark/docs/concepts/accessing/workforce-identity-federation
Investigate workloads with AI-powered Gemini Cloud Assisthttps://docs.cloud.google.com/managed-spark/docs/guides/ai-investigations-with-gca
Monitor batch workloadshttps://docs.cloud.google.com/managed-spark/docs/guides/monitor-troubleshoot-batches
Monitor interactive sessionshttps://docs.cloud.google.com/managed-spark/docs/guides/monitor-troubleshoot-sessions
Spark metricshttps://docs.cloud.google.com/managed-spark/docs/concepts/metrics-serverless
Enable data lineagehttps://docs.cloud.google.com/managed-spark/docs/guides/data-lineage
Profile resource usagehttps://docs.cloud.google.com/managed-spark/docs/guides/profiling
Resource Manager audit logshttps://docs.cloud.google.com/managed-spark/docs/guides/dataprocrm-audit-logging
Troubleshoot batch and session creation failureshttps://docs.cloud.google.com/managed-spark/docs/support/batch-session-creation-failures
Troubleshoot batch and session connectivityhttps://docs.cloud.google.com/managed-spark/docs/support/troubleshoot-connectivity
Investigate clusters and jobs with AI-powered Gemini Cloud Assisthttps://docs.cloud.google.com/managed-spark/docs/guides/gca-clusters
Overviewhttps://docs.cloud.google.com/managed-spark/docs/guides/monitor-clusters-jobs
Managed Service for Apache Spark metricshttps://docs.cloud.google.com/managed-spark/docs/guides/metrics
Create metric alertshttps://docs.cloud.google.com/managed-spark/docs/guides/alerts
Profile resource usagehttps://docs.cloud.google.com/managed-spark/docs/guides/profiling
Managed Service for Apache Spark logshttps://docs.cloud.google.com/managed-spark/docs/guides/logging
Job output logshttps://docs.cloud.google.com/managed-spark/docs/guides/job-output
Audit logshttps://docs.cloud.google.com/managed-spark/docs/guides/audit-logging
View cluster diagnostic datahttps://docs.cloud.google.com/managed-spark/docs/support/diagnose-clusters
Troubleshoot cluster creation issueshttps://docs.cloud.google.com/managed-spark/docs/support/troubleshoot-cluster-creation
Troubleshoot jobshttps://docs.cloud.google.com/managed-spark/docs/concepts/jobs/troubleshoot-jobs
Troubleshoot memory errorshttps://docs.cloud.google.com/managed-spark/docs/support/troubleshoot-oom-errors
Troubleshoot job delayshttps://docs.cloud.google.com/managed-spark/docs/concepts/jobs/troubleshoot-job-delays
View job historyhttps://docs.cloud.google.com/managed-spark/docs/concepts/jobs/history-server
Troubleshoot workflow templateshttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/debugging
Enable Kubernetes logginghttps://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-logging
Diagnose Kubernetes clustershttps://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-diagnose-cluster
Managed Service for Apache Spark MCP serverhttps://docs.cloud.google.com/managed-spark/docs/guides/use-dataproc-mcp
Code sampleshttps://docs.cloud.google.com/dataproc/docs/samples
Get Started with Google Cloud https://docs.cloud.google.com/docs/get-started
Product List https://docs.cloud.google.com/docs/product-list
Cloud Customer Care https://docs.cloud.google.com/support/docs
Agent Platform https://docs.cloud.google.com/products/gemini-enterprise-agent-platform
Apigee API Management https://docs.cloud.google.com/apigee
BigQuery https://docs.cloud.google.com/bigquery
Compute Engine https://docs.cloud.google.com/products/compute
Cloud CDN https://docs.cloud.google.com/cdn
Cloud Run https://docs.cloud.google.com/run
Cloud Storage https://docs.cloud.google.com/storage
Cloud SQL https://docs.cloud.google.com/sql
Gemini Enterprise https://docs.cloud.google.com/gemini/enterprise/docs
Google Kubernetes Engine https://docs.cloud.google.com/kubernetes-engine
Looker https://docs.cloud.google.com/looker
Access and resources management https://docs.cloud.google.com/docs/access-resources
Costs and usage management https://docs.cloud.google.com/docs/costs-usage
Infrastructure as code https://docs.cloud.google.com/docs/iac
SDK, languages, frameworks, and tools https://docs.cloud.google.com/docs/devtools
AI and ML https://docs.cloud.google.com/docs/ai-ml
Application development https://docs.cloud.google.com/docs/application-development
Application hosting https://docs.cloud.google.com/docs/application-hosting
Compute https://docs.cloud.google.com/docs/compute-area
Data analytics and pipelines https://docs.cloud.google.com/docs/data
Databases https://docs.cloud.google.com/docs/databases
Distributed, hybrid, and multicloud https://docs.cloud.google.com/docs/dhm-cloud
Industry solutions https://docs.cloud.google.com/docs/industry
Migration https://docs.cloud.google.com/docs/migration
Networking https://docs.cloud.google.com/docs/networking
Observability and monitoring https://docs.cloud.google.com/docs/observability
Security https://docs.cloud.google.com/docs/security
Storage https://docs.cloud.google.com/docs/storage
Home https://docs.cloud.google.com/
Documentation https://docs.cloud.google.com/docs
Data analytics https://docs.cloud.google.com/docs/data
Managed Service for Apache Spark https://docs.cloud.google.com/managed-spark/docs
Guides https://docs.cloud.google.com/managed-spark/docs/overview/key-concepts
bucketshttps://docs.cloud.google.com/storage/docs/xml-api/put-bucket-create
create an accounthttps://console.cloud.google.com/freetrial
Learn how to grant roleshttps://docs.cloud.google.com/iam/docs/granting-changing-revoking-access
Go to project selectorhttps://console.cloud.google.com/projectselector2/home/dashboard
Verify that billing is enabled for your Google Cloud projecthttps://docs.cloud.google.com/billing/docs/how-to/verify-billing-enabled#confirm_billing_is_enabled_on_a_project
Learn how to grant roleshttps://docs.cloud.google.com/iam/docs/granting-changing-revoking-access
Enable the APIshttps://console.cloud.google.com/apis/enableflow?apiid=dataproc,compute_component,storage-component.googleapis.com,cloudfunctions.googleapis.com
Installhttps://docs.cloud.google.com/sdk/docs/install
sign in to the gcloud CLI with your federated identityhttps://docs.cloud.google.com/iam/docs/workforce-log-in-gcloud
initializehttps://docs.cloud.google.com/sdk/docs/initializing
Learn how to grant roleshttps://docs.cloud.google.com/iam/docs/granting-changing-revoking-access
Go to project selectorhttps://console.cloud.google.com/projectselector2/home/dashboard
Verify that billing is enabled for your Google Cloud projecthttps://docs.cloud.google.com/billing/docs/how-to/verify-billing-enabled#confirm_billing_is_enabled_on_a_project
Learn how to grant roleshttps://docs.cloud.google.com/iam/docs/granting-changing-revoking-access
Enable the APIshttps://console.cloud.google.com/apis/enableflow?apiid=dataproc,compute_component,storage-component.googleapis.com,cloudfunctions.googleapis.com
Installhttps://docs.cloud.google.com/sdk/docs/install
sign in to the gcloud CLI with your federated identityhttps://docs.cloud.google.com/iam/docs/workforce-log-in-gcloud
initializehttps://docs.cloud.google.com/sdk/docs/initializing
Go to Bucketshttps://console.cloud.google.com/storage/browser
bucket naming requirementshttps://docs.cloud.google.com/storage/docs/bucket-naming#requirements
bucket labelhttps://docs.cloud.google.com/storage/docs/tags-and-labels#bucket-labels
Location typehttps://docs.cloud.google.com/storage/docs/locations
Location typehttps://docs.cloud.google.com/storage/docs/locations#available-locations
dual-regionhttps://docs.cloud.google.com/storage/docs/locations#location-dr
turbo replicationhttps://docs.cloud.google.com/storage/docs/availability-durability#turbo-replication
cross-bucket replicationhttps://docs.cloud.google.com/storage/docs/availability-durability#cross-bucket-replication
default storage classhttps://docs.cloud.google.com/storage/docs/storage-classes
Autoclasshttps://docs.cloud.google.com/storage/docs/autoclass
hierarchical namespacehttps://docs.cloud.google.com/storage/docs/hns-overview
public access preventionhttps://docs.cloud.google.com/storage/docs/public-access-prevention
access control methodhttps://docs.cloud.google.com/storage/docs/access-control
organization policyhttps://docs.cloud.google.com/storage/docs/org-policy-constraints#public-access-prevention
soft deletehttps://docs.cloud.google.com/storage/docs/soft-delete
Object Versioninghttps://docs.cloud.google.com/storage/docs/object-versioning
Object Retention Lockhttps://docs.cloud.google.com/storage/docs/object-lock
Bucket Lockhttps://docs.cloud.google.com/storage/docs/bucket-lock
Data encryption methodhttps://docs.cloud.google.com/storage/docs/encryption
workflow templatehttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/using-workflows
Cloud Shellhttps://console.cloud.google.com/?cloudshell=true
regionhttps://docs.cloud.google.com/compute/docs/regions-zones#available
gcloud config set compute/regionhttps://docs.cloud.google.com/sdk/gcloud/reference/config/set#compute
managedhttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/overview#managed_cluster
single-nodehttps://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/single-node-clusters
Workflowshttps://console.cloud.google.com/dataproc/workflows/templates
Parameterizehttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/workflow-parameters
Cloud Run functionshttps://console.cloud.google.com/functions/list
Create a Cloud Storage bucket in your projecthttps://docs.cloud.google.com/managed-spark/docs/tutorials/workflow-function#create_a_bucket_in_your_project
Functionshttps://console.cloud.google.com/functions/list
Storage browserhttps://console.cloud.google.com/storage/browser
Jobshttps://console.cloud.google.com/dataproc/jobs
Go to Manage resourceshttps://console.cloud.google.com/iam-admin/projects
Go to Bucketshttps://console.cloud.google.com/storage/browser
Cloud Run functionshttps://console.cloud.google.com/functions/list
Overview of Managed Service for Apache Spark workflow templateshttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/overview
Workflow scheduling solutionshttps://docs.cloud.google.com/managed-spark/docs/concepts/workflows/workflow-schedule-solutions
Creative Commons Attribution 4.0 Licensehttps://creativecommons.org/licenses/by/4.0/
Apache 2.0 Licensehttps://www.apache.org/licenses/LICENSE-2.0
Google Developers Site Policieshttps://developers.google.com/site-policies
See all products https://cloud.google.com/products/
Google Cloud pricing https://cloud.google.com/pricing/
Google Cloud Marketplace https://cloud.google.com/marketplace/
Contact sales https://cloud.google.com/contact/
Community forums https://discuss.google.dev/c/google-cloud/14/
Support https://cloud.google.com/support-hub/
Release Notes https://docs.cloud.google.com/release-notes
System status https://status.cloud.google.com
GitHub https://github.com/googlecloudPlatform/
Getting Started with Google Cloud https://docs.cloud.google.com/docs/get-started/
Code samples https://docs.cloud.google.com/docs/samples
Cloud Architecture Center https://docs.cloud.google.com/architecture/
Training and Certification https://cloud.google.com/learn/training/
Builders Hub https://me.developers.google.com/
Blog https://cloud.google.com/blog/
Events https://cloud.google.com/events/
X (Twitter) https://x.com/googlecloud
Google Cloud on YouTube https://www.youtube.com/googlecloud
Google Cloud Tech on YouTube https://www.youtube.com/googlecloudplatform
https://developers.google.com/
Android https://developer.android.com/
Chrome https://developer.chrome.com/
Firebase https://firebase.google.com/
Google AI https://ai.google.com/
About Google https://about.google/
Privacy https://policies.google.com/privacy
Site terms https://policies.google.com/terms?hl=en
Google Cloud terms https://cloud.google.com/product-terms
Manage cookies https://docs.cloud.google.com/managed-spark/docs/tutorials/workflow-function
Our third decade of climate action: join us https://cloud.google.com/sustainability
Subscribe https://cloud.google.com/newsletter/

Viewport: width=device-width, initial-scale=1

URLs of crawlers that visited me.