|
Skip to main content
| https://cloud.google.com/dataproc/docs/tutorials/python-library-example#main-content |
|
| https://cloud.google.com/ |
|
Technology areas
| https://docs.cloud.google.com/docs |
|
AI and ML
| https://docs.cloud.google.com/docs/ai-ml |
|
Application development
| https://docs.cloud.google.com/docs/application-development |
|
Application hosting
| https://docs.cloud.google.com/docs/application-hosting |
|
Compute
| https://docs.cloud.google.com/docs/compute-area |
|
Data analytics and pipelines
| https://docs.cloud.google.com/docs/data |
|
Databases
| https://docs.cloud.google.com/docs/databases |
|
Distributed, hybrid, and multicloud
| https://docs.cloud.google.com/docs/dhm-cloud |
|
Industry solutions
| https://docs.cloud.google.com/docs/industry |
|
Migration
| https://docs.cloud.google.com/docs/migration |
|
Networking
| https://docs.cloud.google.com/docs/networking |
|
Observability and monitoring
| https://docs.cloud.google.com/docs/observability |
|
Security
| https://docs.cloud.google.com/docs/security |
|
Storage
| https://docs.cloud.google.com/docs/storage |
|
Cross-product tools
| https://docs.cloud.google.com/docs/cross-product-overviews |
|
Access and resources management
| https://docs.cloud.google.com/docs/access-resources |
|
Costs and usage management
| https://docs.cloud.google.com/docs/costs-usage |
|
Infrastructure as code
| https://docs.cloud.google.com/docs/iac |
|
SDK, languages, frameworks, and tools
| https://docs.cloud.google.com/docs/devtools |
|
Console
| https://console.cloud.google.com/ |
|
| https://docs.cloud.google.com/managed-spark/docs |
|
Managed Service for Apache Spark
| https://docs.cloud.google.com/managed-spark/docs |
| Start free | https://console.cloud.google.com/freetrial |
|
Overview
| https://docs.cloud.google.com/managed-spark/docs |
|
Guides
| https://docs.cloud.google.com/managed-spark/docs/overview/key-concepts |
|
Reference
| https://docs.cloud.google.com/managed-spark/docs/reference |
|
Samples
| https://docs.cloud.google.com/managed-spark/docs/samples |
|
Resources
| https://docs.cloud.google.com/managed-spark/docs/resources |
|
| https://cloud.google.com/ |
|
Technology areas
| https://cloud.google.com/docs |
|
Overview
| https://cloud.google.com/managed-spark/docs |
|
Guides
| https://cloud.google.com/managed-spark/docs/overview/key-concepts |
|
Reference
| https://cloud.google.com/managed-spark/docs/reference |
|
Samples
| https://cloud.google.com/managed-spark/docs/samples |
|
Resources
| https://cloud.google.com/managed-spark/docs/resources |
|
Cross-product tools
| https://cloud.google.com/docs/cross-product-overviews |
|
Console
| https://console.cloud.google.com/ |
| Key Concepts | https://cloud.google.com/managed-spark/docs/overview/key-concepts |
| Overview | https://cloud.google.com/managed-spark/docs/serverless-overview |
| Managed Service for Apache Spark serverless tiers | https://cloud.google.com/managed-spark/docs/tiers |
| Managed Service for Apache Spark on clusters | https://cloud.google.com/managed-spark/docs/concepts/clusters-overview |
| Compare serverless and cluster deployments | https://cloud.google.com/managed-spark/docs/concepts/serverless-spark-compare |
| Managed Service for Apache Spark on GKE | https://cloud.google.com/managed-spark/docs/concepts/gke-overview |
| Create a serverless Spark batch workload | https://cloud.google.com/managed-spark/docs/quickstarts/spark-batch |
| Create a cluster | https://cloud.google.com/managed-spark/docs/quickstarts/create-cluster |
| Submit a Spark job to a cluster | https://cloud.google.com/managed-spark/docs/guides/submit-job |
| Use Gemini to develop Spark applications | https://cloud.google.com/managed-spark/docs/guides/gemini-spark |
| Analyze public datasets with Spark | https://cloud.google.com/managed-spark/docs/tutorials/monte-carlo-methods-with-hadoop-spark |
| Sentiment analysis with Spark MLlib | https://cloud.google.com/managed-spark/docs/tutorials/bigquery-sparkml |
| Run a Spark job on Kubernetes | https://cloud.google.com/managed-spark/docs/guides/dpgke/quickstarts/gke-quickstart-create-cluster |
| Use custom containers | https://cloud.google.com/managed-spark/docs/guides/custom-containers |
| Use GPUs | https://cloud.google.com/managed-spark/docs/guides/gpus-serverless |
| Use Dynamic Workload Scheduler | https://cloud.google.com/managed-spark/docs/guides/dws-serverless |
| Network configuration | https://cloud.google.com/managed-spark/docs/concepts/network-serverless |
| Overview | https://cloud.google.com/managed-spark/docs/concepts/versions/serverless-versions |
| Spark runtime version 3.0 | https://cloud.google.com/managed-spark/docs/concepts/versions/spark-runtime-3.0 |
| Spark runtime version 2.3 | https://cloud.google.com/managed-spark/docs/concepts/versions/spark-runtime-2.3 |
| Spark runtime version 2.2 | https://cloud.google.com/managed-spark/docs/concepts/versions/spark-runtime-2.2 |
| Spark runtime version 1.2 | https://cloud.google.com/managed-spark/docs/concepts/versions/spark-runtime-1.2 |
| Service accounts | https://cloud.google.com/managed-spark/docs/concepts/service-accounts-serverless |
| Spark properties | https://cloud.google.com/managed-spark/docs/concepts/spark-properties-serverless |
| Staging bucket | https://cloud.google.com/managed-spark/docs/concepts/staging-bucket-serverless |
| Create a serverless Spark batch workload | https://cloud.google.com/managed-spark/docs/guides/spark-batch |
| Create serverless interactive sessions and session templates | https://cloud.google.com/managed-spark/docs/guides/create-serverless-sessions-templates |
| Use the serverless Spark Connect client | https://cloud.google.com/managed-spark/docs/guides/spark-connect-client |
| Use JupyterLab for serverless batch and notebook sessions | https://cloud.google.com/managed-spark/docs/guides/jupyterlab-sessions-serverless |
| Overview | https://cloud.google.com/managed-spark/docs/templates/templates |
| Cloud Spanner to Cloud Storage | https://cloud.google.com/managed-spark/docs/templates/spanner-to-storage |
| Cloud Storage to BigQuery | https://cloud.google.com/managed-spark/docs/templates/storage-to-bigquery |
| Cloud Storage to Cloud Spanner | https://cloud.google.com/managed-spark/docs/templates/storage-to-spanner |
| Cloud Storage to Cloud Storage | https://cloud.google.com/managed-spark/docs/templates/storage-to-storage |
| Cloud Storage to JDBC | https://cloud.google.com/managed-spark/docs/templates/storage-to-jdbc |
| Hive to BigQuery | https://cloud.google.com/managed-spark/docs/templates/hive-to-bigquery |
| Hive to Cloud Storage | https://cloud.google.com/managed-spark/docs/templates/hive-to-storage |
| JDBC to BigQuery | https://cloud.google.com/managed-spark/docs/templates/jdbc-to-bigquery |
| JDBC to Cloud Spanner | https://cloud.google.com/managed-spark/docs/templates/jdbc-to-spanner |
| JDBC to Cloud Storage | https://cloud.google.com/managed-spark/docs/templates/jdbc-to-storage |
| JDBC to JDBC | https://cloud.google.com/managed-spark/docs/templates/jdbc-to-jdbc |
| Pub/Sub to Cloud Storage | https://cloud.google.com/managed-spark/docs/templates/pubsub-to-storage |
| Create an Apache Iceberg table with metadata in Lakehouse runtime catalog | https://cloud.google.com/managed-spark/docs/guides/spark-workloads-with-bigquery-metastore |
| Overview | https://cloud.google.com/managed-spark/docs/guides/spark-bigquery-connector |
| Query BigQuery tables | https://cloud.google.com/managed-spark/docs/guides/query-bigquery-tables |
| Run PySpark code in BigQuery Studio notebooks | https://cloud.google.com/managed-spark/docs/use-spark |
| Autoscale workload resources | https://cloud.google.com/managed-spark/docs/concepts/autoscaling-serverless |
| Autotune Spark workloads | https://cloud.google.com/managed-spark/docs/concepts/autotuning |
| Accelerate batch workloads and sessions with Lightning Engine | https://cloud.google.com/managed-spark/docs/guides/lightning-engine-serverless |
| Run the Native Query Execution qualification tool | https://cloud.google.com/managed-spark/docs/guides/run-nqe-tool |
| Serverless Spark solution accelerators | https://github.com/GoogleCloudPlatform/serverless-spark-workshop |
| Manage Spark dependencies | https://cloud.google.com/managed-spark/docs/guides/manage-spark-dependencies |
| Customize Spark environment | https://cloud.google.com/managed-spark/docs/guides/docker-yarn |
| Enable concurrent writes | https://cloud.google.com/managed-spark/docs/guides/fileoutput-committer |
| Enhance Spark performance | https://cloud.google.com/managed-spark/docs/guides/performance-enhancements |
| Tune Spark | https://cloud.google.com/managed-spark/docs/support/spark-job-tuning |
| Use Lightning Engine | https://cloud.google.com/managed-spark/docs/guides/lightning-engine |
| Use the console | https://cloud.google.com/managed-spark/docs/quickstarts/create-cluster-console |
| Use the command line | https://cloud.google.com/managed-spark/docs/quickstarts/create-cluster-gcloud |
| Create a cluster | https://cloud.google.com/managed-spark/docs/quickstarts/create-cluster-template |
| Run a Spark job | https://cloud.google.com/managed-spark/docs/quickstarts/submit-spark-job-template |
| Update a cluster | https://cloud.google.com/managed-spark/docs/quickstarts/update-cluster-template |
| Delete a cluster | https://cloud.google.com/managed-spark/docs/quickstarts/quickstart-explorer-delete |
| Use client libraries | https://cloud.google.com/managed-spark/docs/quickstarts/create-cluster-client-libraries |
| Run Hadoop jobs | https://cloud.google.com/managed-spark/docs/tutorials/bigquery-connector-mapreduce-example |
| Write and run Spark Scala jobs | https://cloud.google.com/managed-spark/docs/tutorials/spark-scala |
| Run Trino | https://cloud.google.com/managed-spark/docs/tutorials/trino-dataproc |
| Run Flink | https://cloud.google.com/managed-spark/docs/concepts/components/flink |
| Run Hive | https://cloud.google.com/managed-spark/docs/tutorials/kafka-tutorial |
| Run Pig | https://cloud.google.com/managed-spark/docs/concepts/components/pig |
| Run HBase | https://cloud.google.com/managed-spark/docs/tutorials/spark-hbase |
| Configure the Python environment | https://cloud.google.com/managed-spark/docs/tutorials/python-configuration |
| Use Cloud Client Libraries for Python | https://cloud.google.com/managed-spark/docs/tutorials/python-library-example |
| Overview | https://cloud.google.com/managed-spark/docs/tutorials/bigquery-connector-spark-example |
| BigQuery connector code samples | https://cloud.google.com/managed-spark/docs/examples/bigquery-example |
| Use the Cloud Storage connector | https://cloud.google.com/managed-spark/docs/tutorials/gcs-connector-spark-tutorial |
| Use the Spark Spanner connector | https://cloud.google.com/managed-spark/docs/tutorials/spanner-connector-spark-example |
| Explore and extract data | https://cloud.google.com/managed-spark/docs/guides/explore-data |
| Transform data | https://cloud.google.com/managed-spark/docs/guides/transform-data |
| Load data into BigQuery | https://cloud.google.com/managed-spark/docs/guides/load-data |
| Create a lake house with Spark and BigQuery | https://cloud.google.com/managed-spark/docs/guides/create-lakehouse |
| Configure metastores | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/secure-hive-metastore |
| Create an Apache Iceberg table with metadata in BigLake metastore | https://cloud.google.com/managed-spark/docs/guides/iceberg-metadata-biglake-metastore |
| Use Iceberg | https://cloud.google.com/managed-spark/docs/concepts/components/iceberg |
| Use Delta | https://cloud.google.com/managed-spark/docs/concepts/components/delta |
| Use Hudi | https://cloud.google.com/managed-spark/docs/concepts/components/hudi |
| Overview | https://cloud.google.com/managed-spark/docs/concepts/notebooks |
| Run a Jupyter notebook on a cluster | https://cloud.google.com/managed-spark/docs/tutorials/jupyter-notebook |
| Run a genomics analysis on a notebook | https://cloud.google.com/managed-spark/docs/tutorials/genomics-single-cell |
| Use the JupyterLab extension to develop serverless Spark workloads | https://cloud.google.com/managed-spark/docs/guides/jupyterlab-sessions-serverless |
| Use the Component Gateway | https://cloud.google.com/managed-spark/docs/concepts/accessing/gateways |
| Connect to data sources | https://cloud.google.com/managed-spark/docs/guides/connect-to-data-sources |
| Connect to Cloud Storage | https://cloud.google.com/managed-spark/docs/concepts/connectors/cloud-storage |
| Connect to BigQuery | https://cloud.google.com/managed-spark/docs/concepts/connectors/bigquery |
| Connect Hive to BigQuery | https://cloud.google.com/managed-spark/docs/concepts/connectors/hive-bigquery |
| Connect to Bigtable | https://cloud.google.com/managed-spark/docs/concepts/connectors/cloud-bigtable |
| Connect to Pub/Sub Lite | https://cloud.google.com/managed-spark/docs/concepts/connectors/pubsub-lite |
| Fleet management | https://cloud.google.com/managed-spark/docs/guides/fleet-management |
| Enable Spark data lineage | https://cloud.google.com/managed-spark/docs/guides/spark-lineage |
| Enable Hive data lineage | https://cloud.google.com/managed-spark/docs/guides/hive-lineage |
| Overview | https://cloud.google.com/managed-spark/docs/concepts/components/overview |
| Delta Lake | https://cloud.google.com/managed-spark/docs/concepts/components/delta |
| Docker | https://cloud.google.com/managed-spark/docs/concepts/components/docker |
| Flink | https://cloud.google.com/managed-spark/docs/concepts/components/flink |
| HBase | https://cloud.google.com/managed-spark/docs/concepts/components/hbase |
| Hive WebHCat | https://cloud.google.com/managed-spark/docs/concepts/components/hivewebhcat |
| Hudi | https://cloud.google.com/managed-spark/docs/concepts/components/hudi |
| Iceberg | https://cloud.google.com/managed-spark/docs/concepts/components/iceberg |
| Jupyter | https://cloud.google.com/managed-spark/docs/concepts/components/jupyter |
| Pig | https://cloud.google.com/managed-spark/docs/concepts/components/pig |
| Presto | https://cloud.google.com/managed-spark/docs/concepts/components/presto |
| Install Ranger | https://cloud.google.com/managed-spark/docs/concepts/components/ranger |
| Use Ranger with Kerberos | https://cloud.google.com/managed-spark/docs/concepts/components/ranger-w-kerberos |
| Use Ranger with caching and downscoping | https://cloud.google.com/managed-spark/docs/concepts/components/ranger-caching-downscoping |
| Back up and restore a Ranger schema | https://cloud.google.com/managed-spark/docs/concepts/components/backup-ranger-schema |
| Solr | https://cloud.google.com/managed-spark/docs/concepts/components/solr |
| Trino | https://cloud.google.com/managed-spark/docs/concepts/components/trino |
| Zeppelin | https://cloud.google.com/managed-spark/docs/concepts/components/zeppelin |
| Zookeeper | https://cloud.google.com/managed-spark/docs/concepts/components/zookeeper |
| Overview | https://cloud.google.com/managed-spark/docs/concepts/versioning/overview |
| Services | https://cloud.google.com/managed-spark/docs/concepts/services |
| Overview | https://cloud.google.com/managed-spark/docs/concepts/versioning/image-version-lists |
| 3.0.x release image versions | https://cloud.google.com/managed-spark/docs/concepts/versioning/image-release-3.0 |
| 2.3.x release image versions | https://cloud.google.com/managed-spark/docs/concepts/versioning/image-release-2.3 |
| 2.2.x release image versions | https://cloud.google.com/managed-spark/docs/concepts/versioning/image-release-2.2 |
| 2.1.x release image versions | https://cloud.google.com/managed-spark/docs/concepts/versioning/image-release-2.1 |
| Machine types | https://cloud.google.com/managed-spark/docs/concepts/compute/supported-machine-types |
| GPUs | https://cloud.google.com/managed-spark/docs/concepts/compute/gpus |
| Minimum CPU platform | https://cloud.google.com/managed-spark/docs/concepts/compute/min-cpu |
| Flexible VMs | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/flexible-vms |
| Secondary workers | https://cloud.google.com/managed-spark/docs/concepts/compute/secondary-vms |
| Local solid state drives | https://cloud.google.com/managed-spark/docs/concepts/compute/local-ssds |
| Boot disks | https://cloud.google.com/managed-spark/docs/concepts/compute/persistent-disks |
| Attached hyperdisks | https://cloud.google.com/managed-spark/docs/concepts/compute/attached-disks |
| Overview | https://cloud.google.com/managed-spark/docs/guides/create-cluster |
| Overview | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network |
| Use secure tags | https://cloud.google.com/managed-spark/docs/guides/use-secure-tags |
| Configure Private Service Connect | https://cloud.google.com/managed-spark/docs/networking-private-service-connect |
| Select a cluster region | https://cloud.google.com/managed-spark/docs/concepts/cluster-region |
| Enable auto zone placement | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/auto-zone |
| Schedule cluster deletion | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/scheduled-deletion |
| Use initialization actions | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/init-actions |
| Set cluster metadata | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/metadata |
| Set cluster properties | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/cluster-properties |
| Enable cluster web interfaces | https://cloud.google.com/managed-spark/docs/concepts/accessing/cluster-web-interfaces |
| Create a high availability cluster | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/high-availability |
| Create a node group cluster | https://cloud.google.com/managed-spark/docs/guides/node-groups/driver-node-groups |
| Create a partial cluster | https://cloud.google.com/managed-spark/docs/guides/create-partial-cluster |
| Create a zero-scale cluster | https://cloud.google.com/managed-spark/docs/guides/create-zero-scale-cluster |
| Create a single-node cluster | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/single-node-clusters |
| Create sole-tenant cluster | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/sole-tenant-nodes |
| Create a custom image | https://cloud.google.com/managed-spark/docs/guides/images |
| Set labels for filtering | https://cloud.google.com/managed-spark/docs/guides/creating-managing-labels |
| Scale clusters | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/scaling-clusters |
| Overview | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/autoscaling |
| Schedule cluster stop | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/scheduled-stop |
| Start and stop a cluster manually | https://cloud.google.com/managed-spark/docs/guides/start-stop |
| Recreate a cluster | https://cloud.google.com/managed-spark/docs/guides/recreate-cluster |
| Rotate clusters | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/cluster-rotation |
| Update and delete a cluster | https://cloud.google.com/managed-spark/docs/guides/manage-cluster |
| Use SSH to connect to a cluster | https://cloud.google.com/managed-spark/docs/concepts/accessing/ssh |
| Follow best practices | https://cloud.google.com/managed-spark/docs/guides/best-practices |
| Hadoop data storage | https://cloud.google.com/managed-spark/docs/concepts/hdfs |
| Select storage type | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/staging-bucket |
| Cache cluster data | https://cloud.google.com/managed-spark/docs/concepts/cluster-caching |
| Offload shuffle data | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/enhanced-flexibility-mode |
| View cluster logs | https://cloud.google.com/managed-spark/docs/guides/logging#cluster-logs |
| Life of a job | https://cloud.google.com/managed-spark/docs/concepts/jobs/life-of-a-job |
| Restart jobs | https://cloud.google.com/managed-spark/docs/concepts/jobs/restartable-jobs |
| Overview | https://cloud.google.com/managed-spark/docs/concepts/workflows/overview |
| Parameterization | https://cloud.google.com/managed-spark/docs/concepts/workflows/workflow-parameters |
| Use YAML files | https://cloud.google.com/managed-spark/docs/concepts/workflows/use-yamls |
| Use cluster selectors | https://cloud.google.com/managed-spark/docs/concepts/workflows/cluster-selectors |
| Use inline workflows | https://cloud.google.com/managed-spark/docs/concepts/workflows/inline-workflows |
| GitHub templates | https://cloud.google.com/managed-spark/docs/guides/templates |
| Workflow scheduling solutions | https://cloud.google.com/managed-spark/docs/concepts/workflows/workflow-schedule-solutions |
| Use workflow templates | https://cloud.google.com/managed-spark/docs/concepts/workflows/use-workflows |
| Use Cloud Composer | https://cloud.google.com/managed-spark/docs/tutorials/workflow-composer |
| Use Cloud Functions | https://cloud.google.com/managed-spark/docs/tutorials/workflow-function |
| Use Cloud Scheduler | https://cloud.google.com/managed-spark/docs/tutorials/workflow-scheduler |
| Create node pools | https://cloud.google.com/managed-spark/docs/guides/dpgke/gke-nodepools |
| Recreate a GKE virtual cluster | https://cloud.google.com/managed-spark/docs/guides/dpgke/gke-recreate-cluster |
| Create a GKE custom custom container image | https://cloud.google.com/managed-spark/docs/guides/dpgke/gke-custom-images |
| Scale GKE clusters | https://cloud.google.com/managed-spark/docs/guides/dpgke/gke-scale-cluster |
| Delete a GKE virtualcluster | https://cloud.google.com/managed-spark/docs/guides/dpgke/gke-delete-cluster |
| Release versions | https://cloud.google.com/managed-spark/docs/guides/dpgke/gke-versions |
| Permissions and roles | https://cloud.google.com/managed-spark/docs/concepts/iam-serverless |
| Use CMEK with Serverless for Apache Spark | https://cloud.google.com/managed-spark/docs/guides/cmek-serverless |
| Default security measures | https://cloud.google.com/managed-spark/docs/concepts/security-serverless |
| Use Secret Manager | https://cloud.google.com/managed-spark/docs/guides/hadoop-google-secret-manager-credential-provider-serverless |
| Create custom constraints | https://cloud.google.com/managed-spark/docs/guides/custom-constraints-serverless |
| Security best practices | https://cloud.google.com/managed-spark/docs/concepts/security-best-practices |
| Authenticate to Managed Service for Apache Spark | https://cloud.google.com/managed-spark/docs/authentication |
| Authenticate personal clusters | https://cloud.google.com/managed-spark/docs/concepts/iam/personal-auth |
| Workforce identity federation | https://cloud.google.com/managed-spark/docs/concepts/accessing/workforce-identity-federation |
| Permissions and roles | https://cloud.google.com/managed-spark/docs/concepts/iam/iam |
| Managed Service for Apache Spark principals | https://cloud.google.com/managed-spark/docs/concepts/iam/principals |
| Granular IAM | https://cloud.google.com/managed-spark/docs/concepts/iam/granular-iam |
| Assign roles for Kubernetes | https://cloud.google.com/managed-spark/docs/guides/dpgke/gke-iam |
| Service accounts | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/service-accounts |
| Secure multi-tenancy using Kerberos | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/security |
| Secure multi-tenancy using service accounts | https://cloud.google.com/managed-spark/docs/concepts/iam/sa-multi-tenancy |
| Encrypt memory | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/confidential-compute |
| Manage data encryption keys | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/customer-managed-encryption |
| Enable Ranger authorization service | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/ranger-plugin |
| Use the Secret Manager credential provider | https://cloud.google.com/managed-spark/docs/guides/hadoop-google-secret-manager-credential-provider |
| Create and secure a Hive metastore cluster | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/secure-hive-metastore |
| Create custom constraints | https://cloud.google.com/managed-spark/docs/concepts/custom-constraints |
| Assured Workloads | https://cloud.google.com/assured-workloads/docs |
| FedRAMP Compliance | https://cloud.google.com/managed-spark/docs/concepts/fedramp-compliance |
| Use Kerberos | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/security |
| VPC Service Controls | https://cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network#dataproc-vpc-sc-networks |
| Set Workforce access | https://cloud.google.com/managed-spark/docs/concepts/accessing/workforce-identity-federation |
| Investigate workloads with AI-powered Gemini Cloud Assist | https://cloud.google.com/managed-spark/docs/guides/ai-investigations-with-gca |
| Monitor batch workloads | https://cloud.google.com/managed-spark/docs/guides/monitor-troubleshoot-batches |
| Monitor interactive sessions | https://cloud.google.com/managed-spark/docs/guides/monitor-troubleshoot-sessions |
| Spark metrics | https://cloud.google.com/managed-spark/docs/concepts/metrics-serverless |
| Enable data lineage | https://cloud.google.com/managed-spark/docs/guides/data-lineage |
| Profile resource usage | https://cloud.google.com/managed-spark/docs/guides/profiling |
| Resource Manager audit logs | https://cloud.google.com/managed-spark/docs/guides/dataprocrm-audit-logging |
| Troubleshoot batch and session creation failures | https://cloud.google.com/managed-spark/docs/support/batch-session-creation-failures |
| Troubleshoot batch and session connectivity | https://cloud.google.com/managed-spark/docs/support/troubleshoot-connectivity |
| Investigate clusters and jobs with AI-powered Gemini Cloud Assist | https://cloud.google.com/managed-spark/docs/guides/gca-clusters |
| Overview | https://cloud.google.com/managed-spark/docs/guides/monitor-clusters-jobs |
| Managed Service for Apache Spark metrics | https://cloud.google.com/managed-spark/docs/guides/metrics |
| Create metric alerts | https://cloud.google.com/managed-spark/docs/guides/alerts |
| Profile resource usage | https://cloud.google.com/managed-spark/docs/guides/profiling |
| Managed Service for Apache Spark logs | https://cloud.google.com/managed-spark/docs/guides/logging |
| Job output logs | https://cloud.google.com/managed-spark/docs/guides/job-output |
| Audit logs | https://cloud.google.com/managed-spark/docs/guides/audit-logging |
| View cluster diagnostic data | https://cloud.google.com/managed-spark/docs/support/diagnose-clusters |
| Troubleshoot cluster creation issues | https://cloud.google.com/managed-spark/docs/support/troubleshoot-cluster-creation |
| Troubleshoot jobs | https://cloud.google.com/managed-spark/docs/concepts/jobs/troubleshoot-jobs |
| Troubleshoot memory errors | https://cloud.google.com/managed-spark/docs/support/troubleshoot-oom-errors |
| Troubleshoot job delays | https://cloud.google.com/managed-spark/docs/concepts/jobs/troubleshoot-job-delays |
| View job history | https://cloud.google.com/managed-spark/docs/concepts/jobs/history-server |
| Troubleshoot workflow templates | https://cloud.google.com/managed-spark/docs/concepts/workflows/debugging |
| Enable Kubernetes logging | https://cloud.google.com/managed-spark/docs/guides/dpgke/gke-logging |
| Diagnose Kubernetes clusters | https://cloud.google.com/managed-spark/docs/guides/dpgke/gke-diagnose-cluster |
| Managed Service for Apache Spark MCP server | https://cloud.google.com/managed-spark/docs/guides/use-dataproc-mcp |
| Code samples | https://cloud.google.com/dataproc/docs/samples |
|
AI and ML
| https://cloud.google.com/docs/ai-ml |
|
Application development
| https://cloud.google.com/docs/application-development |
|
Application hosting
| https://cloud.google.com/docs/application-hosting |
|
Compute
| https://cloud.google.com/docs/compute-area |
|
Data analytics and pipelines
| https://cloud.google.com/docs/data |
|
Databases
| https://cloud.google.com/docs/databases |
|
Distributed, hybrid, and multicloud
| https://cloud.google.com/docs/dhm-cloud |
|
Industry solutions
| https://cloud.google.com/docs/industry |
|
Migration
| https://cloud.google.com/docs/migration |
|
Networking
| https://cloud.google.com/docs/networking |
|
Observability and monitoring
| https://cloud.google.com/docs/observability |
|
Security
| https://cloud.google.com/docs/security |
|
Storage
| https://cloud.google.com/docs/storage |
|
Access and resources management
| https://cloud.google.com/docs/access-resources |
|
Costs and usage management
| https://cloud.google.com/docs/costs-usage |
|
Infrastructure as code
| https://cloud.google.com/docs/iac |
|
SDK, languages, frameworks, and tools
| https://cloud.google.com/docs/devtools |
|
Home
| https://docs.cloud.google.com/ |
|
Documentation
| https://docs.cloud.google.com/docs |
|
Data analytics
| https://docs.cloud.google.com/docs/data |
|
Managed Service for Apache Spark
| https://docs.cloud.google.com/managed-spark/docs |
|
Guides
| https://docs.cloud.google.com/managed-spark/docs/overview/key-concepts |
| Cloud Shell walkthrough | https://cloud.google.com/shell/docs/tutorials |
| Google Cloud client libraries for Python | https://cloud.google.com/python/docs/reference/dataproc/latest |
| Managed Service for Apache Spark gRPC APIs | https://cloud.google.com/managed-spark/docs/reference/rpc |
| GoogleCloudPlatform/python-dataproc | https://github.com/googleapis/google-cloud-python/tree/main/packages/google-cloud-dataproc/samples/generated_samples |
| inline workflow | https://cloud.google.com/managed-spark/docs/reference/rpc/google.cloud.dataproc.v1#google.cloud.dataproc.v1.WorkflowTemplateService.InstantiateInlineWorkflowTemplate |
| instantiate_inline_workflow_template.py | https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-dataproc/samples/generated_samples/dataproc_v1_generated_workflow_template_service_instantiate_inline_workflow_template_async.py |
|
Open in Cloud Shell
| https://ssh.cloud.google.com/cloudshell/open?cloudshell_git_repo=https://github.com/googleapis/python-dataproc&cloudshell_working_dir=samples/snippets&tutorial=python-api-walkthrough.md&cloudshell_open_in_editor=submit_job_to_cluster.py |
| Cloud Shell walkthrough | https://cloud.google.com/dataproc/docs/tutorials/python-library-example#run-walkthrough |
| service account credentials | https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually |
| CreateCluster | https://cloud.google.com/managed-spark/docs/reference/rpc/google.cloud.dataproc.v1#google.cloud.dataproc.v1.ClusterController.CreateCluster |
| SubmitJob | https://cloud.google.com/managed-spark/docs/reference/rpc/google.cloud.dataproc.v1#google.cloud.dataproc.v1.JobController.SubmitJob |
| DeleteCluster | https://cloud.google.com/managed-spark/docs/reference/rpc/google.cloud.dataproc.v1#google.cloud.dataproc.v1.JobController.DeleteCluster |
| Creative Commons Attribution 4.0 License | https://creativecommons.org/licenses/by/4.0/ |
| Apache 2.0 License | https://www.apache.org/licenses/LICENSE-2.0 |
| Google Developers Site Policies | https://developers.google.com/site-policies |
|
See all products
| https://cloud.google.com/products/ |
|
Google Cloud pricing
| https://cloud.google.com/pricing/ |
|
Google Cloud Marketplace
| https://cloud.google.com/marketplace/ |
|
Contact sales
| https://cloud.google.com/contact/ |
|
Community forums
| https://discuss.google.dev/c/google-cloud/14/ |
|
Support
| https://cloud.google.com/support-hub/ |
|
Release Notes
| https://docs.cloud.google.com/release-notes |
|
System status
| https://status.cloud.google.com |
|
GitHub
| https://github.com/googlecloudPlatform/ |
|
Getting Started with Google Cloud
| https://cloud.google.com/docs/get-started/ |
|
Code samples
| https://cloud.google.com/docs/samples |
|
Cloud Architecture Center
| https://cloud.google.com/architecture/ |
|
Training and Certification
| https://cloud.google.com/learn/training/ |
|
Blog
| https://cloud.google.com/blog/ |
|
Events
| https://cloud.google.com/events/ |
|
X (Twitter)
| https://x.com/googlecloud |
|
Google Cloud on YouTube
| https://www.youtube.com/googlecloud |
|
Google Cloud Tech on YouTube
| https://www.youtube.com/googlecloudplatform |
|
About Google
| https://about.google/ |
|
Privacy
| https://policies.google.com/privacy |
|
Site terms
| https://policies.google.com/terms?hl=en |
|
Google Cloud terms
| https://cloud.google.com/product-terms |
|
Manage cookies
| https://cloud.google.com/dataproc/docs/tutorials/python-library-example |
|
Our third decade of climate action: join us
| https://cloud.google.com/sustainability |
|
Subscribe
| https://cloud.google.com/newsletter/ |