Cloud Dataproc API
Manages Hadoop-based clusters and jobs on Google Cloud Platform.
Data & analytics#openapi#analytics#media
Fork copies the collection into your workspace, without the publisher's secrets, so you can add your own key and send requests. Star saves it to your starred APIs (Account › Marketplace) and helps others find it.
Overview
Manages Hadoop-based clusters and jobs on Google Cloud Platform.
Endpoints 66
projects.regions.autoscalingPolicies
- GETLists autoscaling policies in the project.
/v1/:parent/autoscalingPolicies?pageSize=0&pageToken=stringLists autoscaling policies in the project. - POSTCreates new autoscaling policy.
/v1/:parent/autoscalingPoliciesCreates new autoscaling policy.
projects.regions.workflowTemplates
- GETRetrieves the latest workflow template.Can retrieve previously instantiated template by specifying optional version…
/v1/:name?version=0Retrieves the latest workflow template.Can retrieve previously instantiated template by specifying optional version parameter. - PUTUpdates (replaces) workflow template
/v1/:nameUpdates (replaces) workflow template. The updated template must contain version that matches the current server version. - DELETEDeletes a workflow template
/v1/:name?version=0Deletes a workflow template. It does not cancel in-progress workflows. - POSTGets the access control policy for a resource
/v1/:resource:getIamPolicyGets the access control policy for a resource. Returns an empty policy if the resource exists and does not have a policy set. - POSTSets the access control policy on the specified resource
/v1/:resource:setIamPolicySets the access control policy on the specified resource. Replaces any existing policy.Can return NOT_FOUND, INVALID_ARGUMENT, and PERMISSION_DENIED errors. - POSTReturns permissions that a caller has on the specified resource
/v1/:resource:testIamPermissionsReturns permissions that a caller has on the specified resource. If the resource does not exist, this will return an empty set of permissions, not a NOT_FOUND… - GETLists workflows that match the specified filter in the request.
/v1/:parent/workflowTemplates?pageSize=0&pageToken=stringLists workflows that match the specified filter in the request. - POSTCreates new workflow template.
/v1/:parent/workflowTemplatesCreates new workflow template. - POSTInstantiates a template and begins execution.The returned Operation can be used to track execution of workflow by…
/v1/:name:instantiateInstantiates a template and begins execution.The returned Operation can be used to track execution of workflow by polling operations.get. The Operation will… - POSTInstantiates a template and begins execution.This method is equivalent to executing the sequence…
/v1/:parent/workflowTemplates:instantiateInline?requestId=stringInstantiates a template and begins execution.This method is equivalent to executing the sequence CreateWorkflowTemplate, InstantiateWorkflowTemplate,…
projects.locations.sessionTemplates
- PATCHUpdates the session template synchronously.
/v1/:nameUpdates the session template synchronously. - GETLists session templates.
/v1/:parent/sessionTemplates?filter=string&pageSize=0&pageToken=stringLists session templates. - POSTCreate a session template synchronously.
/v1/:parent/sessionTemplatesCreate a session template synchronously.
projects.locations.batches
- POSTAnalyze a Batch for possible recommendations and insights.
/v1/:name:analyzeAnalyze a Batch for possible recommendations and insights. - GETLists batch workloads.
/v1/:parent/batches?filter=string&orderBy=string&pageSize=0&pageToken=stringLists batch workloads. - POSTCreates a batch workload that executes asynchronously.
/v1/:parent/batches?batchId=string&requestId=stringCreates a batch workload that executes asynchronously.
projects.locations.sessions.sparkApplications
- GETObtain high level information corresponding to a single Spark Application.
/v1/:name:access?parent=stringObtain high level information corresponding to a single Spark Application. - GETObtain environment details for a Spark Application
/v1/:name:accessEnvironmentInfo?parent=stringObtain environment details for a Spark Application - GETObtain data corresponding to a spark job for a Spark Application.
/v1/:name:accessJob?jobId=string&parent=stringObtain data corresponding to a spark job for a Spark Application. - GETObtain data corresponding to Native Build Information for a Spark Application.
/v1/:name:accessNativeBuildInfo?parent=stringObtain data corresponding to Native Build Information for a Spark Application. - GETObtain data corresponding to a particular Native SQL Query for a Spark Application.
/v1/:name:accessNativeSqlQuery?executionId=string&parent=stringObtain data corresponding to a particular Native SQL Query for a Spark Application. - GETObtain Spark Plan Graph for a Spark Application SQL execution
/v1/:name:accessSqlPlan?executionId=string&parent=stringObtain Spark Plan Graph for a Spark Application SQL execution. Limits the number of clusters returned as part of the graph to 10000. - GETObtain data corresponding to a particular SQL Query for a Spark Application.
/v1/:name:accessSqlQuery?details=true&executionId=string&parent=string&planDescription=trueObtain data corresponding to a particular SQL Query for a Spark Application. - GETObtain data corresponding to a spark stage attempt for a Spark Application.
/v1/:name:accessStageAttempt?parent=string&stageAttemptId=0&stageId=string&summaryMetricsMask=stringObtain data corresponding to a spark stage attempt for a Spark Application. - GETObtain RDD operation graph for a Spark Application Stage
/v1/:name:accessStageRddGraph?parent=string&stageId=stringObtain RDD operation graph for a Spark Application Stage. Limits the number of clusters returned as part of the graph to 10000. - GETObtain high level information and list of Spark Applications corresponding to a batch
/v1/:parent/sparkApplications:search?applicationStatus=APPLICATION_STATUS_UNSPECIFIED&maxEndTime=string&maxTime=string&minEndTime=string&minTime=string&pageSize=0&pageToken=stringObtain high level information and list of Spark Applications corresponding to a batch - GETObtain executor summary with respect to a spark stage attempt.
/v1/:name:searchExecutorStageSummary?pageSize=0&pageToken=string&parent=string&stageAttemptId=0&stageId=stringObtain executor summary with respect to a spark stage attempt. - GETObtain data corresponding to executors for a Spark Application.
/v1/:name:searchExecutors?executorStatus=EXECUTOR_STATUS_UNSPECIFIED&pageSize=0&pageToken=string&parent=stringObtain data corresponding to executors for a Spark Application. - GETObtain list of spark jobs corresponding to a Spark Application.
/v1/:name:searchJobs?jobIds=string&jobStatus=JOB_EXECUTION_STATUS_UNSPECIFIED&pageSize=0&pageToken=string&parent=stringObtain list of spark jobs corresponding to a Spark Application. - GETObtain data corresponding to Native SQL Queries for a Spark Application.
/v1/:name:searchNativeSqlQueries?pageSize=0&pageToken=string&parent=stringObtain data corresponding to Native SQL Queries for a Spark Application. - GETObtain data corresponding to SQL Queries for a Spark Application.
/v1/:name:searchSqlQueries?details=true&operationIds=string&pageSize=0&pageToken=string&parent=string&planDescription=trueObtain data corresponding to SQL Queries for a Spark Application. - GETObtain data corresponding to tasks for a spark stage attempt for a Spark Application.
/v1/:name:searchStageAttemptTasks?pageSize=0&pageToken=string&parent=string&sortRuntime=true&stageAttemptId=0&stageId=string&taskStatus=TASK_STATUS_UNSPECIFIEDObtain data corresponding to tasks for a spark stage attempt for a Spark Application. - GETObtain data corresponding to a spark stage attempts for a Spark Application.
/v1/:name:searchStageAttempts?pageSize=0&pageToken=string&parent=string&stageId=string&summaryMetricsMask=stringObtain data corresponding to a spark stage attempts for a Spark Application. - GETObtain data corresponding to stages for a Spark Application.
/v1/:name:searchStages?pageSize=0&pageToken=string&parent=string&stageIds=string&stageStatus=STAGE_STATUS_UNSPECIFIED&summaryMetricsMask=stringObtain data corresponding to stages for a Spark Application. - GETObtain summary of Executor Summary for a Spark Application
/v1/:name:summarizeExecutors?parent=stringObtain summary of Executor Summary for a Spark Application - GETObtain summary of Jobs for a Spark Application
/v1/:name:summarizeJobs?jobIds=string&parent=stringObtain summary of Jobs for a Spark Application - GETObtain summary of Tasks for a Spark Application Stage Attempt
/v1/:name:summarizeStageAttemptTasks?parent=string&stageAttemptId=0&stageId=stringObtain summary of Tasks for a Spark Application Stage Attempt - GETObtain summary of Stages for a Spark Application
/v1/:name:summarizeStages?parent=string&stageIds=stringObtain summary of Stages for a Spark Application - POSTWrite wrapper objects from dataplane to spanner
/v1/:name:writeWrite wrapper objects from dataplane to spanner
projects.locations.batches.sparkApplications
- GETReturns autotuning configuration for a specific query plan id and cohort id
/v1/:name:computeTuningConfig?executionId=string&parent=string&semanticQueryId=stringReturns autotuning configuration for a specific query plan id and cohort id. Called by Spark during query planning.
projects.regions.operations
- POSTStarts asynchronous cancellation on a long-running operation
/v1/:name:cancelStarts asynchronous cancellation on a long-running operation. The server makes a best effort to cancel the operation, but success is not guaranteed. If the…
projects.locations.sessions
- GETLists interactive sessions.
/v1/:parent/sessions?filter=string&pageSize=0&pageToken=stringLists interactive sessions. - POSTCreate an interactive session asynchronously.
/v1/:parent/sessions?requestId=string&sessionId=stringCreate an interactive session asynchronously. - POSTTerminates the interactive session.
/v1/:name:terminateTerminates the interactive session.
projects.regions.clusters
- GETLists all regions/{region}/clusters in a project alphabetically.
/v1/projects/:projectId/regions/:region/clusters?filter=string&pageSize=0&pageToken=stringLists all regions/{region}/clusters in a project alphabetically. - POSTCreates a cluster in a project
/v1/projects/:projectId/regions/:region/clusters?actionOnFailedPrimaryWorkers=FAILURE_ACTION_UNSPECIFIED&requestId=stringCreates a cluster in a project. The returned Operation.metadata will be ClusterOperationMetadata… - GETGets the resource representation for a cluster in a project.
/v1/projects/:projectId/regions/:region/clusters/:clusterNameGets the resource representation for a cluster in a project. - DELETEDeletes a cluster in a project
/v1/projects/:projectId/regions/:region/clusters/:clusterName?clusterUuid=string&gracefulTerminationTimeout=string&requestId=stringDeletes a cluster in a project. The returned Operation.metadata will be ClusterOperationMetadata… - PATCHUpdates a cluster in a project
/v1/projects/:projectId/regions/:region/clusters/:clusterName?gracefulDecommissionTimeout=string&requestId=string&updateMask=stringUpdates a cluster in a project. The returned Operation.metadata will be ClusterOperationMetadata… - POSTGets cluster diagnostic information
/v1/projects/:projectId/regions/:region/clusters/:clusterName:diagnoseGets cluster diagnostic information. The returned Operation.metadata will be ClusterOperationMetadata… - POSTInject encrypted credentials into all of the VMs in a cluster.The target cluster must be a personal auth cluster…
/v1/:project/:region/:cluster:injectCredentialsInject encrypted credentials into all of the VMs in a cluster.The target cluster must be a personal auth cluster assigned to the user who is issuing the RPC. - POSTRepairs a cluster.
/v1/projects/:projectId/regions/:region/clusters/:clusterName:repairRepairs a cluster. - POSTStarts a cluster in a project.
/v1/projects/:projectId/regions/:region/clusters/:clusterName:startStarts a cluster in a project. - POSTStops a cluster in a project.
/v1/projects/:projectId/regions/:region/clusters/:clusterName:stopStops a cluster in a project.
projects.regions.clusters.nodeGroups
- POSTCreates a node group in a cluster
/v1/:parent/nodeGroups?nodeGroupId=string&parentOperationId=string&requestId=stringCreates a node group in a cluster. The returned Operation.metadata is NodeGroupOperationMetadata… - POSTRepair nodes in a node group.
/v1/:name:repairRepair nodes in a node group. - POSTResizes a node group in a cluster
/v1/:name:resizeResizes a node group in a cluster. The returned Operation.metadata is NodeGroupOperationMetadata…
projects.regions.jobs
- POSTStarts a job cancellation request
/v1/projects/:projectId/regions/:region/jobs/:jobId:cancelStarts a job cancellation request. To access the job resource after cancellation, call regions/{region}/jobs.list… - GETGets the resource representation for a job in a project.
/v1/projects/:projectId/regions/:region/jobs/:jobIdGets the resource representation for a job in a project. - DELETEDeletes the job from the project
/v1/projects/:projectId/regions/:region/jobs/:jobIdDeletes the job from the project. If the job is active, the delete fails, and the response returns FAILED_PRECONDITION. - PATCHUpdates a job in a project.
/v1/projects/:projectId/regions/:region/jobs/:jobId?updateMask=stringUpdates a job in a project. - GETLists regions/{region}/jobs in a project.
/v1/projects/:projectId/regions/:region/jobs?clusterName=string&filter=string&jobStateMatcher=ALL&pageSize=0&pageToken=stringLists regions/{region}/jobs in a project. - POSTSubmits a job to a cluster.
/v1/projects/:projectId/regions/:region/jobs:submitSubmits a job to a cluster. - POSTSubmits job to a cluster.
/v1/projects/:projectId/regions/:region/jobs:submitAsOperationSubmits job to a cluster.
API details
- Base URL
https://dataproc.googleapis.com- Authentication
- OAuth 2.0 (authorization code flow) with 3 scopes
An API key in the key query parameter - Operations
- 66 operations on 53 pathsGET 33POST 26PATCH 3DELETE 3PUT 1
- Main resources
- projects.locations.sessions.sparkApplications 23projects.regions.clusters 10projects.regions.workflowTemplates 10projects.regions.jobs 7projects.locations.batches 3projects.locations.sessionTemplates 3projects.locations.sessions 3projects.regions.clusters.nodeGroups 3projects.regions.autoscalingPolicies 2projects.locations.batches.sparkApplications 1
- Response formats
application/json- Description format
- Google API Discovery, API version
v1 - Links
- Documentation · OpenAPI document
- Listed from
- Google API Discovery document (googleapis/discovery-artifact-manager) (Apache-2.0)
How to test the Cloud Dataproc API in Swastik Codes
- Open the APISelect “Fork & run in Swastik Codes” on this page. Sign in or create a free account if asked.
- Fork it into your workspaceChoose a workspace. You get your own copy of the Cloud Dataproc API collection, without the publisher's secrets.
- Add your credentialsThis API uses OAuth 2.0 (authorization code flow) with 3 scopes. Add yours in the collection's variables or an environment; the requests already reference it.
- Send a requestPick an endpoint, for example “Lists autoscaling policies in the project.” (GET /v1/:parent/autoscalingPolicies?pageSize=0&pageToken=string), and press Send. Check the response, add tests and run the whole collection with the runner.
Frequently asked questions
What is the Cloud Dataproc API?
Manages Hadoop-based clusters and jobs on Google Cloud Platform. It is listed under Data & analytics and has 66 endpoints.
Who publishes the Cloud Dataproc API?
It is listed from Google's public API description (an OpenAPI document), not published by Google on Swastik Codes. Anyone who controls googleapis.com can claim the profile by adding a DNS TXT record.
What is the base URL of the Cloud Dataproc API?
Requests go to https://dataproc.googleapis.com. Endpoint paths are added to this base URL.
How do I authenticate with the Cloud Dataproc API?
It accepts 2 methods: OAuth 2.0 (authorization code flow) with 3 scopes; an API key in the key query parameter.
How many endpoints does the Cloud Dataproc API have?
66 operations on 53 paths (33 GET, 26 POST, 3 PATCH, 3 DELETE, 1 PUT). The largest groups are projects.locations.sessions.sparkApplications, projects.regions.clusters, projects.regions.workflowTemplates, projects.regions.jobs.
Is forking the Cloud Dataproc API free?
Yes. Forking copies the collection into your workspace for free. It has been starred 0 times and forked 0 times.