Google

Dataflow API

Manages Google Cloud Dataflow projects on Google Cloud Platform.

Data & analytics#openapi#analytics#media

Fork copies the collection into your workspace, without the publisher's secrets, so you can add your own key and send requests. Star saves it to your starred APIs (Account › Marketplace) and helps others find it.

Overview

Manages Google Cloud Dataflow projects on Google Cloud Platform.

Endpoints 47

projects.locations.configStoreSettings

  • GETLists ConfigStoreSettings./v1b3/:parent/configStoreSettings?pageSize=0&pageToken=stringLists ConfigStoreSettings.
  • POSTCreates a new ConfigStoreSetting./v1b3/:parent/configStoreSettings?configStoreSettingId=stringCreates a new ConfigStoreSetting.
  • GETGets a ConfigStoreSetting./v1b3/:nameGets a ConfigStoreSetting.
  • DELETEDeletes an existing ConfigStoreSetting./v1b3/:nameDeletes an existing ConfigStoreSetting.
  • POSTResolves effective value of a ConfigStoreSetting./v1b3/:name:resolveResolves effective value of a ConfigStoreSetting.

projects.snapshots

  • GETLists snapshots./v1b3/projects/:projectId/snapshots?jobId=string&location=stringLists snapshots.
  • GETGets information about a snapshot./v1b3/projects/:projectId/snapshots/:snapshotId?location=stringGets information about a snapshot.

projects

  • DELETEDeletes a snapshot./v1b3/projects/:projectId/snapshots?location=string&snapshotId=stringDeletes a snapshot.
  • POSTSend a worker_message to the service./v1b3/projects/:projectId/WorkerMessagesSend a worker_message to the service.

projects.jobs

  • GETList the jobs of a project across all regions/v1b3/projects/:projectId/jobs:aggregated?createTimeRange.endTime=string&createTimeRange.startTime=string&filter=UNKNOWN&location=string&name=string&pageSize=0&pageToken=string&regionalFanoutRequested=true&terminationTimeRange.endTime=string&terminationTimeRange.startTime=string&view=JOB_VIEW_UNKNOWNList the jobs of a project across all regions. Note: This method doesn't support filtering the list of jobs by name. # IAM Permissions Requires the…
  • GETList the jobs of a project/v1b3/projects/:projectId/jobs?createTimeRange.endTime=string&createTimeRange.startTime=string&filter=UNKNOWN&location=string&name=string&pageSize=0&pageToken=string&regionalFanoutRequested=true&terminationTimeRange.endTime=string&terminationTimeRange.startTime=string&view=JOB_VIEW_UNKNOWNList the jobs of a project. To list the jobs of a project in a region, we recommend using projects.locations.jobs.list with a [regional endpoint]…
  • POSTCreates a Dataflow job/v1b3/projects/:projectId/jobs?location=string&replaceJobId=string&view=JOB_VIEW_UNKNOWNCreates a Dataflow job. To create a job, we recommend using projects.locations.jobs.create with a [regional endpoint]…
  • GETGets the state of the specified Cloud Dataflow job/v1b3/projects/:projectId/jobs/:jobId?location=string&view=JOB_VIEW_UNKNOWNGets the state of the specified Cloud Dataflow job. To get the state of a job, we recommend using projects.locations.jobs.get with a [regional endpoint]…
  • PUTUpdates the state of an existing Cloud Dataflow job/v1b3/projects/:projectId/jobs/:jobId?location=string&updateMask=stringUpdates the state of an existing Cloud Dataflow job. To update the state of an existing job, we recommend using projects.locations.jobs.update with a…
  • GETRequest the job status/v1b3/projects/:projectId/jobs/:jobId/metrics?location=string&startTime=stringRequest the job status. To request the status of a job, we recommend using projects.locations.jobs.getMetrics with a [regional endpoint]…
  • POSTSnapshot the state of a streaming job/v1b3/projects/:projectId/jobs/:jobId:snapshotSnapshot the state of a streaming job. # IAM Permissions Requires the dataflow.jobs.snapshot permission on the job.

projects.jobs.debug

  • POSTGet encoded debug configuration for component/v1b3/projects/:projectId/jobs/:jobId/debug/getConfigGet encoded debug configuration for component. Not cacheable.
  • POSTSend encoded debug capture data for component./v1b3/projects/:projectId/jobs/:jobId/debug/sendCaptureSend encoded debug capture data for component.

projects.jobs.messages

  • GETRequest the job status/v1b3/projects/:projectId/jobs/:jobId/messages?endTime=string&location=string&minimumImportance=JOB_MESSAGE_IMPORTANCE_UNKNOWN&pageSize=0&pageToken=string&startTime=stringRequest the job status. To request the status of a job, we recommend using projects.locations.jobs.messages.list with a [regional endpoint]…

projects.jobs.workItems

  • POSTLeases a dataflow WorkItem to run./v1b3/projects/:projectId/jobs/:jobId/workItems:leaseLeases a dataflow WorkItem to run.
  • POSTReports the status of dataflow WorkItems leased by a worker./v1b3/projects/:projectId/jobs/:jobId/workItems:reportStatusReports the status of dataflow WorkItems leased by a worker.

projects.locations

  • POSTSend a worker_message to the service./v1b3/projects/:projectId/locations/:location/WorkerMessagesSend a worker_message to the service.

projects.locations.flexTemplates

  • POSTLaunch a job with a FlexTemplate/v1b3/projects/:projectId/locations/:location/flexTemplates:launchLaunch a job with a FlexTemplate. # IAM Permissions Requires the following IAM permission(s) on the resource: - dataflow.jobs.create -…

projects.locations.jobs

  • GETList the jobs of a project/v1b3/projects/:projectId/locations/:location/jobs?createTimeRange.endTime=string&createTimeRange.startTime=string&filter=UNKNOWN&name=string&pageSize=0&pageToken=string&regionalFanoutRequested=true&terminationTimeRange.endTime=string&terminationTimeRange.startTime=string&view=JOB_VIEW_UNKNOWNList the jobs of a project. To list the jobs of a project in a region, we recommend using projects.locations.jobs.list with a [regional endpoint]…
  • POSTCreates a Dataflow job/v1b3/projects/:projectId/locations/:location/jobs?replaceJobId=string&view=JOB_VIEW_UNKNOWNCreates a Dataflow job. To create a job, we recommend using projects.locations.jobs.create with a [regional endpoint]…
  • GETGets the state of the specified Cloud Dataflow job/v1b3/projects/:projectId/locations/:location/jobs/:jobId?view=JOB_VIEW_UNKNOWNGets the state of the specified Cloud Dataflow job. To get the state of a job, we recommend using projects.locations.jobs.get with a [regional endpoint]…
  • PUTUpdates the state of an existing Cloud Dataflow job/v1b3/projects/:projectId/locations/:location/jobs/:jobId?updateMask=stringUpdates the state of an existing Cloud Dataflow job. To update the state of an existing job, we recommend using projects.locations.jobs.update with a…
  • GETRequest detailed information about the execution status of the job/v1b3/projects/:projectId/locations/:location/jobs/:jobId/executionDetails?pageSize=0&pageToken=stringRequest detailed information about the execution status of the job. EXPERIMENTAL. This API is subject to change or removal without notice. # IAM Permissions…
  • GETRequest the job status/v1b3/projects/:projectId/locations/:location/jobs/:jobId/metrics?startTime=stringRequest the job status. To request the status of a job, we recommend using projects.locations.jobs.getMetrics with a [regional endpoint]…
  • POSTSnapshot the state of a streaming job/v1b3/projects/:projectId/locations/:location/jobs/:jobId:snapshotSnapshot the state of a streaming job. # IAM Permissions Requires the dataflow.jobs.snapshot permission on the job.

projects.locations.jobs.debug

  • POSTGet encoded debug configuration for component/v1b3/projects/:projectId/locations/:location/jobs/:jobId/debug/getConfigGet encoded debug configuration for component. Not cacheable.
  • POSTGet worker stacktraces from debug capture./v1b3/projects/:projectId/locations/:location/jobs/:jobId/debug/getWorkerStacktracesGet worker stacktraces from debug capture.
  • POSTSend encoded debug capture data for component./v1b3/projects/:projectId/locations/:location/jobs/:jobId/debug/sendCaptureSend encoded debug capture data for component.

projects.locations.jobs.messages

  • GETRequest the job status/v1b3/projects/:projectId/locations/:location/jobs/:jobId/messages?endTime=string&minimumImportance=JOB_MESSAGE_IMPORTANCE_UNKNOWN&pageSize=0&pageToken=string&startTime=stringRequest the job status. To request the status of a job, we recommend using projects.locations.jobs.messages.list with a [regional endpoint]…

projects.locations.jobs.snapshots

  • GETLists snapshots./v1b3/projects/:projectId/locations/:location/jobs/:jobId/snapshotsLists snapshots.

projects.locations.jobs.stages

  • GETRequest detailed information about the execution status of a stage of the job/v1b3/projects/:projectId/locations/:location/jobs/:jobId/stages/:stageId/executionDetails?endTime=string&pageSize=0&pageToken=string&startTime=stringRequest detailed information about the execution status of a stage of the job. EXPERIMENTAL. This API is subject to change or removal without notice. # IAM…

projects.locations.jobs.workItems

  • POSTLeases a dataflow WorkItem to run./v1b3/projects/:projectId/locations/:location/jobs/:jobId/workItems:leaseLeases a dataflow WorkItem to run.
  • POSTReports the status of dataflow WorkItems leased by a worker./v1b3/projects/:projectId/locations/:location/jobs/:jobId/workItems:reportStatusReports the status of dataflow WorkItems leased by a worker.

projects.locations.snapshots

  • GETGets information about a snapshot./v1b3/projects/:projectId/locations/:location/snapshots/:snapshotIdGets information about a snapshot.
  • DELETEDeletes a snapshot./v1b3/projects/:projectId/locations/:location/snapshots/:snapshotIdDeletes a snapshot.
  • GETLists snapshots./v1b3/projects/:projectId/locations/:location/snapshots?jobId=stringLists snapshots.

projects.locations.templates

  • POSTCreates a Cloud Dataflow job from a template/v1b3/projects/:projectId/locations/:location/templatesCreates a Cloud Dataflow job from a template. Do not enter confidential information when you supply string values using the API. To create a job, we recommend…
  • GETGet the template associated with a template/v1b3/projects/:projectId/locations/:location/templates:get?gcsPath=string&view=METADATA_ONLYGet the template associated with a template. To get the template, we recommend using projects.locations.templates.get with a [regional endpoint]…
  • POSTLaunches a template/v1b3/projects/:projectId/locations/:location/templates:launch?dynamicTemplate.gcsPath=string&dynamicTemplate.stagingLocation=string&gcsPath=string&validateOnly=trueLaunches a template. To launch a template, we recommend using projects.locations.templates.launch with a [regional endpoint]…

projects.templates

  • POSTCreates a Cloud Dataflow job from a template/v1b3/projects/:projectId/templatesCreates a Cloud Dataflow job from a template. Do not enter confidential information when you supply string values using the API. To create a job, we recommend…
  • GETGet the template associated with a template/v1b3/projects/:projectId/templates:get?gcsPath=string&location=string&view=METADATA_ONLYGet the template associated with a template. To get the template, we recommend using projects.locations.templates.get with a [regional endpoint]…
  • POSTLaunches a template/v1b3/projects/:projectId/templates:launch?dynamicTemplate.gcsPath=string&dynamicTemplate.stagingLocation=string&gcsPath=string&location=string&validateOnly=trueLaunches a template. To launch a template, we recommend using projects.locations.templates.launch with a [regional endpoint]…

API details

Base URL
https://dataflow.googleapis.com
Authentication
OAuth 2.0 (authorization code flow) with 2 scopes
An API key in the key query parameter
Operations
47 operations on 39 pathsPOST 22GET 20DELETE 3PUT 2
Main resources
projects.jobs 7projects.locations.jobs 7projects.locations.configStoreSettings 5projects.locations.jobs.debug 3projects.locations.snapshots 3projects.locations.templates 3projects.templates 3projects 2projects.jobs.debug 2projects.jobs.workItems 2
Response formats
application/json
Description format
Google API Discovery, API version v1b3

How to test the Dataflow API in Swastik Codes

  1. Open the APISelect “Fork & run in Swastik Codes” on this page. Sign in or create a free account if asked.
  2. Fork it into your workspaceChoose a workspace. You get your own copy of the Dataflow API collection, without the publisher's secrets.
  3. Add your credentialsThis API uses OAuth 2.0 (authorization code flow) with 2 scopes. Add yours in the collection's variables or an environment; the requests already reference it.
  4. Send a requestPick an endpoint, for example “Lists ConfigStoreSettings.” (GET /v1b3/:parent/configStoreSettings?pageSize=0&pageToken=string), and press Send. Check the response, add tests and run the whole collection with the runner.

Frequently asked questions

What is the Dataflow API?

Manages Google Cloud Dataflow projects on Google Cloud Platform. It is listed under Data & analytics and has 47 endpoints.

Who publishes the Dataflow API?

It is listed from Google's public API description (an OpenAPI document), not published by Google on Swastik Codes. Anyone who controls googleapis.com can claim the profile by adding a DNS TXT record.

What is the base URL of the Dataflow API?

Requests go to https://dataflow.googleapis.com. Endpoint paths are added to this base URL.

How do I authenticate with the Dataflow API?

It accepts 2 methods: OAuth 2.0 (authorization code flow) with 2 scopes; an API key in the key query parameter.

How many endpoints does the Dataflow API have?

47 operations on 39 paths (22 POST, 20 GET, 3 DELETE, 2 PUT). The largest groups are projects.jobs, projects.locations.jobs, projects.locations.configStoreSettings, projects.locations.jobs.debug.

Is forking the Dataflow API free?

Yes. Forking copies the collection into your workspace for free. It has been starred 0 times and forked 0 times.

Suggest an API

Tell us which public API is missing. We check it, then add it here with its endpoints when it has an OpenAPI (Swagger) description.