AWS Data Pipeline
AWS Data Pipeline configures and manages a data-driven workflow called a pipeline. AWS Data Pipeline handles the details of scheduling and ensuring that data…
Data & analytics#openapi#cloud
Fork copies the collection into your workspace, without the publisher's secrets, so you can add your own key and send requests. Star saves it to your starred APIs (Account › Marketplace) and helps others find it.
Overview
AWS Data Pipeline configures and manages a data-driven workflow called a pipeline. AWS Data Pipeline
handles the details of scheduling and ensuring that data dependencies are met so that your application
can focus on processing the data.
AWS Data Pipeline provides a JAR implementation of a task runner called AWS Data Pipeline Task Runner.
AWS Data Pipeline Task Runner provides logic for common data management scenarios, such as performing
database queries and running data analysis using Amazon Elastic MapReduce (Amazon EMR). You can use
AWS Data Pipeline Task Runner as your task runner, or you can write your own task runner to provide
custom data management.
AWS Data Pipeline implements two main sets of functionality. Use the first set to create a pipeline
and define data sources, schedules, dependencies, and the transforms to be performed on the data.
Use the second set in your task runner application to receive the next task ready for processing.
The logic for performing the task, such as querying the data, running data analysis, or converting
the data from one format to another, is contained within the task runner. The task runner performs
the task assigned to it by the web service, reporting progress to the web service as it does so.
When the task is done, the task runner reports the final success or failure of the task to the web service.
Endpoints 19
- POSTActivate pipeline
/#X-Amz-Target=DataPipeline.ActivatePipelineValidates the specified pipeline and starts processing pipeline tasks. If the pipeline does not pass validation, activation fails. If you need to pause the… - POSTAdd tags
/#X-Amz-Target=DataPipeline.AddTagsAdds or modifies tags for the specified pipeline. - POSTCreate pipeline
/#X-Amz-Target=DataPipeline.CreatePipelineCreates a new, empty pipeline. Use PutPipelineDefinition to populate the pipeline. POST / HTTP/1.1 Content-Type: application/x-amz-json-1.1 X-Amz-Target:… - POSTDeactivate pipeline
/#X-Amz-Target=DataPipeline.DeactivatePipelineDeactivates the specified running pipeline. The pipeline is set to the DEACTIVATING state until the deactivation process completes. To resume a deactivated… - POSTDelete pipeline
/#X-Amz-Target=DataPipeline.DeletePipelineDeletes a pipeline, its pipeline definition, and its run history. AWS Data Pipeline attempts to cancel instances associated with the pipeline that are… - POSTDescribe objects
/#X-Amz-Target=DataPipeline.DescribeObjectsGets the object definitions for a set of objects associated with the pipeline. Object definitions are composed of a set of fields that define the properties… - POSTDescribe pipelines
/#X-Amz-Target=DataPipeline.DescribePipelinesRetrieves metadata about one or more pipelines. The information retrieved includes the name of the pipeline, the pipeline identifier, its current state, and… - POSTEvaluate expression
/#X-Amz-Target=DataPipeline.EvaluateExpressionTask runners call EvaluateExpression to evaluate a string in the context of the specified object. For example, a task runner can evaluate SQL queries stored… - POSTGet pipeline definition
/#X-Amz-Target=DataPipeline.GetPipelineDefinitionGets the definition of the specified pipeline. You can call GetPipelineDefinition to retrieve the pipeline definition that you provided using… - POSTList pipelines
/#X-Amz-Target=DataPipeline.ListPipelinesLists the pipeline identifiers for all active pipelines that you have permission to access. POST / HTTP/1.1 Content-Type: application/x-amz-json-1.1… - POSTPoll for task
/#X-Amz-Target=DataPipeline.PollForTaskTask runners call PollForTask to receive a task to perform from AWS Data Pipeline. The task runner specifies which tasks it can perform by setting a value for… - POSTPut pipeline definition
/#X-Amz-Target=DataPipeline.PutPipelineDefinitionAdds tasks, schedules, and preconditions to the specified pipeline. You can use PutPipelineDefinition to populate a new pipeline. PutPipelineDefinition also… - POSTQuery objects
/#X-Amz-Target=DataPipeline.QueryObjectsQueries the specified pipeline for the names of objects that match the specified set of conditions. POST / HTTP/1.1 Content-Type: application/x-amz-json-1.1… - POSTRemove tags
/#X-Amz-Target=DataPipeline.RemoveTagsRemoves existing tags from the specified pipeline. - POSTReport task progress
/#X-Amz-Target=DataPipeline.ReportTaskProgressTask runners call ReportTaskProgress when assigned a task to acknowledge that it has the task. If the web service does not receive this acknowledgement within… - POSTReport task runner heartbeat
/#X-Amz-Target=DataPipeline.ReportTaskRunnerHeartbeatTask runners call ReportTaskRunnerHeartbeat every 15 minutes to indicate that they are operational. If the AWS Data Pipeline Task Runner is launched on a… - POSTSet status
/#X-Amz-Target=DataPipeline.SetStatusRequests that the status of the specified physical or logical pipeline objects be updated in the specified pipeline. This update might not occur immediately,… - POSTSet task status
/#X-Amz-Target=DataPipeline.SetTaskStatusTask runners call SetTaskStatus to notify AWS Data Pipeline that a task is completed and provide information about the final status. A task runner makes this… - POSTValidate pipeline definition
/#X-Amz-Target=DataPipeline.ValidatePipelineDefinitionValidates the specified pipeline definition to ensure that it is well formed and can be run without error. Example 1 This example sets an valid pipeline…
API details
- Base URL
https://datapipeline.{region}.amazonaws.com- Authentication
- AWS Signature Version 4 (requests signed with your AWS access key)
- Operations
- 19 operations on 19 pathsPOST 19
- Description format
- AWS Smithy model, API version
2012-10-29 - Links
- Documentation · OpenAPI document
- Listed from
- AWS Smithy API model (aws/api-models-aws) (Apache-2.0)
How to test the AWS Data Pipeline in Swastik Codes
- Open the APISelect “Fork & run in Swastik Codes” on this page. Sign in or create a free account if asked.
- Fork it into your workspaceChoose a workspace. You get your own copy of the AWS Data Pipeline collection, without the publisher's secrets.
- Add your credentialsThis API uses AWS Signature Version 4 (requests signed with your AWS access key). Add yours in the collection's variables or an environment; the requests already reference it.
- Send a requestPick an endpoint, for example “Activate pipeline” (POST /#X-Amz-Target=DataPipeline.ActivatePipeline), and press Send. Check the response, add tests and run the whole collection with the runner.
Frequently asked questions
What is the AWS Data Pipeline?
AWS Data Pipeline configures and manages a data-driven workflow called a pipeline. AWS Data Pipeline handles the details of scheduling and ensuring that data… It is listed under Data & analytics and has 19 endpoints.
Who publishes the AWS Data Pipeline?
It is listed from Amazon Web Services, Inc.'s public API description (an OpenAPI document), not published by Amazon Web Services, Inc. on Swastik Codes. Anyone who controls amazonaws.com can claim the profile by adding a DNS TXT record.
What is the base URL of the AWS Data Pipeline?
Requests go to https://datapipeline.{region}.amazonaws.com. Endpoint paths are added to this base URL.
How do I authenticate with the AWS Data Pipeline?
With AWS Signature Version 4 (requests signed with your AWS access key).
How many endpoints does the AWS Data Pipeline have?
19 operations on 19 paths (19 POST).
Is forking the AWS Data Pipeline free?
Yes. Forking copies the collection into your workspace for free. It has been starred 0 times and forked 0 times.