Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS : What's the difference between Simple Workflow Service and Data Pipeline?

What's the difference between Amazon Simple Workflow Service and Amazon Data Pipeline ? It seems that they are pretty much the same product. The Data Pipeline has a nice web based diagram editor though.

Cheers !

like image 312
MadSeb Avatar asked Apr 22 '13 23:04

MadSeb


People also ask

What is a data pipeline in AWS?

AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.

What is the difference between AWS glue and AWS data pipeline?

AWS Glue provides support for Amazon S3, Amazon RDS, Redshift, SQL, and DynamoDB and also provides built-in transformations. On the other hand, AWS Data Pipeline allows you to create data transformations through APIs and also through JSON, while only providing support for DynamoDB, SQL, and Redshift.

What is the main purpose of using Amazon SWF?

Amazon Simple Workflow Service (Amazon SWF) is a web service that makes it easy to coordinate work across distributed application components.

For which use cases can simple workflow service be used?

It is suitable for a range of use cases that require coordination of tasks, including media processing, web application back-ends, business process workflows, and analytics pipelines.


2 Answers

From http://aws.amazon.com/datapipeline/faqs/

Q: How is AWS Data Pipeline different from Amazon Simple Workflow Service?

While both services provide execution tracking, retry and exception-handling capabilities, and the ability to run arbitrary actions, AWS Data Pipeline is specifically designed to facilitate the specific steps that are common across a majority of data-driven workflows – inparticular, executing activities after their input data meets specific readiness criteria, easily copying data between different data stores, and scheduling chained transforms. This highly specific focus means that its workflow definitions can be created [with] very rapidly and with no code or programming knowledge.

like image 74
qix Avatar answered Sep 21 '22 05:09

qix


Data Pipeline is service used to transfer data between various services of AWS. Example you can use DataPipeline to read the log files from your EC2 and periodically move them to S3.

Simple Workflow service is very powerful service. You can write even your workflow logic using it. Example : Most of the ecommerce systems have scalability problems in their order systems. You can use write code in SWF to make this ordering workflow process itself.

like image 27
Harish Ganesan Avatar answered Sep 18 '22 05:09

Harish Ganesan