What is the difference between Google Cloud Dataflow and Google Cloud Dataproc?

Question

I am using Google Data Flow to implement an ETL data ware house solution.

Looking into google cloud offering, it seems DataProc can also do the same thing.

It also seems DataProc is little bit cheaper than DataFlow.

Does anybody know the pros / cons of DataFlow over DataProc

Why does google offer both?

Answer 1

Yes, Cloud Dataflow and Cloud Dataproc can both be used to implement ETL data warehousing solutions.

An overview of why each of these products exist can be found in the Google Cloud Platform Big Data Solutions Articles

Quick takeaways:

• Cloud Dataproc provides you with a Hadoop cluster, on GCP, and access to Hadoop-ecosystem tools (e.g. Apache Pig, Hive, and Spark); this has strong appeal if you are already familiar with Hadoop tools and have Hadoop jobs
• Cloud Dataflow provides you with a place to run Apache Beam based jobs, on GCP, and you do not need to address common aspects of running jobs on a cluster (e.g. Balancing work, or Scaling the number of workers for a job; by default, this is automatically managed for you, and applies to both batch and streaming) -- this can be very time consuming on other systems
  • Apache Beam is an important consideration; Beam jobs are intended to be portable across "runners," which include Cloud Dataflow, and enable you to focus on your logical computation, rather than how a "runner" works -- In comparison, when authoring a Spark job, your code is bound to the runner, Spark, and how that runner works
  • Cloud Dataflow also offers the ability to create jobs based on "templates," which can help simplify common tasks where the differences are parameter values

Answer 2

Here are three main points to consider while trying to choose between Dataproc and Dataflow

• Provisioning
  Dataproc - Manual provisioning of clusters
  Dataflow - Serverless. Automatic provisioning of clusters

• Hadoop Dependencies
  Dataproc should be used if the processing has any dependencies to tools in the Hadoop ecosystem.

• Portability
  Dataflow/Beam provides a clear separation between processing logic and the underlying execution engine. This helps with portability across different execution engines that support the Beam runtime, i.e. the same pipeline code can run seamlessly on either Dataflow, Spark or Flink.

This flowchart from the google website explains how to go about choosing one over the other.

https://cloud.google.com/dataflow/images/flow-vs-proc-flowchart.svg

Further details are available in the below link
https://cloud.google.com/dataproc/#fast--scalable-data-processing