Using Google Cloud Dataflow for merging flat files and importing into Cloud SQL

Question

We have to read data from CSV files and map two files with respect to one column and push data to Cloud SQL using Google Cloud Dataflow.

We are able to read data from CSV files but stuck with the next steps. Please provide me information or links regarding the following:

1. Merging/joining to flat files based on one column or condition with multiple columns
2. Copying merged pcollection into Сloud SQL database

Answer 1

Here's some pointers that may be helpful:

• https://cloud.google.com/dataflow/model/joins describes the ways to join PCollection's in Dataflow
• There is currently no built-in sink for writing to CloudSQL, however you can either simply process the results of your join using a ParDo which writes each individual record or in batches (flushing periodically or in finishBundle()) - or if your needs are more complex than that, consider writing a CloudSQL sink - see https://cloud.google.com/dataflow/model/sources-and-sinks