Flink : How to process and output two datasets in a single transformation?

Question

The join and coGroup transformation can read 2 input datasets and output one ("Y" flux) (correct me if I'm wrong).

I would like to process and update 2 datasets. To do this, I plan to use 2 coGroup transformations.

But, for performance purpose, can these both transformations be done in a single one ("H" flux)?

Also, as the datasets are updated, I would like to iterate over them. If it's not currently possible, do you plan to support this kind of transformation in the future?

Answer 1

All Flink DataSet operators support only a single output, but the output of an operator can be consumed by two or more following operators.

There are two ways to solve your issue:

1. Use a single CoGroup to compute the result for both outputs and add two Filters that filter out the records of one of both outputs. If both outputs have different data types, you need to compute return something like Tuple2<FirstType, SecondType>. This solution would look like:

    input1--\         /--> Filter_output1 
              CoGroup 
    input2--/         \--> Filter_output2

2. Partition and sort both CoGroup inputs on the grouping key and call two individual CoGroups. Each CoGroup computes one output. By sorting the data before the CoGroup, the partitioning and sorting can be reused. Important, all operators must use the same parallelism!

    input1 --> PartitionHash --> SortPartition -\-/-> CoGroup1 --> Output1
                                                 X
    input2 --> PartitionHash --> SortPartition -/-\-> CoGroup2 --> Output2

Regarding the iterations, have a look at Flink's iteration operators.