Flatten tuple like a bag

Question

My dataset looks like the following:

( A, (1,2) )
( B, (2,9) )

I would like to "flatten" the tuples in Pig, basically repeating each record for each value found in the inner-tuple, such that the expected output is:

( A, 1 )
( A, 2 )
( B, 2 ) 
( B, 9 )

I know this is possible when the tuples (1,2) and (2,9) are bags instead.

Answer 1

Your insight is good; it's possible by transforming the tuple in a bag. The schema we want to aim for is: {a: chararray,{(chararray)}} for example: (A,{(1),(2)})

Here is the solution to your problem:

A = LOAD 'data.txt' AS (a:chararray,b:(b1:chararray,b2:chararray));
B = FOREACH A GENERATE a, TOBAG(b.b1,b.b2);
C = FOREACH B GENERATE a, FLATTEN($1);

The magic part is the TOBAG operator.

Answer 2

You can use DataFu's UDF TransposeTupleToBag (http://datafu.incubator.apache.org/docs/datafu/1.1.0/datafu/pig/util/TransposeTupleToBag.html) and then flatten the bag to get a row per item in the bag.