Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to load files recursively using apache pig

I am pretty new to Pig and I have a a very basic question : can I make make Pig load all files from a directory including the ones in the subfolders ? Here is how I proceed :

records = LOAD '/worldwide/data/' USING PigStorage() AS (event:chararray, user:chararray);

Here, repo/data may have subfolders such as

repo/data/region/cluster1
repo/data/region/cluster2 

Can I get it to load everything from both those subdirectories and any new directories that might get added at a future date?

like image 716
Gr-Disarray Avatar asked Nov 01 '22 03:11

Gr-Disarray


1 Answers

confirmed that the above statement just works and loads all of the data from the subdirectories into the records variable.

like image 185
Gr-Disarray Avatar answered Nov 15 '22 06:11

Gr-Disarray