I am pretty new to Pig and I have a a very basic question : can I make make Pig load all files from a directory including the ones in the subfolders ? Here is how I proceed :
records = LOAD '/worldwide/data/' USING PigStorage() AS (event:chararray, user:chararray);
Here, repo/data
may have subfolders such as
repo/data/region/cluster1
repo/data/region/cluster2
Can I get it to load everything from both those subdirectories and any new directories that might get added at a future date?
confirmed that the above statement just works and loads all of the data from the subdirectories into the records variable.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With