Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

s3distcp srcPattern not working?

I have files like this in S3:

1-2013-08-22-22-something
2-2013-08-22-22-something
etc

without srcPattern I can get all of the files from the bucket easily but I want to get a specific prefix, for example all of the 1's. I've tried using srcPattern but for some reason it's not picking up any of the files.

My current command is:

elastic-mapreduce --jobflow $JOBFLOW --jar /home/hadoop/lib/emr-s3distcp-1.0.jar \
--args '--src,s3n://some-bucket/,--dest,hdfs:///hdfs-input,--srcPattern,[0-9]-.*' \
--step-name "copying over s3 files" 
like image 433
Julian Avatar asked Mar 23 '23 16:03

Julian


1 Answers

Turns out you need .* in front of the regex

for example I needed

.*[0-9]-.*

I'm guessing because the source pattern also includes the bucket name?

like image 165
Julian Avatar answered Apr 01 '23 07:04

Julian