My hadoop job generate large number of files on HDFS and I want to write a separate thread which will copy these files from HDFS to S3.
Could any one point me to any java API that handles it.
Thanks
"Support for the S3 block filesystem was added to the ${HADOOP_HOME}/bin/hadoop distcp tool in Hadoop 0.11.0 (See HADOOP-862). The distcp tool sets up a MapReduce job to run the copy. Using distcp, a cluster of many members can copy lots of data quickly. The number of map tasks is calculated by counting the number of files in the source: i.e. each map task is responsible for the copying one file. Source and target may refer to disparate filesystem types. For example, source might refer to the local filesystem or hdfs with S3 as the target. "
Check out Running Bulk Copies in and out of S3 here http://wiki.apache.org/hadoop/AmazonS3
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With