Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Amazon S3 Java SDK multiple files upload

What I need is to download many files from Internet and upload them to S3 as fast as it possible. For now I open every file as stream and upload it one after another. Average speed is 1 picture per second =(

Any ideas?

like image 240
Pavel Vyazankin Avatar asked Mar 27 '14 19:03

Pavel Vyazankin


People also ask

How many files can you upload to S3 at once?

There are no limits to the number of files/folders you can place in a bucket. You are however limited to 100 buckets per account. 100 buckets per account.

How does multipart upload work on S3?

Multipart Upload allows you to upload a single object as a set of parts. After all parts of your object are uploaded, Amazon S3 then presents the data as a single object. With this feature you can create parallel uploads, pause and resume an object upload, and begin uploads before you know the total object size.


1 Answers

Split in parts and upload in parallel would be a good start, the TransferManager class can help with that. Also, the article "Pushing the Limits of S3 Upload Performance" has some nice ideas on the subject.

like image 116
Julio Faerman Avatar answered Sep 29 '22 16:09

Julio Faerman