Menu
What I need is to download many files from Internet and upload them to S3 as fast as it possible. For now I open every file as stream and upload it one after another. Average speed is 1 picture per second =(
Any ideas?
240
There are no limits to the number of files/folders you can place in a bucket. You are however limited to 100 buckets per account. 100 buckets per account.
Multipart Upload allows you to upload a single object as a set of parts. After all parts of your object are uploaded, Amazon S3 then presents the data as a single object. With this feature you can create parallel uploads, pause and resume an object upload, and begin uploads before you know the total object size.
Split in parts and upload in parallel would be a good start, the TransferManager class can help with that. Also, the article "Pushing the Limits of S3 Upload Performance" has some nice ideas on the subject.
116
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
