Menu
Does amazon s3 support batch uploads? I have a job that needs to upload each night ~100K of files that can be up to 1G but is strongly skewed towards small files (90% are less than 100 bytes and 99% are less than 1000 bytes long).
Does the s3 API support uploading multiple objects in a single HTTP call?
All the objects must be available in S3 as individual objects. I cannot host them anywhere else (FTP, etc) or in another format (Database, EC2 local drive, etc). That is an external requirement that I cannot change.
517
There are no limits to the number of files/folders you can place in a bucket. You are however limited to 100 buckets per account. 100 buckets per account. You can make as many accounts as you need, it would seem.
S3 Batch Operations is a managed solution for performing storage actions like copying and tagging objects at scale, whether for one-time tasks or for recurring, batch workloads. S3 Batch Operations can perform actions across billions of objects and petabytes of data with a single request.
S3 batch operations in action: the implementation However, AWS informs us that it may take up to 48 hours to deliver the first report!
Alternatively, you can upload S3 via AWS CLI tool using the sync command.
aws s3 sync local_folder s3://bucket-name
You can use this method to batch upload files to S3 very fast.
58
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
