Menu
Have a few small files on Amazon-S3 and wondering if it's possible to get 3-4 of them in a single request.
Looked around docs and few SDK's and didn't find anything obvious. I saw they now have "multi-delete", which is nice, but multi get would be great
Anyone know of something like this is possible?
Thanks
641
You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an Amazon S3 bucket. There are no limits to the number of prefixes that you can have in your bucket. Note: LIST and GET objects don't share the same limit.
S3 provides unlimited scalability, and there is no official limit on the amount of data and number of objects you can store in an S3 bucket. The size limit for objects stored in a bucket is 5 TB.
Many common AWS S3 libraries (including the widely used s3cmd) do not by default make many connections at once to transfer data. Both s4cmd and AWS' own aws-cli do make concurrent connections, and are much faster for many files or large transfers (since multipart uploads allow parallelism).
You can use the NotPrincipal element of an IAM or S3 bucket policy to limit resource access to a specific set of users. This element allows you to block all users who are not defined in its value array, even if they have an Allow in their own IAM user policies.
If you take a look to the lowest level documentation (REST API, for example), the GET operation on objects brings you only one object per request, so whatever you find out in any of AWS SDKs will be a loop over this kind of request.
There are some tools that make it easier to download more than one object. For example, in command-line tools:s3cmd get object1 object2 object3 s3://bucket-name
This call will make sequential requests, meaning the total time won't be reduced, but it indeed makes your task easier to accomplish.
If you prefer doing it in a programmatic way, I suggest using whichever SDK that makes more sense for your platform, and, if it does not contain a batch GET object operation yet, implement your own version according to your own preferences.
199
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
