AWS S3 presigned URL limit

Question

Is there any limit on the number of pre signed URL's per object in AWS S3 presigned URL's. Say If I want to create 1000 presigned url's per object in a 2 minutes. Is that valid scenario ?

Answer 1

You can create as many signed URLs as you wish. Depending on your motivation and strategy, however, there is a practical limitation on the number of unique presigned URLs for the exact same object.

S3 (in S3 regions that were first deployed before 2014) supports two authentication algorithms, V2 and V4, and the signed urls look very different since the algorithms are very different.

In V2, the signed URL for a given expiration time will always look the same, if signed by the same AWS key.

If you sign the url for an object, set to expire one minute in the future... and immediately repeat the process, the two signed URLs will be identical.

Next, exactly one second later, sign a url for the same object to expire 59 seconds in the future, and that new signed URL will also be identical.

Why? Because in V2, the expiration time is an absolute wall clock time in UTC, and the particular time in history when you actually generated the signed URL doesn't change anything.

V4 is different. In the scenario above, the first two would still be identical, but the second one would not, because V4 auth includes the date and time when you created the signed url, or when you say you did. The expiration time is relative to the signing time, instead of absolute.

Note that both forms of signed URL are tamper-resistant -- the expiration time is embedded in the url, but attempting to tweak it after signing will invalidate the signing and make it useless.

If you need to generate a large number of signed urls for the same object, you'll need to increment the expiration time for each individual signing attempt in order to get unique values. (Edit: or not, if you're feeling clever... see below).

It also occurs to me that you may be under the impression that S3 has an active role in the signing process, but it doesn't. That's all done in your local code.

S3 isn't aware, in any sense, of the signed urls you generate unless or until they are used. When a signed request arrives, S3 does exactly the same thing your code will do -- it canonicalizes certain attributes of the request, and generates a signature. Then it compares what it generated with what your code should have generated, given precisely the same parameters. If their generated signature matches your provided signature (and the key you used has permission to perform the requested action) then the request succeeds.

---

Update: it turns out, there is an unofficial mechanism that allows you to embed additional "entropy" into the signing process, generating unique, per-user (for example) signed URLs for the same object and expiration time.

Under V2 authentication, which doesn't nornally want you to include non-S3-specific parameters in your signing logic, it looks suspiciously like a bug as well as a feature... add &x-amz-meta-{anything-here}={unique-value-here} query string parameters to your URL. These are used as headers in PUT request but are meaningless in a GET request, and yet, if present, S3 still requires them to be included in the signature calculation, even though the parameter keys and values will ultimately be discarded by S3... but the added values are tamper-resistant and can't be maliciously removed or altered without invalidating the signature.

The same mechanism works in V4, even though it's for a different reason.

Credit for this technique: http://www.bennadel.com/blog/2488-generating-pre-signed-query-string-authentication-amazon-s3-urls-with-user-specific-data.htm

Answer 2

The accepted answer is now outdated. For future viewers, there is no need to include anything as extra header as now AWS includes a Signature field in every signed url which is different everytime you make a request.

Answer 3

Yes. In fact, i believe AWS can't even limit that, as there is no such API call on S3. URL signing is done purely by the SDK.

But if creating so many URLs is a good idea or not is completely context dependent though...