Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Setting metadata on S3 multipart upload

I'd like to upload a file to S3 in parts, and set some metadata on the file. I'm using boto to interact with S3. I'm able to set metadata with single-operation uploads like so:

Is there a way to set metadata with a multipart upload? I've tried this method of copying the key to change the metadata, but it fails with the error: InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: <size>

I've also tried doing the following:

key = bucket.create_key(key_name)
key.set_metadata('some-key', 'value')
<multipart upload>

...but the multipart upload overwrites the metadata.

I'm using code similar to this to do the multipart upload.

like image 817
user3776364 Avatar asked Dec 05 '14 21:12

user3776364


People also ask

Does S3 support multipart upload?

Multipart upload initiationWhen you send a request to initiate a multipart upload, Amazon S3 returns a response with an upload ID, which is a unique identifier for your multipart upload. You must include this upload ID whenever you upload parts, list the parts, complete an upload, or stop an upload.

How do I upload a multipart file to S3 bucket?

For instructions on creating and testing a working sample, see Testing the Amazon S3 Java Code Examples. To upload a file to an S3 bucket, use the TransferUtility class. When uploading data from a file, you must provide the object's key name. If you don't, the API uses the file name for the key name.

How does aws multipart S3 upload work?

Multipart Upload allows you to upload a single object as a set of parts. After all parts of your object are uploaded, Amazon S3 then presents the data as a single object. With this feature you can create parallel uploads, pause and resume an object upload, and begin uploads before you know the total object size.

What is metadata in S3 bucket?

Object metadata is a set of name-value pairs. After you upload the object, you cannot modify object metadata. The only way to modify object metadata is to make a copy of the object and set the metadata. When you create an object, you also specify the key name, which uniquely identifies the object in the bucket.


1 Answers

Sorry, I just found the answer:

Per the docs:

If you want to provide any metadata describing the object being uploaded, you must provide it in the request to initiate multipart upload.

So in boto, the metadata can be set in the initiate_multipart_upload call. Docs here.

like image 124
user3776364 Avatar answered Sep 28 '22 12:09

user3776364