Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to upload multiple files and combine them in S3

I have a large size exe file (> 100 MB) which I need to upload to S3. Now since I have a reliable network, I want to make this process faster by dividing the file into chunks and then uploading these chunks in parallel. But I don't know how to merge these chunks back together to get the original exe?

Although python provides a way to upload a file into chunks but those chunks are uploaded serially. Is there any way to upload these chunks in parallel and then merge them back together?

like image 717
niks_4143 Avatar asked Sep 26 '22 13:09

niks_4143


1 Answers

S3 already has support for this.

It's called a multipart upload: http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html

Use a client (for example s3cmd http://s3tools.org/kb/item13.htm) that knows how to do this or use do it yourself through the SDK and your favorite language.

like image 64
Mircea Avatar answered Sep 30 '22 07:09

Mircea