AWS S3 Client for linux with multipart upload [closed]

Question

What amazon s3 client do you use in linux with multipart upload feature? I have 6GB of zip files to upload and s3curl is not possible due to maximum limit of 5GB only.

Thanks. James

Answer 1

I use S3 Tools, it will automatically use the multipart upload feature for files larger than 15MB for all PUT commands:

Multipart is enabled by default and kicks in for files bigger than 15MB. You can set this treshold as low as 5MB (Amazon’s limit) with —multipart-chunk-size-mb=5 or to any other value between 5 and 5120 MB

Once installed and configured, just issue the following command:

~$ s3cmd put largefile.zip s3://bucketname/largefile.zip

Alternatively, you could just use split from the command-line on your zip file:

split -b1024m largefile.zip largefile.zip-

and recombine later on your filesystem using:

cat largefile.zip-* > largefile.zip

If you choose the second option, you may want to store MD5 hashes of the files prior to upload so you can verify the integrity of the archive when it's recombined later.

Answer 2

The official AWS Command Line Interface supports multi-part upload. (It uses the boto successor botocore under the hood):

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.

On top of this unified approach to all AWS APIs, it also adds a new set of simple file commands for efficient file transfers to and from Amazon S3, with characteristics similar to the well known Unix commands, e.g.:

• ls - List S3 objects and common prefixes under a prefix or all S3 buckets.
• cp - Copies a local file or S3 object to another location locally or in S3.
• sync - Syncs directories and S3 prefixes.
• ...

So cp would be sufficient for the use case at hand, but be sure to check out sync as well, it is particularly powerful for many frequently encountered scenarios (and sort of implies cp depending on the arguments).

Answer 3

The boto library includes an s3 command line tool called s3put that can handle multipart upload of large files.

Answer 4

You can have a look at the FTP/Amazon S3/Glacier client CrossFTP.

Answer 5

Personally I created python file s3upload.py with simple function to upload large files using boto and multipart upload.

Now every time I need to upload large file, I just run command like this:

python s3upload.py bucketname extremely_large_file.txt

More details and function code can be found here.