S3: how to upload large number of files

Question

I have to upload 100k small files (Total size: 200MB).

I've tried to do that via web browser (AWS Console), but for the first 15 min, i've uploaded only 2MB.

What is the fastest way to upload 100k small files to S3?

Answer 1

I have 99,561 small files totalling 466MB and experimented with uploading these to S3 as quickly as possible from my m4.16xlarge (64 CPU) EC2 instance.

• aws s3 sync took 10 minutes
• CyberDuck promised to take 17 minutes, but hung. I terminated it after 45 minutes.
• CloudBerry seems to be windows only at first glance, so I did not test it.
• s3-parallel-put --put=stupid took 9 minutes
• s3-parallel-put --put=stupid --processes=64 took 1 minute
• s3-parallel-put --put=stupid --processes=256 took 19 seconds
• s3-parallel-put --put=stupid --processes=512 took 18 seconds

I'll be moving forward with a s3-parallel-put solution.