What can I do to speed up S3 uploads/updates?

Question

I have been trying to upload something small to s3 all day today. About 20k files in 500 directories that total about 3GB. Something absolutely reasonable for a service called Simple Storage Service. I can upload to different places on average at about 500k/s - 1mb/s (between 1.8 and 3.6 gb/h). I have been trying to upload these files to s3 all day, I must have uploaded at dismal rate on aggregate (think about 100 mb/h or something).

I have tried:

• the s3 web console with a variety of browsers on a variety of OS
• boto using a variety of scripts I've written, and found online (mainly here on SO).

My problems, which I was hoping you would be so kind to help me diagnose are the following:

• dragging and dropping to the s3 console (just for it to count the 20k files, takes like an hour). why? unless I can solve this the web console is mostly useless to me.
• the upload itself is extremely slow, seldomly faster than 100 k/s.
• after all day uploading I noticed a simple problem with the filenames, not wanting to spend all night uploading again, I used this script: Amazon S3 boto: How do you rename a file in a bucket? which everyone claims works really fast. It manages to rename about 1 200kb file every 2-3 seconds. why?
• after uploading, making the all the files public (using the web console) has taken like 4 hours and still it has not finished.

It is really frustrating, there must be something I am doing wrong. I expect everything to work about 10x faster and it doesn't. I've read that if split the file s3 runs faster and I've read that the zone (I'm in NYC) is really important. What change will give me the biggest increase in upload speed?

Answer 1

Maybe the slow upload connection can be fixed with a change of the AWS server location

I just figured out, what the problem was in my case: duration of upload (size 35MB)

• Oregon, US us-west-2: 5-6mins
• Frankfurth, Germany eu-central: 1mins! (that's about max of my connection)

I'm based in Vienna, not in the US -> check your AWS server location