Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Amazon S3 PutObject is very slow

Tags:

c#

.net

amazon-s3

I am placing files into an S3 storage using the below code. I am finding it is exceedingly slow. The stopwatch indicated 18 seconds+. Any suggests or other experiences?

        // upload the file to S3
        AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretAccessKey);

        PutObjectRequest request = new PutObjectRequest();

        FileStream fs = new FileStream(sourceFileName, FileMode.Open);

        request.WithInputStream(fs);
        request.WithBucketName(bucketName);
        request.WithKey(keyName);
        Stopwatch stp1 = new Stopwatch();
        stp1.Start();
        client.PutObject(request);
        stp1.Stop();
        fs.Close();

This code is C#. I am using the amazon .net sdk.

The file is only 56K in size and my upload bandwidth is 1.87Mbps.

like image 239
Jeff Avatar asked Jul 24 '11 17:07

Jeff


People also ask

How can I speed up my Amazon S3?

You can increase your read or write performance by using parallelization. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second. Similarly, you can scale write operations by writing to multiple prefixes.

What is S3 latency?

AWS S3 provides a great performance. It automatically scales to high request rates, with a very low latency of 100–200 milliseconds. Your application can achieve at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per prefix in a bucket.

What is the maximum file size on S3?

Individual Amazon S3 objects can now range in size from 1 byte all the way to 5 terabytes (TB). Now customers can store extremely large files as single objects, which greatly simplifies their storage experience.


1 Answers

It sounds very similar to a problem I had recently, which was caused by the automatic proxy detection settings in "Internet Options" on Windows.

The Amazon SDK uses WebRequest to make it's HTTP requests and by default WebRequest adheres to the computers "Internet Option" settings for detecting local proxys. Luckily WebRequest has a static property WebRequest.DefaultWebProxy which, when set to null, removes the automatic proxy detection.

All you need to do is set it to null before you start using AmazonS3:

WebRequest.DefaultWebProxy = null; // here

AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretAccessKey);

[...]

It's worth noting that this static property only needs to be set once per application domain and not every time you want to create an AmazonS3 object.

Alternate approach:

If you don't mind reconfiguring the machine, is to go to:

Windows Control Panel > Internet Options > Connections > Lan Settings

and un-check "Automatically detect settings". If you use this approach, you do not need to set the DefaultWebProxy property at all.

Further info:

When I encountered the issue I asked the following question on SO:

How to turn off the automatic proxy detection in the `AmazonS3` object?

It has more details than my answer here if your interested.

like image 63
InvertedAcceleration Avatar answered Sep 25 '22 22:09

InvertedAcceleration