Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Scaling images stored at S3

I'm in a situation where I need to push image storage for a number of websites out to a service that can scale indefinitely (S3, CloudFiles, etc.). Up until this point we've been able to allow our users to generate custom thumbnail sizes on the fly using Python's Imaging library with some help from sorl-thumbnail in Django.

By moving our images to something like S3, the ability to quickly create thumbnails on the fly is lost. We can either:

  1. Do it slowly by downloading the source from S3 and creating the thumbnail locally
    con: it is slow and bandwidth intensive
  2. Do it upfront by creating a pre-determined set of thumbnail sizes (a'la Flickr) and pushing them all to S3
    con: it limits the sizes that can be generated and stores lots of files that will never be used
  3. Let the browser resize using the height/width attributes on the img tag.
    con: extra bandwidth used by downloading larger than necessary files

At this point #3 looks to be a simple solution to the problem with few drawbacks. Some quick tests and data from this website show that the quality isn't as bad as expected (we could assure the aspect ratio is maintained).

Any suggestions on other options or drawbacks we might not be taking into consideration?

note: The images are digital photos and are only used for display on the web. Sizes would range from 1000-50 pixels in height/width.

like image 652
Pete Avatar asked Oct 13 '09 22:10

Pete


People also ask

Does S3 optimize images?

The S3 Image Optimizer will optimize all your images in 1-1,000+ Amazon S3 buckets using the EWWW Image Optimizer. Since EWWW IO integrates directly with plugins like WP Offload Media, S3 IO is generally for folks who use a solution other than WP Offload Media to put their images on S3.

Does S3 storage scale automatically?

Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned prefix. There are no limits to the number of prefixes in a bucket.

Is Amazon S3 scalable?

Since Amazon S3 is highly scalable and you only pay for what you use, you can start small and grow your application as you wish, with no compromise on performance or reliability. Amazon S3 is also designed to be highly flexible.

How is data stored in Amazon S3 for high durability?

The S3 One Zone-IA storage class stores data redundantly across multiple devices within a single Availability Zone. These services are designed to handle concurrent device failures by quickly detecting and repairing any lost redundancy, and they also regularly verify the integrity of your data using checksums.


1 Answers

I would recommend using EC2 to scale the images on demand. Since bandwidth between EC2 and S3 is free and it should be fast I think that eliminates all the problems with solution #1.

like image 83
Amok Avatar answered Oct 23 '22 03:10

Amok