Serving Images with on-the-fly resize

Question

my company has recently started to get problems with the image handling for our websites.

We have several websites (adult entertainment) that display images like dvd covers, snapshots and similar. We have about 100'000 movies and for each movie we have an average of 30 snapshots + covers. Almost every image has an additional version with blurring and overlay for non-members, this results in about 50 images per movie or a total of 5 million base images. Each of the images is available in several versions, depending on where it's placed on the page (thumbnail, original, small preview, not-so-small preview, small image in the top-list, etc.) which results in more images than i cared to count.

Now i had the idea to use a server to generate the images on-the-fly since it became quite clumsy to generate all the different images for all the different pages (as different pages sometimes even need different image sizes for basically the same task).

Does anyone know of an image processing server that can scale down images on-the-fly so we only need to provide the original images and the web guys can just request whatever size they need?

Requirements:

• Very High performance (Several thousand users per day)
• On-the-fly blurring and overlay creation
• On-the-fly resize (with and without keeping aspect ratio)
• Can handle millions of images
• Must be able to read JPG, GIF, PNG and BMP and convert between them

Security is not that much of a concern as i.e. the unblurred images can already be reached by URL manipulation and more security would be nice but it's not required and frankly i stopped caring (After failing to get into my coworkers heads why (for our small reseller page) it's a bad idea to use http://example.com/view_image.php?filename=/data/images/01020304.jpg to display the images).

We tried PHP scripts to do this but the performance was too slow for this many users.

Thanks in advance for any suggestions you have.

Answer 1

I suggest you set up a dedicated web server to handle image resize and serve the final result. I have done something similar, although on a much smaller scale. It basically eliminates the process of checking for the cache.

It works like this:

• you request the image appending the required size to the filename like http://imageserver/someimage.150x120.jpg
• if the image exists, it will be returned with no other processing (this is the main point, the cache check is implicit)
• if the image does not exist, handle the 404 not found via .htaccess and reroute the request to the script that generates the image of the required size
• in the script specify the list of allowed sizes to avoid attacks like scripts requesting every possible size to shut your server down
• keep this on a cookieless domain to minimize unnecessary traffic

EDIT: I don't think that PHP itself would slow the process much, as PHP scripting in this case is reduced to a minimum: the image scaling is done by a builtin library written in C. Whatever you do you'll have to use a library like this (GD or libmagick or so) so that's unavoidable. With my system at least you totally skip the overhead of checking the cache, thus further reducing PHP interaction. You can implement this on your existing server, so I guess it's a solution well suited for your budget.

Answer 2

Based on

We tried PHP scripts to do this but the performance was too slow for this many users.

I'm going to assume you weren't caching the results. I'd recommend caching the resulting images for a day or two (i.e. have your script check to see if the thumbnail has already been generated, if so use it, if it hasn't generate it on the fly).

This would improve performance dramatically as I'd imagine the main/start page probably has a lot more hits than random video X, thus when viewing the main page no images have to be created as they're cached. When User Y views Movie X, they won't notice the delay as much since it just has to generate that one page.

For the "On-the-fly resize" aspect - how important is bandwidth to you? I'd want to assume you're going through so much with movies that a few extra kb in images per request wouldn't do too much harm. If that's the case, you could just use larger images and set the width and height and let the browser do the scaling for you.