Recommended architecture for handling user image uploads

Question

In the past, I've handled user image uploads in two different ways:

• Save the image data in a database table, and load it via a PHP script
• Upload the image, convert it to jpeg, put it in a directory and load it via HTML tags

The first option worked fairly well, but I had to keep fairly constraining size restrictions on uploaded images. The second option kind of worked, except for PHP's image library would often botch the file conversion (I think that can be fixed by using ImageMagick instead, however).

I'm now faced with doing this for a third time, on a potentially larger scale of users. I already plan on using ImageMagick to do some post processing to the image after the upload. I'd love to keep the restrictions on image uploads small as possible and possibly even retain every photo that someone uploads.

There will be times when hundreds of user's photo uploads will be displayed on the screen as a thumbnail at once. Storing these in a database and pulling them for each one and displaying via PHP doesn't seem like a good way to go, but throwing all images in to a single directory doesn't either.

What is your recommendation in this situation? Do you go with one of the options above or do you have a different solution?

Answer 1

Storing these in a database and pulling them for each one and displaying via PHP doesn't seem like a good way to go, but throwing all images in to a single directory doesn't either.

You can take a hybrid approach.

Store the images in a heirarchy of folders (according to whatever scheme you determine to be appropriate for your application). Store the full path to each image in your database.

In a background job, have thumbnails of the images produced (say with ImageMagick) whose filenames slightly differ from the images themselves (eg: add 'thumb-' on the front) but which are stored alongside the real images. You can have a field in the database for each image which means "My thumbnail is ready, so please include me in galleries".

When you get a request for a gallery, slice and dice the group of images using database fields, then produce a piece of HTML which refers to the appropriate thumbnail and image paths.

---

Edit: What Aaron F says is important when you need to handle a very large number of requests. Partitioning of the image / sql data is a good road to scalability. You'll need to look at access patterns in your application to determine where the partition points lie. Something you can do even sooner is to cache the generated HTML for the galleries in order to reduce SQL load.

Answer 2

The two examples you cite don't clarify the most important part: partitioning.

e.g. if stored in a database, do the images reside entirely in one table, on one database server? Or is the table partitoned across several servers/clusters?

e.g. if stored in a directory, do all the images reside on one hard drive? Or do images map to separate [RAID] drives, based on the first letter in the user's login name?

A good partitioning scheme is needed for scalability.

As far as displaying thumbnails in bulk, you'll probably want some precomputation here. i.e. create the thumbnails (perhaps via an ayschronous job, kicked off just after the image is uploaded) and stage them on a dedicated server. This is how Youtube does image snapshots of uploaded videos.