What is the most efficient way to store 500.000 images?

Question

I'm coding a basic gallery for a website with around 40.000 online people at any given time. Users will be able to create galleries and upload images.

My question is, should I make a seperate folder for each gallery and put the images in them, or make a single folder and put all images in it, but keep the gallery_id for each image in the database? Or, should I make a directory for every user, then another directory inside them for the gallery names?

How would you do this?

Ps. I need it to be as light as it can.

Answer 1

I would store them by id and i would split them into folders (dependant of filesystem, some don't perform well with lots of files in 1 folder), plus it is easier to find them if you have to manually look at something

Give each file an id, then using the first 3 digits of the file name, split them into folders. (you could start your auto-increment counter at 100000 or zero pad the id, so there is at least 3 levels

/photos/1/0/3/103456.jpg
/photos/9/4/1/941000.jpg
/photos/0/0/0/000001.jpg

You can store the relationship of photo to user / gallery / etc in the database

Or if you want to see how the big boys do it

Needle in a haystack: efficient storage of billions of photos

Answer 2

Typically web servers don't want you to have more than a few thousand images in a single folder (I recently had to deal with 70,000 images causing super slow reads and sorts so trust me on this) so certainly not a single folder if you think you will have thousands of images. I would suggest the best solution would be to host off of amazon's S3 connected to their CDN CloudFront but if that isn't realistic you can still do several things just on your own server.

Make a separate folder for each gallery like you suggest only if you know some bounds on how large a gallery can get and have an idea of how many galleries will be created. (This is what I would suggest for your specific problem right now)

Put the image name through a hash function then use the first 1-3 characters of the hash to name folders to put the images into. The hash ensures that the images are roughly equally split among the folders and you can decide how many folders you need.

At any rate having the information of what gallery and the image id in the actual path will probably be useful to you moving forward both in code and whenever a human has to hunt bugs on the server. I would probably name the folders based on the gallery id and just make sure that no gallery has more than a few thousand images in it.