Tips for managing a large number of files?

Question

There are some very good questions here on SO about file management and storing within a large project.

Storing Images in DB - Yea or Nay?
Would you store binary data in database or in file system?

The first one having some great insights and in my project i've decided to go the file route and not the DB route.

A major point against using the filesystem is backup. But in our system we have a great backup scheme so i am not worried about that.

The next path is how to store the actual files. And I've thought about having the files' location static at all times and create a virtual directory system in the database side of things. So links to the file don't change.

The system i am building will have one global file management so all files are accessible to all users. But many that have gone the file route talk about physical directory size (if all the files are within one directory for example)

So my question is, what are some tips or best practice methods in creating folders for these static files, or if i shouldn't go the virtual directory route at all.

(the project is on the LAMP stack (PHP) if that helps at all)

Answer 1

One way is to assign a unique number to each file and use it to look up the actual file location. Then you an use that number to distribute files in different directories in the filesystem. For example you could use something like this scheme:

/images/{0}/{1}/{2}

{0}: file_number % 100
{1}: (file_number / 100) % 100
{2}: file_number

Answer 2

I've ran into this problem some time ago for a website that was hosting a lot of files. What we did was take a GUID (which is also the Primary Key field of a file) (e.g. BCC46E3F-2F7A-42b1-92CE-DBD6EC6D6301) and store a file like this: /B/C/C/BCC46E3F-2F7A-42b1-92CE-DBD6EC6D6301/filename.ext

This has certain advantages:

• You can scale out the file servers over multiple servers (and assign specific directories to each one)
• You don't have to rename the file
• Your directories are guaranteed to be unique

Hope this helps!