Methods for storing metadata associated with individual files?

Question

Given a collection of files which will have associated metadata, what are the recommended methods for storing this metadata?

Some files formats support storing metadata internally (EXIF,ID3,etc), but not all file formats support this, so what are more general options?

Some of the metadata would almost certainly be unique (titles/descriptions/etc), whilst some would be repetitive to varying degrees (categories/tags/etc).
It may also be useful to group the metadata, if different types of attribute are required.

Ideally, solutions should cover concepts, rather than specific language implementations.

Answer 1

To store metadata in database has some advantages but main problem with database is that metadata are not directly connected to your data. It is more robust if metada stay with data - like special file in the directory or something like that.

Some filesystems offer special functionality that can be used for metadata - like NTFS Alternate streams. Unfortunately, this can be used for metadata storage in special cases only, because those streams can be easily lost when copying data to storage system that does not support it. I believe that linux filesystems have also similar storage mechanism.

Anyway, most common solutions are :

• separate hidden file(s) (per directory) that hold metadata
• some application use special hidden directory with metadata (like subversion, cvs etc).
• or database (of various kinds) for all application specific metada - this database can be used also for caching purposes in most cases

IMO there is no general purpose solution. I would choose storage of metadata in hidden file (robustness) with use of the database for fast access and caching.