Need a file system metadata layer for applications

Question

I'm looking for a metadata layer that sits on top of files which can interpret key-value pairs of information in file names for apps that work with thousands of files. More info:

• These aren't necessarily media files that have built-in metadata - hence the key-value pairs.
• The metadata goes beyond os information (file sizes, etc) - to whatever the app puts into the key-values.
• It should be accessible via command line as well as a python module so that my applications can talk to it.
• ADDED: It should also be supported by common os commands (cp, mv, tar, etc) so that it doesn't get lost if a file is copied or moved.

Examples of functionality I'd like include:

• list files in directory x for organization_id 3375
• report on files in directory y by translating load_time to year/month and show file count & size for each year/month combo
• get oldest file in directory z based upon key of loadtime

Files with this simple metadata embedded within them might look like:

• bowling_state-ky_league-15_game-8_gametime-201209141830.tgz
• bowling_state-ky_league-15_game-9_gametime-201209141930.tgz

This metadata is very accessible & tightly joined to the file. But - I'd prefer to avoid needing to use cut or wild-cards for all operations.

I've looked around and can only find media & os metadata solutions and don't want to build something if it already exists.

Answer 1

Have you looked at extended file attributes? See: http://en.wikipedia.org/wiki/Extended_file_attributes

Basically, you store the key-value pairs as zero terminated strings in the filesystem itself. You can set these attributes from the command line like this:

$ setfattr -n user.comment -v "this is a comment" testfile
$ getfattr testfile
# file: testfile
user.comment
$ getfattr -n user.comment testfile
# file: testfile
user.comment="this is a comment"

To set and query extended file system attributes from python, you can try the python module xattr. See: http://pypi.python.org/pypi/xattr

EDIT Extended attributes are supported by most filesystem manipulation commands, such as cp, mv and tar by adding command line flags. E.g. cp -a or tar --xattr. You may need to make these commands to work transparently. (You may have users who are unaware of your extended attributes.) In this case you can create an alias, e.g. alias cp="cp -a".