Dynamic creation of models and/or tables in Rails

Question

I'm working on a project at the moment that has a rather unusual requirement and I'm hoping to get some advice on the best way to handle it or even some pointers to info that can help me build a solution.

Ok, so this is what I need to do. The application stores and manages various types of media files but each deployment of the application has completely different metadata requirements for the media files.

This metadata can contain an arbitrary number of fields of different types (single line text, multi-line text, checkboxes, selected values, etc.) and also often requires validation particularly presence and uniqueness validations.

The application needs to be able to easily retrieve values and most importantly has to be able to handle full searching capabilities on these fields.

One option I considered was using a property list arrangement where the database table simply contained a property name and value for each metadata field of each media file. However, when prototyping this solution it quickly became apparent that it simply wasn't going to be efficient enough for the searching and retrieval of records particularly when the database can be reasonably large e.g. a recent deployment had 3000 media files and there were over 20 metadata fields. Also, the queries to do a search and retrieve the relevant records quickly became very complex.

Another option that the system is currently using is that the metadata config is defined upfront and a migration is run during deployment to create a the table and model with a standard name so that the media model can be associated with it which the system then uses. This generally works pretty fine but it does cause some significant deployment and testing issues.

For example, writing unit tests becomes much more challenging when you don't know the config until deployment. Although I could write a sample config and test the code that way, it won't allow me to test the specific requirements of a particular deployment.

Similarly, in development, it currently requires me to copy a migration from the config into the main folder, run it, do all of my testing and development and then I have to remember to rollback and remove that migration from the main folder so that the application is in a standard state. This particularly becomes challenging when I'm bug fixing and I need to have the application in a specific configuration for testing and debugging purposes. Trying to switch between the various configurations becomes a real nightmare.

Ideally, what I would like is to be able to dynamically create the table and model including validations, etc. from a config file when the server is started. Even better would be if I could maintain multiple metadata setups in the one database with each one having its own table so that all I need to do to switch between them is change which config file the application is currently using.

I'm sure this can be done with Rails but there is very little information that I've been able to find that can point me in the right direction of how to build it during my research over the past few days so any help or suggestions would be much appreciated!

Answer 1

If I understand you correctly, Rails has some nifty tricks to help you solve these problems.

In the ActiveRecord ORM it's possible to model what you're trying to do in a relational database, either using the single table inheritance pattern, or with polymorphic associations (...a bit more complex but more flexible too). A polymorphic association allows a model to belong_to different types of other models. There's a recent railscast on this topic but I won't link to it since it requires a paid subscription.

On the deployment side, it sounds like you're doing a lot of things manually, which is the right way to start until a pattern emerges. Once you start seeing the pattern, there are excellent programs available for configuration, build, and deployment automation such as Capistrano, OpsCode Chef, and Puppet, to name just a few. You might also benefit from integrating your configuration and deployment with your source code repository to achieve a better workflow. For example, with Git you could define topic branches for the various media file type and have a different configuration in each branch that matches the topic branch.

You may want to check out Martin Fowler's excellent book 'PoEAA' and some of the topics on his website. I hope this answer helps even though the answer is pretty generic. Your question is very broad and does not have one simple answer.

Answer 2

each deployment of the application has completely different metadata requirements for the media files.

I recommend using mongoDB for your database and Mongoid for your ORM. This will give you the flexibility you need to change the schema as needed without horrific schema manipulations, dynamic models/tables, and all that horror.

The application needs to be able to easily retrieve values and most importantly has to be able to handle full searching capabilities on these fields.

This is a search problem rather than a database problem. I recommend trying out the full-text search capabilities in the latest version of mongoDB. If that doesn't meet your needs, try elasticsearch in conjuction with the Tire gem (an elasticsearch client that integrates nicely with Rails).