Dynamic Models in Phoenix Framework

Question

Is there any way to dynamically create and use models in Phoenix? I have an application that stores metadata about clients' tables: they set a handful of fields (column names and types) and then send me CSV files to parse and store. From the stored metadata, I'd like to generate a model so that I can use Ecto to manage the client table and perform queries against it.

I'm coming from a Django background where I can use the built in ORM and type() function to build models on the fly and then use them without having to generate migrations or other model code in the application.

In Python, I would (roughly) do:

class MetaModel(models.Model):
    table_name = models.CharField(...)
    model_name = models.CharField(...)
    field_defs = JSONField(...)

    def resolve_fields(self):
        # takes values from `field_defs` and converts them into
        # django field instances

     def get_model(self):
         class Meta:
             app_label = 'dynamic'
             db_table = self.table_name
         fields = self.resolve_fields()
         attrs = {'__module__': 'dynamic', 'Meta': Meta}
         attrs.update(fields)
         model = type(self.model_name, (models.Model,), attrs)
         return model

     def create_table(self):
         with connection.schema_editor() as se:
             model = self.get_model()
             se.create_model(model)

With that, I'm able to create the table in the database and then leverage the ORM to work with client supplied data.

I know I can do it with raw SQL and just use Ecto to run the commands and queries, but I'd like to make it more codified and rely on the internals of Ecto rather than writing and maintaining a bunch of SQL templates.

Any advice (even a "nope, you can't do that") is super helpful. Thanks!

Answer 1

Yes, it's possible with Module.create/3. There are some caveats: you need to choose a unique name for each module and the module's code will live in memory until the VM is restarted so you might want to restrict the number of times you call this function.

Here's a basic implementation you can build off of. It allows you to pass a module name, a table name, and a list of field name and type pairs.

defmodule A do
  def go(module, table, fields) do
    Module.create(module, quote do
      use MyApp.Web, :model
      schema unquote(table) do
        unquote(for {name, type} <- fields do
          quote do
            field unquote(name), unquote(type)
          end
        end)
      end
    end, Macro.Env.location(__ENV__))
  end
end

A.go MyPost, "posts", [
  {:title, :string},
  {:content, :string},
]

# MyPost is now just like any other Schema module in Phoenix.

IO.inspect MyApp.Repo.all(MyPost)

Output:

[debug] QUERY OK source="posts" db=2.8ms queue=0.1ms
SELECT p0."id", p0."title", p0."content" FROM "posts" AS p0 []
[%MyPost{__meta__: #Ecto.Schema.Metadata<:loaded, "posts">,
  content: "Hello from Joe", id: 1, title: "Hello"},
 %MyPost{__meta__: #Ecto.Schema.Metadata<:loaded, "posts">,
  content: "Hello from Mike", id: 2, title: "Hello"},
 %MyPost{__meta__: #Ecto.Schema.Metadata<:loaded, "posts">,
  content: "Hello from Robert", id: 3, title: "Hello"}]