Building a DSL query language

Question

i'm working on a project (written in Django) which has only a few entities, but many rows for each entity.

In my application i have several static "reports", directly written in plain SQL. The users can also search the database via a generic filter form. Since the target audience is really tech-savvy and at some point the filter doesn't fit their needs, i think about creating a query language for my database like YQL or Jira's advanced search.

I found http://sourceforge.net/projects/littletable/ and http://www.quicksort.co.uk/DeeDoc.html, but it seems that they only operate on in-memory objects. Since the database can be too large for holding it in-memory, i would prefer that the query is translated in SQL (or better a Django query) before doing the actual work.

Are there any library or best practices on how to do this?

Answer 1

Writing such a DSL is actually surprisingly easy with PLY, and what ho—there's already an example available for doing just what you want, in Django. You see, Django has this fancy thing called a Q object which make the Django querying side of things fairly easy.

At DjangoCon EU 2012, Matthieu Amiguet gave a session entitled Implementing Domain-specific Languages in Django Applications in which he went through the process, right down to implementing such a DSL as you desire. His slides, which include all you need, are available on his website. The final code (linked to from the last slide, anyway) is available at http://www.matthieuamiguet.ch/media/misc/djangocon2012/resources/compiler.html.

Reinout van Rees also produced some good comments on that session. (He normally does!) These cover a little of the missing context.

You see in there something very similar to YQL and JQL in the examples given:

• groups__name="XXX" AND NOT groups__name="YYY"
• (modified > 1/4/2011 OR NOT state__name="OK") AND groups__name="XXX"

It can also be tweaked very easily; for example, you might want to use groups.name rather than groups__name (I would). This modification could be made fairly trivially (allow . in the FIELD token, by modifying t_FIELD, and then replacing . with __ before constructing the Q object in p_expression_ID).

So, that satisfies simple querying; it also gives you a good starting point should you wish to make a more complex DSL.