Reverse JSON query: find all queries in a collection matching an object

Question

I am designing a generic notification subscription system where user can specify a compound rule at the time of subscription in terms of MongoDB query, or more generally, json query. The subscription data is stored in MongoDB collection. For example,

{ "userId": 1, "rule": {"p1": "a"} }
{ "userId": 2, "rule": {"p1": "a", "p2": "b"} }
{ "userId": 3, "rule": {"p3": {$gt: 3} } }

Later when an event in the form of a json object, such as the following, arrives, I want to find all user rules the event maches:

{"p1": "a", "p3": 4}

The above event should match rules specified by userId 1 and 3 in the example. The event object doesn't have to be stored in MongoDB.

Although I can probably meet the requirement by writing a loop at application layer. For efficiency I really want to implement it at db layer, preferably allow distributed (sharded) execution due to volume and latency requirement.

Is it achievable? Any help is appreciated. In fact, I am open to other NOSQL dbs as long as supporting dynamic event schema and there is a way to specify compound rule.

Answer 1

What you are trying to achieve is not possible, at least in MongoDB.

If you reason about how a query engine works, you will realize that this has not a straightforward solution.

On high-level terms, the engine will generate a condition object from your query that then will get evaluated against each document in the set that will result in a boolean value which determines if the document belongs to the result set or not. In your case you want to do the other way round, you want to generate a condition object based on the document and then apply it to something (e.g an object) that you give it.

Even if it were possible, the cost of doing this on the DB would be too high as it would require to compile an expression function for each object and execute it and there would be no way to optimize the execution of the query. It is more reasonable to actually do that outside the database, where you could have the expression functions already created.

Answer 2

You cant store "Comparison Query Operators" in a mongo database, but you can do this:

{ "userId": 1, "rule": {"p1": "a"} }
{ "userId": 2, "rule": {"p1": "a", "p2": "b"} }
{ "userId": 3, "rule": {"p3":  {"value": 3, "operator":"gt"} } }

You store value AND OPERATOR, in string form, and you can make a query like this:

db.test.find({"rule.p3.comparator":"gt", "rule.p3.value":{$lt:4}})

Notice, if your "operator" is "gt", you must use $lt (the opposite comparison operator) in the query

Your complete example is something like this:

db.test.find({$or:[{"rule.p3.comparator":"gt", "rule.p3.value":{$lt:4}}, {"rule.p1":"a"}]})

This query match userId 1 and 3 like you want

Answer 3

Update: Following solution doesn't work. Problem with mongodb is that it doesn't use NodeJs to run map-reduce javascript, nor support any package manager. So it's hard to use any 3rd party libraries.

My own proposed solution, which hasn't been confirmed :

1. Compose query conforming to json-query syntax

2. upon arrival of evt, call MongoDB mapReduce function on user rules collection to invoke jsonQuery in mapper

    var jsonQuery = require('json-query')
    var mapper = function(evt) {
        function map() {
            if(jsonQuery(this.rule, {data: evt}){
                emit(this.userId);
            }
        }
        return map;
    };
    db.userRules.mapReduce(mapper(evt), ...);
   

The reason to compose query into json-query syntax instead of MongoDB query syntax is only json-query offers the jsonQuery method that tries to match one rule to one object. For above code to meet the requirements in question, following assumptions have to be met:

1. MongoDB can execute mapReduce on distributed nodes
2. In mapReduce I can use external library such as json-query, which implies the library code has to be distributed to all MongoDB nodes, perhaps as a result of closure.