Adding mongoDB document-array-element using Python Eve

Question

Background: (using Eve and Mongo)

I'm working in Python using the Eve REST provider library connecting and to a mongoDB to expose a number of REST endpoints from the database. I've had good luck using Eve so far, but I've run into a problem that might be a bit beyond what Eve can do natively.

My problem is that my mongoDb document format has a field (called "slots"), whose value is a list/array of dictionaries/embedded-documents.

So the mongoDB document structure is:

{
   blah1: data1,
   blah2: data2,
   ...
   slots: [
       {thing1:data1, thing2:data2},
       {thingX:dataX, thingY:dataY}
   ]
}

I need to add new records (I.E. add pre-populated dictionaries) to the 'slots' list.

If I imagine doing the insert directly via pymongo it would look like:

mongo.connection = MongoClient()
mongo.db = mongo.connection['myDB']
mongo.coll = mongo.db['myCollection']

...

mongo.coll.update({'_id' : document_id}, 
                  {'$push': { "slot" : {"thing1":"data1","thingX":"dataX"}  }  } )

The REST action/URI combo that I would like to do this action is a POST to '_id/slots', e.g. URI of /app/012345678901234567890123/slots.

Problem: (inserting an element into an array in Eve)

From SO: How to add to a list type in Python Eve without replacing old values and eve project issue it appears Eve doesn't currently support operating on mongoDB embedded documents (or arrays?) unless the entire embedded document is rewritten, and rewriting the whole array is very undesirable in my case.

---

So, assuming its true Eve doesn't have a method to allow inserting of array elements (and given I already have numerous other endpoints working well inside of Eve)...

---

... I'm now looking for a way, inside of an Eve/Flask configuration with multiple working endpoints, to intercept and change Eve's mongoDB write for just this one endpoint.

I know (worst case) I can override the routing of Eve and to completely do the write by hand, but then I would have manage the _updated and hand check & change the documents _etag value, both things I would certainly prefer not to have to write new code for.

I've looked at Eve's Datebase event hooks but I don't see a way to modify the database commands that are executed (I can see how to change the data, but not the commands).

Anyone else already solved this problem already? If not any ideas on the most direct path to implement by hand? (hopefully reusing as much of Eve as possible because I do want to continue using Eve for all my (already working) endpoints)

Answer 1

This is an interesting question. I believe that in order to achieve your goal you would need to perform two actions:

1. Build and pass a custom Validator.
2. Build and pass a custom Mongo data layer.

This might sound like too much work, but that's probably not the case.

---

Custom Validator

A custom validator is going to be needed because when you perform your PATCH request on the "push-enabled" endpoint you want to pass a document which is syntactically different from endpoint validation schema. You are going to pass a dict ({"slot": {"thing1": "data1", "thingX": "dataX"}}) whereas the endpoint expects a list:

'mycollection': {
    'type': 'list',
    'schema': {
        'type': 'dict',
        'schema': {
            'thing1': {'type': 'string'},
            'thingX': {'type': 'string'},
        }
    }
}

If you don't customize validation you will end up with a validation error (list type expected). I guess your custom validator could look something like:

from eve.data.mongo.validation import Validator
from flask import request

class MyValidator(Validator):
    def validate_replace(self, document, _id, original_document=None):
        if self.resource = 'mycollection' and request.method = 'PATCH':
            # you want to perform some real validation here
            return True
        return super(Validator, self).validate(document)

Mind you I did not try this code so it might need some adjustment here and there.

An alternative approach would be to set up an alternative endpoint just for PATCH requests. This endpoint would consume the same datasource and have a dict-like schema. This would avoid the need for a custom validator and also, you would still have normal atomic field updates ($set) on the standard endpoint. Actually I think I like this approach better, as you don't lose functionality and reduce complexity. For guidance on multiple endpoints hitting the same datasource see the docs

---

Custom data layer

This is needed because you want to perform a $push instead of a $set when mycollection is involved in a PATCH request. Something like this maybe:

from eve.io.mongo import Mongo
from flask import request

class MyMongo(Mongo):
    def update(self, resource, id_, updates, original):
        op = '$push' if resource == 'mycollection' else '$set'
        return self._change_request(resource, id_, {op: updates}, original)

---

Putting it all together

You then use your custom validator and data layers upon app initialisation:

app = Eve(validator=MyValidator, data=MyMongo)
app.run()

Again I did not test all of this; it's Sunday and I'm on the beach so it might need some work but it should work.

With all this being said, I am actually going to experiment with adding support for push updates to the standard Mongo data layer. A new pair of global/endpoint settings, like MONGO_UPDATE_OPERATOR/mongo_update_operator are implemented on a private branch. The former defaults to $set so all API endpoints still perform atomic field updates. One could decide that a certain endpoint should perform something else, say a $push. Implementing validation in a clean and elegant way is a little tricky but, assuming I find the time to work on it, it is not unlikely that this could make it to Eve 0.6 or beyond.

Hope this helps.