How to handle outside insert into Meteor database?

Question

We started to use Meteor this week and are very eager to use it for as much projects as we can in the future.

Though we hit a rock this afternoon, in our ecosystem, we need to be able to process and insert data inside mongodb from outside node / meteor.

Our two main solution to do that are over MapReduce in an Hbase cluster and python scripts parsing CSV files and inserting results in MongoDB.

Right now, the only way I can use data from MongoDB inside my Meteor application is when the data was added trough Meteor (console or form).

Let's take a quick example, I want to be able to access a post through an _id. Iron Router is working fine with that and provide helpers to build the URL. With the same helpers, I could find two different results from three different usages.

The ObjectId seems to be wrong and these are the three cases I encountered earlier :

http://0.0.0.0:3000/post/PAXEqRBB7RiNrdTTT => Inserted by Meteor, works fine
http://0.0.0.0:3000/post/ObjectId(526fe0701d41c894b9105bff) => Inserted by python, broken
http://0.0.0.0:3000/post/ObjectId(526fe0701d41c894b8715bff) => Inserted with meteor mongo shell

So I can't access anything inserted inside MongoDB by something else than meteor.

I found some related issues on Github, the main one being this one :

https://github.com/meteor/meteor/issues/61

But it was closed 8 month ago because the fix was in the roadmap. I am using the last version of meteor ( 0.6.6.2 ) and this does not seems to be fixed yet.

My question is what kind of workaround can I find ? I can't call JS methods like Meteor ObjectID generation inside Python so what would be the best solution ?

Should I use a Node DDP app to handle all my external inserts to MongoDB ?

Answer 1

If you are looking to interact with the database directly, I wouldn't advocate the approach you are taking, which seems to be basically to build an API for interacting with the database through some connect middleware.

It seems to me you have two options.

1. Make changes directly into MongoDB.

This is the easiest and quickest option given that there are MongoDB bindings for almost every language. Changes you make to Mongo will be picked up immediately by Meteor's oplog observe driver if you have set up Meteor with the Mongo oplog.

2. Talk to the Meteor server using DDP.

This is probably the "correct" way to do things. It also has the advantage of allowing you to make RPC calls to the server as opposed to just doing database changes, so you can take advantage of all of the Meteor.methods you've defined already in your app.

See here for a really simple example of a Python DDP client. You'd need to take it a bit further to get full reactivity that a Javascript client would see..

Update: Meteor itself has a full-fledged DDP client now, and one way to run it is just as part of a normal Meteor server on Node. You can do anything a normal browser client would do. For example, see https://github.com/VirtualLab/cooperation-loadtest.

Answer 2

Meteor overrides the _id creation by default. The string option makes those ids easier to deal with in console interaction. I prefer it. You could make your outside inserts include an id as a string, or set your meteor env to use object ids.

http://docs.meteor.com/#meteor_collection

...

idGeneration String
The method of generating the _id fields of new documents in this collection. Possible values:

'STRING': random strings
'MONGO': random Meteor.Collection.ObjectID values