Keeping a client-side sync of Sails.js collection, using sockets

Question

I very much like Meteor's pub/sub. I wonder if there is a way to get a similar workflow, using sails.js or just a socket library in general.

In particular, what I would like to be able to do is something along the lines of:

// Server-side:
App.publish('myCollection', -> collection.find({}))

// Client-side:
let myCollection = App.subscribe('myCollection')
let bob = myCollection.find({name: 'Bob'})
myCollection.insert({name: 'Amelie'}, callback)

All interaction with the server should happen in the background.

Answer 1

I very much like Meteor's pub/sub. I wonder if there is a way to get a similar workflow, using sails.js or just a socket library in general

Basically yes, at least about realtime sync between backend and frontend. Let's review what meteor's have and answer point by point.

Pub/sub

The Pub / Sub concept, as stated by Sabbir, is also supported by sails.js. Though the basics are slightly different :

• In meteor, the client can subscribes to everything he wants, and the server control what it receives by only publishing to who he wants;
• whereas in sails.js, the server both does subscribe some clients sockets and publish to all binded sockets

Note that, by default:

• meteor contains the autopublish package that just notify every client without any kind of filtering. To acheive some filtering, you have to meteor remove autopublish then you can handle what will your client receive by adding a mongo request to it, like explained here.
• sails by default, on its automatic "select" blueprints actions, auto-subscribes the calling socket to the events on the objects returned by the "select".

As a server-side conclusion:

• Subscribe: just call findor findOne blueprint default action, through a socket (attaching some where filters or not) and your socket will automatically be subscribed to every events concerning returned objects => you don't have to code anything on the server, in most cases, for the Subscribe logic.
• Publish: every blueprint default actions (create, update, destroy, add, remove) auto-publish to subscribed sockets => you don't have to code anything on the server, in most cases, for the Publish logic.

(Though, if you find yourself implementing some manual controller actions, sails API helps you publishing and subscribing easily)

Client handling

Therefore, with both meteor and sails, clients only receive what they're supposed to receive. Time for front-end now.

Philosophy

• meteor in one hand, with it's isomorphic dimension, does provide a front-end connector by nature, exposing it's data-bound collections.
• sails on the other hand, is front-end agnostic, and can be attacked by any http REST connector (JS or not), such as $http, $resource, or more advanced ones like Restangular.
  Though, being aware of the complexity using raw sockets on their API (when it comes to session, CORS, CSRF and stuff), they developped a javascript socket.io wrapper called sails.io.js designed to be REST-like-over-socket, and just works like a charm.

Basically, The main difference is that meteor is one step higher-level than sails, because it provides the logic of syncing collections and objects.

All interaction with the server should happen in the background.

sails.io.js, the official front-end component, is just not that high-level. When it comes to Angular.js.

Though, you can find some community connectors that aim to, kinda, provide the same feature as mongo data-bound collections and objects. There is sails-resource, spinnaker or angular resource sails. I tried both of them, and I should say that I was disapointed. The abstraction level is so high that it just becomes annoying, IMHO. For example, with not-very-RESTful-friendly custom actions, like a login, it becomes very hard to adapt it for your needs.

==> I would advice to use a low-level connector, such as angularSails or (my prefered) https://github.com/janpantel/angular-sails, or even raw sails.io.js if you're not using Angular.

Edit: just foun a backbone version, by the sails' creator

It just works great, and believe me, the "keep my collection in sync with that socket" code is so ridiculous, that finding a module for this is just not worth it.

Some code please, stop talking

In particular, what I would like to be able to do is something along the lines of:

Server

• Meteor

  # Server-side:
  App.publish('myCollection', -> collection.find({}))
  

• Sails

  //Nothing to do, just sails generate api myCollection
  

Client

• Meteor

  # Client-side:
  myCollection = App.subscribe('myCollection')
  

• Sails, with sails.io.js (Here using lodash for convenience)

  var myCollection;
  sails.io.get('/myCollection').then(
    function(res) {
      myCollection = res.data;
    },
    function(err) {
      //Handle error
    }
  );
  
  sails.io.on('myCollection').function(msg) {
    switch(msg.verb) {
      case 'created':
        myCollection.push(msg.data);
        break;
      case 'updated':
        _.extend(_.find(myCollection, 'id', msg.id), msg.data);
        break;
      case 'destroyed':
        _.remove(myCollection, 'id', msg.id);
        break;
    };
  });
  

  (I leave the find where and create to your imagination with [the doc])

All interaction with the server should happen in the background.

• Well, Sails, only for angular, with sails ressources

I'm not pretty used to that process, so I leave you reading here or here, but once again I'd choose manual .on()method.

Answer 2

Since I asked this question, I've learned a few things and some new projects have popped up. I decided against sails.io, because when developing with React.js, most of the community's weight is behind webpack, but sails.io uses gulp. I realize these can be used together and there is even an npm package for this, but I wasn't too keen on making my stack bigger than it had to be, so I went with a simple express.js server that I could tailor to my needs.

In order to sync my data, I'm using rethinkdb which allows me to asynchronously watch the database for changes and then publish the changes to the clients through websockets.

• I've set up a simple script where I keep an instance of a baobab tree on both the client and the server.

• When the tree gets modified on the server, it sends transaction data to the appropriate clients through the websocket

• The client merges the transaction with the tree.

This method does not make use of local storage and keeps the data in memory in the node.js process. The data in the transaction is also quite redundant.
The future plan has always been to set something up using redis and local storage ...

... until yesterday when I found deepstream.io!

This is a tool that does exactly what I want and need! Nothing more, nothing less.

Another project worth mention is meatier: "like meteor, but meatier". It is composed of many other well supported open source projects, so you could even pick and choose.