meteor observe array server side

Question

I have a recursive function that builds asynchronously a tree on the server side and I would like to 'observe' it and have the calling method in Meteor rerun every time there is a change.

I have made a simplified example that builds a tree with a recursive readdir call (in the real application there is a computation that may take several minutes per node and its results depend on the nodes already explored)

in server/methods.js

var fs = Meteor.npmRequire('fs')
var path = Meteor.npmRequire('path')

var tree = function (dir, r) {
  try
  {
    fs.readdir (dir, function (error, files) {
      if (files && files.length)
        for (var i = 0; i < files.length; i++)
        {
          r[i] = { name : files[i], children : [] }
          tree(path.resolve(dir, files[i]), r[i].children)
        }
    })
  } catch (e) { console.log("exception", e)}
}

Meteor.methods({
  'build_tree' : function () {
    var r = []
    tree("/tmp/", r)
    return r // Wrong !
  }
})

in client/client.js

Meteor.call('build_tree', function (error, result) {
   console.log(error, result)
}

I have already used futures in other parts of the code based on https://www.discovermeteor.com/patterns/5828399.

But in this case I am somehow lost due to

• the recursive nature of the server-side code
• the fact I want the client-side to update automatically every time the server-side data structure is updated

The only workaround that comes to my mind is to insert progressively the asynchronous results in a 'flat' Mongo collection and reactively rebuild it as a tree on the client side.

Answer 1

I managed to do this by

• counting the number of times an asynchronous computation was started or finished
• resolving the future only when those numbers are equal
• relaunching the function every time an asynchronous computation ends (in case it returned to launch more asynchronous computations or resolve the future)

[line to close list markup or code doesn't format properly]

Future = Meteor.npmRequire('fibers/future')
FS = Meteor.npmRequire('fs')
Path = Meteor.npmRequire('path')

const all_files = []

const future = new Future()
const to_process = [dir]
let started = 0
let ended = 0

const tree = function () {
  while (to_process.length) {
    let dir = to_process.pop()
    started++
    FS.readdir (dir, function (error, files) {
      if (error) {
        if (error.code == 'ENOTDIR') all_files.push(dir)
      }
      else if (files && files.length)
      {
        for (let i = 0, leni = files.length; i < leni; i++)
        {
          let f = Path.resolve(dir, files[i])
          to_process.push(f)
        }
      }
      ended++
      tree()
    })
  }
  if (!to_process.length && started == ended)
    future['return']()
}

tree()
future.wait()

It doesn't have the "progressive update" feeling that you get by updating the database and letting the reactivity manage it since all computations are waiting for that final Future['return']() but the code is simpler and self-contained.

Answer 2

That would be indeed very complicated. First of all, as your tree code runs async you need to either provide a success callback/resolve a promise/return a future or something else, so that you can control when the Meteor method returns. Then you need to use Futures to defer the return of the method util you have your result.

But even then I don't see how the server is supposed to know that something has changed.

The only workaround that comes to my mind is to insert progressively the asynchronous results in a 'flat' Mongo collection and reactively rebuild it as a tree on the client side.

This is actually a workable straightforward solution.