How to prevent Meteor from watching files?

Question

I would like to use Dojo Toolkit with Meteor.

1. I first copy the whole Dojo Toolkit tree in /public

2. Then, I include it on the client side with:

   <script src="/dojo/dojo.js" data-dojo-config="async: true"></script>`
   

Everything works fine, except Meteor is actually monitoring every single file in /public for changes, so that it can restart the server. This is actually causing a very long delay during the first request on localhost:3000.

Is there a way of preventing Meteor from watching files from a certain directory?

---

Dojo Toolkit is 10k+ files so I get the EMFILE error stated here, corrected with

sudo sh -c 'echo 16384 > /proc/sys/fs/inotify/max_user_watches'

Answer 1

realised this is duplicate to: generating and serving static files with Meteor

see: https://github.com/meteor/meteor/issues/437

This was a major problem for me. I have to serve ~12000 static images, which I initially put into the public folder. This caused node to use nearly 100% of one CPU core, constantly. With limited memory the app crashes.

The workaround I'm using for the moment

• create the folder public/.#static/ and move all static assets into it. This folder isn't watched by meteor
• prefix urls with static (/img/cat.png -> /static/img/cat.png)

• install the mime npm package

  cd ~/.meteor/tools/latest/lib/node_modules/
  npm install mime
  

• create a rawConnectionHandler to serve the assets (credits to: https://stackoverflow.com/a/20358612) server/static_files_handler.coffee

  fs = Npm.require('fs')
  mime = Npm.require('mime')
  WebApp.rawConnectHandlers.use (req, res, next) ->
    re = /^\/static\/(.*)$/.exec(req.url)
    if re isnt null # Only handle URLs that start with /static/*
      filePath = process.env.PWD + "/public/.#static/" + re[1]
      type = mime.lookup(filePath)
      data = fs.readFileSync(filePath, data)
      res.writeHead 200,
        "Content-Type": type
  
      res.write data
      res.end()
    else # Other urls will have default behaviors
      next()
    return
  

Limitations of this approach:

• not suitable to serve assets with query parameters. The regex would also match /static/html/image.html?src=/static/img/cat.png trying to serve a file with the filename including the parameters. This is easy to change.
• Meteor is completely unaware of the files, they therefore don't get included into the appcache manifest. If you want to make them available offline, check out the addPaths option I added to https://github.com/buildhybrid/appcache-extra

If you don't want to work around the problems, consider serving the assets from an external service (ex. AWS S3).

Answer 2

This is a big issue for large scale applications built in Meteor. I talked to Matt over at Meteor, and he said that their team is working on a solution to this problem for one of their upcoming releases. So get on their newsletter, and you'll be notified when it is available.

Mathieu, your comment:

I finally found a workaround. I'm putting everything in /public/lib/. Then, line 286 of /usr/lib/meteor/app/run.js, I'm adding the folder I don't want Meteor to watch: self.exclude_paths = [ path.join(app_dir, '.meteor', 'local'), path.join(app_dir, 'public', 'lib') ]; This way I can have as much files as I want in lib, and they don't slow everything down. include path is '/lib/dojo/dojo.js'.

is a good hack for now, and here are the other issues that relate that will also be covered in the upcoming release:

• Some packages have overlapping dependencies, but do not exclude them like in PHP's require_once().

• A native require / define dynamic script loader, so not all files are loaded on entry one that can calculate dependency order without having to go levels, lib or main.*

• An official way to create and deploy packages to a repository. Currently it is using Meteorite (mrt) and Atmosphere.