Grails watch files doesn't work inside Docker container running inside a Vagrant virtual machine

Question

I have a fairly nested structure:

1. MacOSX workstation running a...
2. Vagrant VirtualBox virtual machine with ubuntu/trusty64 running a...
3. Docker container running...
4. my application written in Grails

Every layer is configured in such a way as to share a portion of the file system from the layer above. This way:

• Vagrant, with config.vm.synced_folder directive in Vagrantfile
• Docker, with the -v command like switch and VOLUME directive in the Dockerfile

This way I can do development on my workstation and the Grails application at the bottom should (ideally) detect changes and recompile/reload on the fly. This is a feature that used to work when I was running the same application straight on the MacOSX, but now grails seems totally unaware of file changes. Of course, if I open the files with an editor (inside the Docker container) they are indeed changed and in fact if I stop/restart the grails app the new code is used.

I don't know how grails implements the watch strategy, but if it depends on some operating system level feature I suspect that file change notifications get lost somewhere in the chain.

Anyone has an idea of what could be the cause and/or how I could go about debugging this?

Answer 1

There are two ways to detect file changes (that I'm aware of):

Polling, which means checking timestamps of all files in a folder at a certain interval. Getting to "near instant" change detection requires very short intervals. This is CPU and disk intensive.

OS Events (inotify on Linux, FSEvents on OS X), where changes are detectable because file operations pass through the OS subsystems. This is easy on the CPU and disk.

Network File Systems (NFS) and the like don't generate events. Since file changes do not pass through the guest OS subsystem, the OS is not aware of changes; only the OS making the changes (OS X) knows about them.

Grails and many other File Watcher tools depend on FSEvents or inotify (or similar) events.

So what to do? It's not practical to 'broadcast' NFS changes from host to all guests under normal circumstances, considering the traffic that would potentially generate. However, I am of the opinion that VirtualBox shares should count as a special exception...

A mechanism to bridge this gap could involve a process that watches the host for changes and triggers a synchronization on the guest.

Check these articles for some interesting ideas and solutions, involving some type of rsync operation:

http://drunomics.com/en/blog/syncd-sync-changes-vagrant-box (Linux) https://github.com/ggreer/fsevents-tools (OS X)

Rsync-ing to a non-NFS folder on your guest (Docker) instance has the additional advantage that I/O performance increases dramatically. VirtualBox shares are just painfully slow.

Update!

Here's what I did. First install lsyncd (OS X example, more info at http://kesar.es/tag/lsyncd/):

brew install lsyncd

Inside my Vagrant folder on my Mac, I created the file lsyncd.lua:

settings {
    logfile = "./lsyncd.log",
    statusFile = "./lsyncd.status",
    nodaemon = true,
    pidfile = "./lsyncd.pid",
    inotifyMode = "CloseWrite or Modify",
}

sync {
    default.rsync,
    delay = 2,
    source = "./demo",
    target = "vagrant@localhost:~/demo",
    rsync = {
        binary   = "/usr/bin/rsync",
        protect_args = false,
        archive = true,
        compress = false,
        whole_file = false,
        rsh = "/usr/bin/ssh -p 2222 -o StrictHostKeyChecking=no"
    },
}

What this does, is sync the folder demo inside my Vagrant folder to the guest OS in /home/vagrant/demo. Note that you need to set up login with SSH keys to make this process frictionless.

Then, with the vagrant VM running, I kicked off the lsyncd process. The -log Exec is optional; it logs its activity to the stdout:

sudo lsyncd lsyncd.lua -log Exec 

On the vagrant VM I started Grails (2.4.4) in my synced folder:

cd /home/vagrant/demo
grails -reloading run-app

Back on my Mac in IntelliJ I edited a Controller class. It nearly immediately triggered lsyncd (2 sec delay) and quickly after that I confirmed Grails recompiled the class!

To summarize:

• Edit your project files on your Mac, execute on your VM
• Use lsyncd to rsync your changes to a folder inside your VM
• Grails notices the changes and triggers a reload
• Much faster disk performance by not using VirtualBox share

Issues: Textmate triggers a type of FSEvent that lsyncd does not (yet) recognize, so changes are not detected. Vim and IntelliJ were fine, though.

Hope this helps someone! Took me a day to figure this stuff out.