Deep, one-way synchronization of two directories using grunt-contrib-watch. Code works, but grunt-contrib-watch re-init time is too slow

Question

I have two directories src and compiled. I would like to ensure one-way data synchronization from src to compiled with Grunt Watch. As an intermediary step, I would like to compile *.less files as well as a subset of *.js files which are written with ES6 syntax.

I've successfully written the tasks which do what I need:

// NOTE: Spawn must be disabled to keep watch running under same context in order to dynamically modify config file.
watch: {
  // Compile LESS files to 'compiled' directory.
  less: {
    options: {
      interrupt: true,
      spawn: false,
      cwd: 'src/less'
    },
    files: ['**/*.less'],
    tasks: ['less']
  },
  // Copy all non-ES6/LESS files to 'compiled' directory. Include main files because they're not ES6. Exclude LESS because they're compiled.
  copyUncompiled: {
    options: {
      event: ['added', 'changed'],
      spawn: false,
      cwd: 'src'
    },
    files: ['**/*', '!**/background/**', '!**/common/**', '!contentScript/youTubePlayer/**/*', '!**/foreground/**', '!**/test/**', '!**/less/**', '**/main.js'],
    tasks: ['copy:compileSingle']
  },
  // Compile and copy ES6 files to 'compiled' directory. Exclude main files because they're not ES6.
  copyCompiled: {
    options: {
      event: ['added', 'changed'],
      spawn: false,
      cwd: 'src/js'
    },
    files: ['background/**/*', 'common/**/*', 'contentScript/youTubePlayer/**/*', 'foreground/**/*', 'test/**/*', '!**/main.js'],
    tasks: ['babel:compileSingle']
  },
  // Whenever a file is deleted from 'src' ensure it is also deleted from 'compiled'
  remove: {
    options: {
      event: ['deleted'],
      spawn: false,
      cwd: 'src'
    },
    files: ['**/*'],
    tasks: ['clean:compiledFile']
  }
}

  grunt.event.on('watch', function(action, filepath, target) {
    // Determine which task config to modify based on the event action.
    var taskTarget = '';
    if (action === 'deleted') {
      taskTarget = 'clean.compiledFile';
    } else if (action === 'changed' || action === 'added') {
      if (target === 'copyCompiled') {
        taskTarget = 'babel.compileSingle';
      } else if (target === 'copyUncompiled') {
        taskTarget = 'copy.compileSingle';
      }
    }

    if (taskTarget === '') {
      console.error('Unable to determine taskTarget for: ', action, filepath, target);
    } else {
      // Drop src off of filepath to properly rely on 'cwd' task configuration.
      grunt.config(taskTarget + '.src', filepath.replace('src\\', ''));
    }
  });

These tasks watch the appropriate files. The event handler dynamically modifies clean copy and babel tasks such that they work upon the files being added/changed/removed.

However, I am watching several thousand files and the watch task takes a non-trivial amount of time to initialize. On my high-end development PC initialization takes 6+ seconds. This issue is exacerbated by the fact that the watch task reinitializes after every task.

This means that if I have two files, fileA and fileB, and I modify fileA and save then there is a 6+ second period where watch fails to detect modifications to fileB. This results in de-synchronization between my two directories.

I found this GitHub issue regarding my problem, but it is still open and unanswered: https://github.com/gruntjs/grunt-contrib-watch/issues/443

The discussion on GitHub highlights that the issue may only occur when spawn: false has been set, but, according to the Grunt Watch documentation:

If you need to dynamically modify your config, the spawn option must be disabled to keep the watch running under the same context.

As such, I believe I need to continue using spawn: false.

I have to assume this is a pretty standard procedure for Grunt tasks. Am I missing something obvious here? Is the Watch task inappropriate for this purpose? Other options?

Answer 1

Alright, so I have a working solution, but it's not pretty.

I did end up using grunt-newer to assist with the solution. Unfortunately, it doesn't play well with grunt-contrib-copy because copying a file does not update its last modified time and so grunt-newer will execute 100% of the time.

So, I forked grunt-contrib-copy and added an option to allow updating the last modified time: https://github.com/MeoMix/grunt-contrib-copy

With that, I'm now able to write:

// NOTE: Spawn must be disabled to keep watch running under same context in order to dynamically modify config file.
watch: {
  // Compile LESS files to 'compiled' directory.
  less: {
    options: {
      interrupt: true,
      cwd: 'src/less'
    },
    files: ['**/*.less'],
    tasks: ['less']
  },
  // Copy all non-ES6/LESS files to 'compiled' directory. Include main files because they're not ES6. Exclude LESS because they're compiled.
  copyUncompiled: {
    options: {
      event: ['added', 'changed'],
      cwd: 'src'
    },
    files: ['**/*', '!**/background/**', '!**/common/**', '!contentScript/youTubePlayer/**/*', '!**/foreground/**', '!**/test/**', '!**/less/**', '**/main.js'],
    tasks: ['newer:copy:compiled']
  },
  // Compile and copy ES6 files to 'compiled' directory. Exclude main files because they're not ES6.
  copyCompiled: {
    options: {
      event: ['added', 'changed'],
      cwd: 'src/js'
    },
    files: ['background/**/*', 'common/**/*', 'contentScript/youTubePlayer/**/*', 'foreground/**/*', 'test/**/*', '!**/main.js'],
    tasks: ['newer:babel:compiled']
  },
  // Whenever a file is deleted from 'src' ensure it is also deleted from 'compiled'
  remove: {
    options: {
      event: ['deleted'],
      spawn: false,
      cwd: 'src'
    },
    files: ['**/*'],
    tasks: ['clean:compiledFile']
  }
}

grunt.event.on('watch', function(action, filepath) {
  if (action === 'deleted') {
    // Drop src off of filepath to properly rely on 'cwd' task configuration.
    grunt.config('clean.compiledFile.src', filepath.replace('src\\', ''));
  }
});

Now copying of ES6 files as well as non-LESS/non-ES6 files will only occur if 'src' is newer than 'dest.'

Unfortunately, grunt-newer doesn't really have support for syncing a delete operation when deleted from 'src'. So, I continue to use my previous code for 'delete' operations. This still has the same flaw where after a delete occurs the watch task will be defunct for a moment.