Incremental gulp less build

Question

In my office we are using gulp to build our less files. I wanted to improve the build task as it took over a second to build on a large project we recently worked on. The idea was to cache the files and only pass the one that changed. So I started with google and found incremental builds for javascript ang thought it would be easy to rewrite them for less. Here's the one I started with: https://github.com/gulpjs/gulp/blob/master/docs/recipes/incremental-builds-with-concatenate.md

After a few unsuccessful tries I ended up with following code (tested with the latest bootstrap distribution):

var gulp            = require('gulp');
var less            = require('gulp-less');
var concat          = require('gulp-concat');
var remember        = require('gulp-remember');
var cached          = require('gulp-cached');

var fileGlob = [
    './bootstrap/**/*.less',
    '!./bootstrap/bootstrap.less',
    '!./bootstrap/mixins.less'
];

gulp.task('less', function () {
    return gulp.src(fileGlob)
        .pipe(cached('lessFiles'))
        .pipe(remember('lessFiles'))
        .pipe(less())
        .pipe(gulp.dest('output'));
});

gulp.task('watch', function () {
    var watcher = gulp.watch(fileGlob, ['less']);
    watcher.on('change', function (e) {
        if (e.type === 'deleted') {
            delete cached.caches.scripts[e.path];
            remember.forget('lessFiles', e.path);
        }
    });
});

But this passes only the changed file and the less compiler fails because of the variable definitions missing. If I pipe the concat plugin before the less task, gulp gets stuck in a (seemingly) endless loop.

gulp.task('less', function () {
    return gulp.src(fileGlob)
        .pipe(cached('lessFiles'))
        .pipe(remember('lessFiles'))
        .pipe(concat('main.less')
        .pipe(less())
        .pipe(gulp.dest('output'));
});

Has anyone experience with those plugins or managed to create an incremental less build in an other way. Here is a (messy) github repository for testing: https://github.com/tuelsch/perfect-less-build

PS: I'm planning on adding linting, sourcemaps, minification, evtl. cache busting and autoprefixer later on.

Answer 1

Like Ashwell, I've found it useful to use imports to ensure that all my LESS files have access to the variables and mixins that they need. I also use a LESS file with imports for bundling purposes. This has a few advantages:

1. I can leverage LESS's features to do complex things like overriding variable values to produce multiple themes, or prepending a class to every rule in another LESS file.
2. There's no need for the concat plugin.
3. Tools like Web Essentials for Visual Studio can give syntax help and output previews, because every LESS file is fully capable of being rendered by itself.

Where you want to import variables, mixins, etc, but you don't want to actually output the entire contents of another file, you can use:

@import (reference) "_colors.less";

After a few days of effort, I was finally able to get an incremental build that correctly rebuilds all the objects that depend on the LESS file I changed. I documented the results here. This is the final gulpfile:

/*
 * This file defines how our static resources get built.
 * From the StaticCommon root folder, call "gulp" to compile all generated
 * client-side resources, or call "gulp watch" to keep checking source 
 * files, and rebuild them whenever they are changed. Call "gulp live" to 
 * do both (build and watch).
 */

/* Dependency definitions: in order to avoid forcing everyone to have 
 * node/npm installed on their systems, we are including all of the 
 * necessary dependencies in the node_modules folder. To install new ones,
 * you must install nodejs on your machine, and use the "npm install XXX" 
 * command. */
var gulp = require('gulp');
var less = require('gulp-less');
var LessPluginCleanCss = require('less-plugin-clean-css'),
    cleanCss = new LessPluginCleanCss();
var sourcemaps = require('gulp-sourcemaps');
var rename = require('gulp-rename');
var cache = require('gulp-cached');
var progeny = require('gulp-progeny');
var filter = require('gulp-filter');
var plumber = require('gulp-plumber');
var debug = require('gulp-debug');

gulp.task('less', function() {
    return gulp
        // Even though some of our LESS files are just references, and 
        // aren't built, we need to start by looking at all of them because 
        // if any of them change, we may need to rebuild other less files.
        .src(
        ['Content/@(Theme|Areas|Css)/**/*.less'],
        { base: 'Content' })
        // This makes it so that errors are output to the console rather 
        // than silently crashing the app.
        .pipe(plumber({
            errorHandler: function (err) {
                console.log(err);
                // And this makes it so "watch" can continue after an error.
                this.emit('end');
            }
        }))
        // When running in "watch" mode, the contents of these files will 
        // be kept in an in-memory cache, and after the initial hit, we'll
        // only rebuild when file contents change.
        .pipe(cache('less'))
        // This will build a dependency tree based on any @import 
        // statements found by the given REGEX. If you change one file,
        // we'll rebuild any other files that reference it.
        .pipe(progeny({
            regexp: /^\s*@import\s*(?:\(\w+\)\s*)?['"]([^'"]+)['"]/
        }))
        // Now that we've set up the dependency tree, we can filter out 
        // any files whose
        // file names start with an underscore (_)
        .pipe(filter(['**/*.less', '!**/_*.less']))
        // This will output the name of each LESS file that we're about 
        // to rebuild.
        .pipe(debug({ title: 'LESS' }))
        // This starts capturing the line-numbers as we transform these 
        // files, allowing us to output a source map for each LESS file 
        // in the final stages.
        // Browsers like Chrome can pick up those source maps and show you 
        // the actual LESS source line that a given rule came from, 
        // despite the source file's being transformed and minified.
        .pipe(sourcemaps.init())
        // Run the transformation from LESS to CSS
        .pipe(less({
            // Minify the CSS to get rid of extra space and most CSS
            // comments.
            plugins: [cleanCss]
        }))
        // We need a reliable way to indicate that the file was built
        // with gulp, so we can ignore it in Mercurial commits.
        // Lots of css libraries get distributed as .min.css files, so
        // we don't want to exclude that pattern. Let's try .opt.css 
        // instead.
        .pipe(rename(function(path) {
            path.extname = ".opt.css";
        }))
        // Now that we've captured all of our sourcemap mappings, add
        // the source map comment at the bottom of each minified CSS 
        // file, and output the *.css.map file to the same folder as 
        // the original file.
        .pipe(sourcemaps.write('.'))
        // Write all these generated files back to the Content folder.
        .pipe(gulp.dest('Content'));
});

// Keep an eye on any LESS files, and if they change then invoke the 
// 'less' task.
gulp.task('watch', function() {
    return gulp.watch('Content/@(Theme|Areas|Css)/**/*.less', ['less']);
});

// Build things first, then keep a watch on any changed files.
gulp.task('live', ['less', 'watch']);

// This is the task that's run when you run "gulp" without any arguments.
gulp.task('default', ['less']);

We can now simply run gulp live to build all our LESS files once, and then allow each subsequent change to just build those files that depend on the changed files.

Answer 2

So when I want to do incremental builds in gulp I do it by abstracting out the inner process of the gulp task, this way I don't have to worry about keeping a cache.

// Create a function that does just the processing
var runCompile = function( src, dest, opts ){
  return gulp.src( src )
    .pipe(less( opts ))
    .pipe(gulp.dest( dest ));
};

// Leverage the function to create the task
gulp.task( 'less', function(){
  return runCompile( fileGlob, 'output', {} );
});

// Use it again in the watch task
gulp.task( 'less:watch', function(){
  return gulp.watch( fileGlob )
    .on( "change", function( event ){
      // might need to play with the dest dir here
      return runCompile( event.path, 'output', {} );
    });
});

This works great for me and I use this pattern all over my gulp tasks. However I have noticed that sometime gulp will squash paths during the watch "on change" if it gets a single file. In that case I do the path manipulation my self, something like path.dirname(srcPath.replace( srcDir, outputDir )) as the argument dest for the runCompile function.

Edit: Just realized this probably isn't going to solve your "lost variables" problem. I don't have anything off the top of my head to solve that one since I organize my LESS files with a heavy use of imports, so every file that would need a set of variables would have an import statement insuring they are there.

Answer 3

We can actually use gulp-newer and gulp-progeny-mtime for this task. The approach by Stripling is almost the best one expect every time you run the gulp less task it will compile everything again from scratch and then will start watching the files. This will cost you a lot of time if you are working with a lot of less style sheets. gulp-progeny-mtime is similar to gulp-progeny except it does the real hardcore stuff. Every time a files passes through gulp-progeny-mtime, it checks for any modification in imports, and if so it will adjust the mtime of the current file in the stream which leads it to pass through gulp-newer. This I feel is better because we are not even caching anything.

   //Compile less for deployment 
   gulp.task("less", () => {
      return gulp
        .src(["static/less/**/*.less"])
        .pipe(progenyMtime())
        .pipe(
          plumber({
            errorHandler: function (err) {
              log(chalk.bgRed.white.bold(err.message));
            },
          })
        )
        .pipe(filter(["**/*.less", "!**/_*.less", "!static/less/includes*/**"]))
        .pipe(newer({ dest: "static/css/", ext: ".css" }))
        .pipe(debug({ title: "LESS" }))
        .pipe(
          less({
            plugins: [cleanCss, autoprefix],
          })
        )
        .pipe(gulp.dest("static/css/"));
    });

    //Watch changes is less and compile if changed.
    gulp.task("watch-less", () => {
      return gulp.watch("static/less/**/*.less", gulp.series("less"));
    });
    
    //Compile all less files on first run ( if changed ) then compile only modified files from next run
    gulp.task("live-less", gulp.series("less", "watch-less"));