Is there some gulp plugin that lets it perform the function of a static site generator?

Question

I've used a bunch of static site generators, but gulp is a better approach because it is so modular. I'm wondering if there is a plugin that performs some of the functionality of a static site generator. To my mind, all that is missing is something that will turn the files in a directory into a json data structure for use in menus on the site.

Answer 1

If all you want is to generate a .json structure and add it to a file with Gulp, you can do this in one of several ways. The first two use pure Gulp techniques:

• You can write a stream plugin using through or through2, which will basically have to build up the data structure one file at a time, then in a second operation, will create (i.e. push() or queue()) a file at the end

• You can use a Highland pipeline to .reduce() the files to a data structure, then .map() the result to a file, maybe doing a .merge() with the original stream

In both cases, you'll need your generated new vinyl file to have an appropriate .base and .path, which your plugin won't actually know, due to the file you're creating not existing yet. Thus, your code will have to make up a phony absolute path so that gulp.dest() will put it in the right place.

The third technique is to write a Metalsmith plugin instead, which will look something like this:

function generate_json(files, smith, done) {
    var menu = {};
    Object.keys(files).forEach(function(path) {
        var file = files[path];
        if (path.slice(-5)===".html") {
            menu[path] = ... // whatever you want to store about `file`
        }
    });
    files["path/to/menu.json"] = { contents: new Buffer(JSON.stringify(menu)) };
    done();
}

While not much shorter than the code required for the other methods, you will need to understand a lot less in order to do it correctly. Just make sure that if you have an error, you call done(err) in order to pass the error on.

If you would like to combine this plugin with a Gulp pipeline, you can wrap it like so, using Gulpsmith:

gulp.src(...)
.pipe( things() )
.pipe( gulpsmith().use(generate_json) )
.pipe( other_stuff() )
.pipe( gulp.dest(...);

It's true that Gulp has certain advantages over Metalsmith. Ease of writing plugins, sadly, is not one of them. Creating new files from a Gulp plugin is harder than it should be, as is correct error handling. There's also a strong impedance mismatch between the streaming approach, and the nature of static sites to need cross-file operations.

For example, if you wanted to embed your menu in every .html page after creating it, you would need a more complex Gulp plugin, because by the time your plugin had "seen" all the files, they would have "gone downstream", or else you'd have to hang on to them, then stream them out again after you were done. In the Metalsmith plugin, you'd just add a second loop after you generated the menu, to go back over the files again to insert the data, in-place. You don't have to do anything to pass along the files you're not doing anything to.

For these kind of tasks, the Metalsmith plugin API is unequivocally superior to Gulp's. But for tasks that can work on streamed files alone, use existing Gulp plugins for the win.

Basically, Metalsmith is really the Gulp of static site generators, while Gulp is the Gulp of streaming build systems. And you can combine the strengths of both, using Gulpsmith.

(By the way, depending on your actual task, you may find some existing Metalsmith plugins that do all or part of it. For example the metalsmith-collections plugin creates indexes for files matching certain patterns, there's a metalsmith-title that extracts an HTML header to a title property, etc.)

Answer 2

Yes, you can do static generation in gulp.

You probably need fewer plugins than you think. The count might perhaps round down to zero.

Consider what you can do with gulp and no plugins:

var gulp = require('gulp'),
    browserify = require('browserify');

gulp.task('browserify', function(callback) {
    browserify('app.js').bundle()
        .pipe(fs.createWriteStream('dist/app.js')
        .on('close', callback); // why? read on...
});

Too many gulp plugins don't need to exist. They adapt some original module to the vinyl object streams used between gulp.src and gulp.dest, but the vinyl object streams are not compulsory.

Need module X to do some part of your static generation for you? Take callback as an argument to your task function, and pass it to X along with your arguments. Often enough, it's a one liner: require-ing a plugin would just make your code longer.

Even for those parts of your workflow best described as "read everything, handle it in memory, and write it once", you can use through2.obj to transform the vinyl objects or gulp-filter to knock them out. Gulp plugins (or, worse, metalsmith plugins via gulpsmith) are a last resort.

var gulp = require('gulp'),
    filter = require('gulp-filter'),
    through2 = require('through2');

gulp.task('generate-assets', function() {
    return gulp.src('path/to/your/content/and/templates/**')
               .pipe(filter(shouldFileSurvive))
               .pipe(through2.obj(makeFileDifferent))
               .pipe(gulp.dest('dist/'));
});

Note: if you're not calling back, you must return the result of the final pipe. Otherwise, gulp will think your task is synchronous and not wait before kicking off dependent tasks.

If the final pipe returns a normal stream rather than gulp.dest, take a callback (see first example) until b59182c7 lands.

Answer 3

I've been writing such a plugin, it's here gulp-static-site. Looking at the other two answers I would say its functionality fits somewhere inbetween vanilla gulp and gulpsmith.

Gulp-filetree

The main functionality indeed is the computing of a tree-like data structure, which is done by gulp-filetree. It incrementally builds up a tree until the stream ends. Then each file (vinyl object) is reemitted with a .tree property.

Have a look at tree.js to get some more info about that data structure.

Gulp-static-site

This plugins takes a stream of HTML files, sends them through gulp-filetree and into a jade template that adds a basic layout with a directory tree menu and the file contents.

Lookie lookie here for the main pipeline. https://github.com/0x01/gulp-static-site/blob/master/index.js#L146 (As you can see it's not the prettiest but OK. Markdown conversion must be removed as it can be done in gulp outside the plugin. Probably same argument goes for other stuff...)

Q: How does this relate to gulpsmith?

Gulpsmith is definitely seems powerful and complete, I would probably use it had I known about it about it at the time I initially coded this.

That being said; this plugin is simple, lightweight and has a small codebase. I Think it is easy to understand what it does and change to your liking, pull requests certainly welcome :-)

If anything else is unclear, please file an issue or let me know here, thanks