browserify and babelify very slow due to large data js files

Question

I have a nodejs project which uses large dictionary lists (millions of entries), stored in js files, that look like this:

module.exports = ["entry1", "entry2", "entry3", "entry4", "entry5", etc.];

and then I use them from the other files like this:

var values = require('./filePath');

This works great and it works in the browser too (using browserify), however bundling takes ages - about 10 minutes.

I use the following command to create the bundle: browserify "./src/myModule.js" --standalone myModule -t [ babelify --presets [ es2015 stage-2 ] --plugins ["transform-es2015-classes", {"loose": true}]

I have tried to avoid parsing of my dictionary js files using --noparse ["path1", "path2", "path3", etc.] but it did not make any difference.

Ideally I would like to just speed up the browserify\babelify process, however if that's not possible I would be very happy to find another way (ie. avoid require) to store and use my lists, so that they don't slow the process down but that crucially work in node and in the browser too.

Answer 1

You can bundle the data files separately, so you'll only need to rebundle them when they change. This is possible using the --require -r and --external -x options.

To create the data bundle, do something like this:

browserify -r ./path/to/data.js -r ./path/to/other/data.js > data-bundle.js

The resulting data-bundle.js will define the require function globally which can be used to obtain any file you listed in the command above. Just make sure you include this bundle in a script tag before your main bundle.

It would be nice to be able to --require a glob pattern, but unfortunately browserify does not support this. If you try to use the shell to expand a pattern, the -r option will only apply to the first, which sucks. You can probably write a shell script that builds a command from an ls or something, to avoid having to list all of the data files explicilty, but that's beyond the scope of the question, I think.

To create your main bundle without rebuilding the data files, simply add an option like this to your command:

-x './path/to/data/*.js'

This tells browserify to basically ignore them and let them be pulled in through the global require function created by your other bundle. As you can see, this does support glob patterns, so it's a bit easier.

Update:

To make the two bundles into one, just put something like this at the end of a shell script that starts with the browserify command that builds your main bundle:

cat data-bundle.js main-bundle.js > bundle.js
rm main-bundle.js

Unfortunately this will always have to write a copy of data-bundle.js to disk, which may be the ultimate cause of the slowdown, as I mentioned in the comments below. Worth giving a shot, though.

If even that doesn't work, there are some other, much more hacky approaches you might take. I'll pass on going into those for now though, because I don't think they're worth it unless you absolutely must have it as one file and have no other way of doing it. :\