npm install dependencies of a git submodule

Question

I've got a git project with handlebars.js as a submodule. I will provide here a bare-bones example - the full project is more complicated with dojo tasks and deployment steps that aren't relevant here.

$ mkdir handlebarstest ; cd handlebarstest ; git init
$ mkdir src ; git add src
$ git submodule add https://github.com/wycats/handlebars.js.git src/handlebars.js

Our package.json doesn't mention handlebars.js, just grunt:

{
  "name": "handlebarstest",
  "version": "1.0.0",
  "dependencies": {},
  "devDependencies": {
    "grunt": "~0.4.1",
    "grunt-contrib-clean": "~0.4.1",
    "grunt-run-grunt": "latest"
  }
}

In this project's history, our install and build procedure has been:

$ npm install   # install dependencies in package.json
$ grunt init    # run the 'init' task which initialises git submodules
$ grunt build   # run the 'build' task which compiles dojo etc.

Having done a fresh clone of our project, say on our build server, git submodules need to be initialised, and we control that from grunt, so we must install grunt first, then run a task to init submodules, including the troublesome handlebars.js.

We do npm install and install grunt, and grunt init which fetches handlebars.js including its package.json. So the root of our problem is that the package.json is not available when the top level npm install is run.

Our Gruntfile.js knows how to call the Gruntfile.js in handlebars.js:

/*jshint node:true */
module.exports = function (grunt) {
    /*jshint camelcase: false */
    var path = require('path');

    grunt.initConfig({
        clean: [ 'dist' ],
        run_grunt: {
            options: {

            },
            handlebars_build: {
                options: {
                    log: true
                },
                src: [ 'src/handlebars.js/Gruntfile.js' ],
                task: 'build'
            },
            handlebars_amd: {
                options: {
                    log: true
                },
                src: [ 'src/handlebars.js/Gruntfile.js' ],
                task: 'amd'
            }
        }
    });

    // var handlebarsLink= grunt.registerTask('handlebarsLink', function () {
    //  var done = this.async(),
    //      child = grunt.util.spawn({
    //          cmd: 'npm',
    //          args: [ 'link', 'src/handlebars.js' ]
    //      }, function (error) {
    //          if (error) {
    //              grunt.warn(error);
    //              done(false);
    //              return;
    //          }
    //          done();
    //      });

    //  child.stdout.on('data', function (data) {
    //      grunt.log.write(data);
    //  });
    //  child.stderr.on('data', function (data) {
    //      grunt.log.error(data);
    //  });
    // });
    var submodules;
    if (grunt.file.exists(__dirname, '.git')) {
        submodules = grunt.registerTask('submodules', function () {
            var done = this.async(),
                child = grunt.util.spawn({
                    cmd: 'git',
                    args: [ 'submodule', 'update', '--init', '--recursive' ]
                }, function (error) {
                    if (error) {
                        grunt.warn(error);
                        done(false);
                        return;
                    }
                    done();
                });

            child.stdout.on('data', function (data) {
                grunt.log.write(data);
            });
            child.stderr.on('data', function (data) {
                grunt.log.error(data);
            });
        });
    }
    var init = submodules ? [ 'submodules'/*, 'handlebarsLink'*/ ] : [];
    grunt.registerTask('init', init);
    grunt.registerTask('default', 'build');
    grunt.registerTask('build', init.concat([ 'clean', 'run_grunt:handlebars_build', 'run_grunt:handlebars_amd' ]));

    grunt.loadTasks(path.join(__dirname, 'grunt'));
    grunt.loadTasks(path.join(__dirname, 'src', 'intern', 'tasks'));
    grunt.loadNpmTasks('grunt-contrib-clean');
    grunt.loadNpmTasks('grunt-run-grunt');
};

Running grunt fails because it recurses into handlebars.js but the module dependencies in its package.json have not been installed.

Running "run_grunt:handlebars_build" (run_grunt) task
--> starting  "src/handlebars.js/Gruntfile.js"
--> reporting "src/handlebars.js/Gruntfile.js"
  |  >> Local Npm module "grunt-contrib-clean" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-concat" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-connect" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-copy" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-requirejs" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-jshint" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-uglify" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-watch" not found. Is it installed?
  |  >> Local Npm module "grunt-saucelabs" not found. Is it installed?
  |  >> Local Npm module "es6-module-packager" not found. Is it installed?
  |  Loading "metrics.js" tasks...ERROR
  |  >> Error: Cannot find module 'underscore'
  |  Loading "publish.js" tasks...ERROR
  |  >> Error: Cannot find module 'underscore'
  |  Loading "version.js" tasks...ERROR
  |  >> Error: Cannot find module 'async'
  |  Warning: Task "clean" not found. Use --force to continue.
  |  
  |  Aborted due to warnings.
  |  
--> failed "src/handlebars.js/Gruntfile.js" (304ms)
--> failed handlebars_build @ "src/handlebars.js/Gruntfile.js"

Warning: 1 gruntfile failed and completed 0 (308ms)
 Use --force to continue.

Solutions might be:

1. Use npm link somehow in our top level package.json to do something clever with a script hook after npm install has finished. This seems impossible, because the handlebars.js/package.json isn't going to be available until well after npm install has finished.
2. Put a manual step before run_grunt:handlebars_build to recurse into src/handlebars.js and run npm install in that directory. This seems more manual than I would expect.
3. Put a step before run_grunt:handlebars_build that installs dependencies from src/handlebars.js/package.json into the project's top level node_modules directory, which might then be picked up when running Gruntfile.js inside src/handlebars.js (this may require that there is no node_modules directory in src/handlebars.js - my knowledge of npm and node_modules is not great).
4. Give up on trying to build handlebars.js as part of our project's build, remove the src/handlebars.js submodule from the project, and just add built files as plain files in its place. This would be a shame because you lose the benefits of tracking a project's versions intelligently via the git submodule approach.

Some advice would be appreciated on the best way forward.

Answer 1

To ensure your submodule's dependencies are installed whenever you install your main repo's dependencies, you need to modify your main package.json.

Assuming your submodule is called 'common' and sits under src/common and its package.json is in src/common, then,

You should run npm install --save file:src/common and it will add the dependency to your package.json (or you can add by yourself, just append to your dependencies key the key:value of "common": "file:src/common",

Running npm install afterwards, will install all dependencies in the main module and submodule.

These steps will also help solve the issue in continuous deployment environments.

Answer 2

this might be of some help to someone stumbling over this question like i did during my research regarding a similar question:

My solution uses the grunt plugin grunt-run to run npm install in the submodule directory (installing its dependencies into its own node_modules directory: that would be src/handlebars.js/node_modules/ in your case).

In my case i needed to compile the TypeScript files using a grunt task inside the sub-module afterwards. I solved this using the plugin grunt-submodule.

The

>> Local Npm module "grunt-contrib-clean" not found. Is it installed?`

errors seem to appear if grunt is being executed in the wrong working directory - if you've managed to install the dependencies before.

Example:

Only the sub-module uses grunt-contrib-clean to clean its build artifacts. You've installed the sub-modules requirements (inside its directory) but you run grunt clean in the parents directory. The node module is not known in this place.

Hope that helps,

Markus

---

These tasks are of course run from within grunt, integrated into my build/development life-cycle.

References:

• https://www.npmjs.com/package/grunt-run
• https://www.npmjs.com/package/grunt-submodule