NPM Best Practices for Continuous Integration

Question

I am building a HTML5 front-end using NPM-based tools (grunt).

One of the first steps of my continuous integration build process is to run an npm install.

npm install is SLOW. Even with a local NPM proxy caching artifacts (Sonatype's Nexus 3), it is still taking 4 minutes!

$> time npm install
real    4m17.427s
user    0m0.170s
sys     0m0.290s

---

If I follow my usual best practices for continuous integration, I would start from a pristine SCM repository and run the build. This means that each time the CI build will have to do a fresh npm install and take on the cost of 4 minutes.

This is a significant proportion of my build time. I am discontent that the build is taking so long.

---

The alternative seems to be to keep the node_modules around between builds. However, I've had problems with the build becoming unstable as a result.

Removing dependencies from package.json does not remove them from node_modules with a simple npm install. I can work-around this with an npm prune first.

What is considered to be best practice here?

Answer 1

Since March 5, 2018 and npm 5.7.1, you can use npm ci . This is much faster than a regular npm install because it omits some user-friendly features and installs packages for a userless CI environment.

The caveat here is that you'll need to make sure your package.json and package-lock.json files are in sync. If you install a new package, commit package.json but forget to do the same for package-lock.json, you'll get an error when running npm ci.

Answer 2

Considering that in order to build you must install new packages, you have no choice but to call install. As for pristine, I strongly believe they refer to the "build" process and not the "dependency management" process.

Why are they different? Let's go through an example to make it more apparent.

As a developer, when you first start your job, you MUST "install" softwares that will enable to code. This is usually done once. Afterwards, you can start coding. The later is the "build" part as you are generating value for each feature your code produce. From time to time, you can update your tool list by removing, adding or updating one.

In this example, installing your tools everyday you arrive at work before starting coding would be hell.

I would suggest you to make sure that the building process, which means producing an artifact (like a Jar for example), is decoupled from the dependency installation process. Meaning that installation is done once and building can proceed without trouble. You don't mention what will be built, but grunt can take care of the rest for sure.

Hence, I believe pruning and installing is a good strategy. You shouldn't worry for the fist times. Think of it as a cold start. Any system implemented with sub components working together as a pipeline have this "issue". Take a car for example. It will not be as fuel efficient when you start it as when you drive it after an hour.