Speeding up Jenkins build

Question

I have a NodeJs application. I have a Jenkins job that does the gollowing:

• Does a git clone
• Runs NPM install
• Runs the tests
• Gzips all of the html, css, js files
• Sends the static content to S3 using the aws sync
• Gzips the non-static content so that it can be deployed to our stage environment later.

I'm looking for ways to improve the build. Some of my ideas

• Maybe caching the node modules and not running npm install each time
• Using artifactory as a local node module storage
• Running aws sync commands in parallel
• Using the s3cmd command instead of aws sync (we have a problem where aws sync sends the same files to S3 because git clone changes the modification times of the files)
• Use git diff to get a list of files that changed, and only sending those files to S3

Is there anything else I can do to improve the time of the build?

Answer 1

Some good ideas - here's what I would also consider:

First, try to measure how long each step (cloning, npm install, run tests, ...) takes, and then try to work on improving each step. You can either try to focus on the one that's taking up most of the time, or you can try to get some quick wins before trying to crack the harder nuts.

Some ideas:

• Do you need to do a git clone every time? Maybe consider doing a git pull instead. At the end of the build, do a git reset --hard ; git clean -f -d as the last step to get rid of any uncommitted or locally changed files. I found that doing this is a lot faster than doing a full git clone every time.
• npm install: In line with the above step, if you keep the node_modules folder from the previous build, this step should be a lot quicker as well. There is of the course the risk of updates or removed dependencies, which will still linger around. Many code changes don't require you to reinstall all of the node modules. You might even be able to put in a bit of logic that detects whether the package.json file has changed since the last build, and only run a full npm install in that case.
• I've solved the above by running a nightly build that does a full, clean build: A fresh git clone and a full npm install once per day. All other builds during the day use the previous build state and are a lot quicker as a result of this. If you find that this leads to wrong results, you could schedule the clean build to run several times per day.
• Tune the s3cmd/aws command - I think these can work in a similar way as rsync, which can be really fast and efficient if used the right way. Only try to the git diff route as a last resort. Doing a git pull instead of a git clone might already solve this issue.
• In general, try to understand whether all of the steps need to be run on every build, or whether you can move some of them to other build plans or to other time slots (at night?). If your main purpose of the build plan is to run the unit tests to give quick feedback to developers, then maybe you don't need to zip/copy/deploy the app every time. Or split it up into two build plans, one that runs the tests and returns quickly, and a second one that does all of the deployment. Having this in separate build plans will allow you to be flexible with the scheduling.

These are some initial ideas that I would try as a first step.

Answer 2

Performing a shallow clone speeds up the cloning part. As it does not download all commits.

git clone --depth 1 clone_url...

You should be able to cache node_modules with this plugin: https://plugins.jenkins.io/jobcacher