Menu
We have Jenkins running within ECS. We are using pipelines for our build and deploy process. The pipeline uses the docker plugin to pull an image which has some dependencies for testing etc, all our steps then occur within this docker container.
The issue we currently have is that our NPM install takes about 8 minutes. We would like to speed this process up. As containers are being torn down at the end of each build then the node_modules that are generated are disposed of. I've considered NPM caching but due to the nature of docker this seemed irrelevant unless we pre-install the dependencies into the docker image (but this triples the size of the image almost). Are there simple solutions to this that will help our NPM install speeds?
618
15 minutes is the default. So you might think, it's great that Netlify builds every commit from my repos…but isn't that a bit wasteful? The answer is, it depends, but we can at least give you some best practices … Thanks for that documentation.
If you find that npm is running slow and it isn't your computer or internet, it is most likely because of a severely outdated version.
Best: Use yarn install or pnpm install - 88% + 80% faster than npm install 2nd Best: Use npm install --prefer-offline --no-audit - 15% faster than npm install If you clean the workspace on every build (or use a build service that doesn't cache environments):
So, you want to make your Jenkins master as “free” from work as you can, leaving the CPU and memory to be used for scheduling and triggering builds on slaves only. In order to do so, you can restrict your jobs to a node label, for example:
The master node is where the application is actually running, this is the brain of your Jenkins and, unlike a slave, it is not replaceable. So, you want to make your Jenkins master as “free” from work as you can, leaving the CPU and memory to be used for scheduling and triggering builds on slaves only.
Here’s what you need to know: Your project must have a package-lock.json file for the following command to work. Instead of using npm install in your build configuration, use npm ci ("clean install") – this command runs faster than npm install and is designed for use in CI environments.
You should be using package caching but not caching node_modules directly. Instead you mount cache directories that your package installer uses, and your install will be blazing fast. Docker does make that possible by allowing you to mount directories in a container, that persist across builds.
For yarn mount ~/.cache or ~/.cache/yarn
For npm mount ~/.npm
docker run -it -v ~/.npm:/.npm ~/.cache:/.cache /my-app:/my-app testing-image:1.0.0 bash -c 'npm ci && npm test`
Note: I'm using npm ci here, which will always delete node_modules and reinstall using exact versions in the package-lock.json, so you get very consistent builds. (In yarn, this is yarn install --frozen-lockfile)
200
You could set up a Http proxy and cache all dependencies (*)(**).
Then use --build-arg to set HTTP_PROXY variable:
docker build --build-arg HTTP_PROXY=http://<cache ip>:3128 .
*: This will not work improve performance on dependencies that need to be compiled (ie: c/c++ bindings)
**: I use a Squid container to share cache configuration
21
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
