I am trying to cache a command line tool needed for my build process. The tool is made out of NodeJS. The build succeeds, but I need it to run faster.
The relevant parts of my circle.yml look like this :
dependencies:
post:
- npm -g list
- if [ $(npm -g list | grep -c starrynight) -lt 1 ]; then npm install -g starrynight; else echo "StarryNight seems to be cached"; fi
test:
override:
- npm -g list
- starrynight run-tests --framework nightwatch
The second npm -g list
shows starrynight available for use, but the first one shows that it is not being cached.
echo $(npm prefix -g)
. . . gets me . . .
/home/ubuntu/nvm/v0.10.33
. . . so I am assuming CircleCI doesn't cache anything installed globally into nvm.
Nothing I have tried gets me my message, "StarryNight seems to be cached".
How can I cache starrynight?
Cache files are stored in ~/. npm on Posix, or %AppData%/npm-cache on Windows.
DLC caches the individual layers of any Docker images built during your CircleCI jobs, and then reuses unchanged image layers on subsequent CircleCI runs, rather than rebuilding the entire image every time. In short, the less your Dockerfiles change from commit to commit, the faster your image-building steps will run.
Global modules are installed in the /usr/local/lib/node_modules project directory in the standard system, which is the system's root. Print the location of all global modules on your system using this command.
Caching node dependencies is one of circleci's features. In the first build, one can run npm install , save the resulting node_modules directory identified by a checksum of the package. json and when running the build again and package.
Ok, I figured this out. Thanks to Hirokuni Kim of CircleCI for pointing me in the right direction.
The relevant bits of the new circle.yml
looks like this :
machine:
node:
version: 0.10.33
dependencies:
cache_directories:
- ~/nvm/v0.10.33/lib/node_modules/starrynight
- ~/nvm/v0.10.33/bin/starrynight
pre:
- if [ ! -e ~/nvm/v0.10.33/bin/starrynight ]; then npm install -g starrynight; else echo "Starrynight seems to be cached"; fi;
Hirokuni suggested caching ~/nvm
but cache retrieval took as long as the build, since it restores every available version of nodejs
.
I had tried previously to cache just ~/nvm/v0.10.33/lib/node_modules/starrynight
on its own, without realizing that the sister 'directory' bin/starrynight
is actually an essential symlink to the entry point of the module.
My working assumption is that NodeJS modules run from the command line through a series of symbolic references, probably as follows. . .
npm install -g starrynight
creates two new artifacts:
npm
named starrynight
${prefix}/bin
directory, which points to the entry point file, starrynight.js
specified with the bin
key in package.json
.When the user types starrynight
as a CLI command the shell interprets it as an alias for npm
and executes it. npm
examines $0, gets starrynight
, and starts up nodejs
with the symlink ${prefix}/bin/starrynight
as the module to execute. That symlink refers to ~/nvm/v0.10.33/lib/node_modules/starrynight
where the real action takes place.
In short, it is necessary to cache both ${prefix}/lib/node_modules/xxx
and ${prefix}/bin/xxx
The path in the accepted answer doesn't seem to work anymore. I've checked on the build instance that now global npm packages are in
/opt/circleci/nodejs/<version>
so I've added to the circle.yml the following:
- "/opt/circleci/nodejs/v4.3.2/lib/node_modules"
- "/opt/circleci/nodejs/v4.3.2/bin"
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With