How does npm handle lock files

Question

So I'm running npm install xyz inside a script that is called by the postinstall npm hook of another npm module. (Let's not get into why please but it has to do with optional dependencies that are not supported by npm)

npm install moduleA
  -> installing moduleA
    -> running postinstall hook (my script)
      -> my script runs npm install xyz

When I run the script itself the npm install runs reasonably fast but when I install the module within the postinstall hook the script waits on a lock file for at least 1 min. (or so I assume)

When I enable verbose loglevel in npm I get an output like below:

[a lot more lines above]
npm verb addNamed "latest" is being treated as a dist-tag for invert-kv
npm info addNameTag [ 'invert-kv', 'latest' ]
npm verb addNameTag registry:https://registry.npmjs.org/invert-kv already in flight; waiting
npm verb addNamed "1.0.0" is a plain semver version for invert-kv
npm verb afterAdd /Users/path/.npm/invert-kv/1.0.0/package/package.json not in flight; writing
npm verb correctMkdir /Users/path/.npm correctMkdir not in flight; initializing
npm verb afterAdd /Users/path/.npm/invert-kv/1.0.0/package/package.json written
npm verb cache add spec camelcase@^3.0.0
npm verb addNamed ">=3.0.0 <4.0.0" is a valid semver range for camelcase
npm verb addNameRange registry:https://registry.npmjs.org/camelcase not in flight; fetching
npm verb get https://registry.npmjs.org/camelcase not expired, no request
npm verb addNamed "3.0.0" is a plain semver version for camelcase
npm verb afterAdd /Users/path/.npm/camelcase/3.0.0/package/package.json not in flight; writing
npm verb correctMkdir /Users/path/.npm correctMkdir not in flight; initializing
npm verb afterAdd /Users/path/.npm/camelcase/3.0.0/package/package.json written
npm verb correctMkdir /Users/path/.npm/_locks correctMkdir not in flight; initializing

waits here for at least one minute

npm verb lock using /Users/path/.npm/_locks/staging-d21af557b41d4821.lock for /Users/path/Desktop/t/node_modules/.staging
npm verb unbuild node_modules/.staging/abbrev-ac014442
npm verb unbuild node_modules/.staging/ansi-regex-12e09986
npm verb unbuild node_modules/.staging/ansi-styles-1359ba2f
npm verb unbuild node_modules/.staging/aproba-d9971840
npm verb unbuild node_modules/.staging/array-find-index-c1ddfc4c
[a lot more lines below]

Now looking at the code of correctMkdir it seems that to just load all all files in /Users/path/.npm/_locks into memory and try to repair permission? Not sure.

correctMkdir is called by lock. This seems to hash the file and some encryption? Also not sure.

After reverse engineering some npm code I made progress by changing the config for cache-lock-stale, cache-lock-wait to 10ms. Unfortunately it still hangs for way too long before installing everything correctly. I suspect the changed config is not taken into account when changing right before and npm being inflight? Maybe? Maybe it's something entirely different? :)

Anyone who can hint me the right direction? Is there something I can do to speed things up?

Answer 1

After a night of sleep the install now seems to be much faster. I'm guessing some caching was still hang in my _locks folder.

So the answer was:

Set cache-lock-stale and cache-lock-wait to 10ms each temporarily and npm won't wait for your lock file to expire. As I understand it npm is trying to wait for all lock files in the _lock folder to be finished and because we are running npm install inside npm install the lock file from the parent process is still in use obviously. This is all more of a guess but I hope it helps someone in the future.