Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Cache node_modules in GitHub Actions

I have a Yarn monorepo (workspace) with 2 packages: backend (Node.js/TypeScript) and frontend (React/Typescript).

/package.json (trimmed)

{
  "workspaces": [
    "backend",
    "frontend"
  ],
}

and I'm trying to add continuous integration with GitHub Actions and trying to use actions/cache@v2 to cache the Yarn cache dir and all of the projects' node_modules dirs

.github/workflows/CI.yml (trimmed)

    steps:
      - uses: actions/checkout@v2

      - name: Get yarn cache directory path
        id: yarn-cache-dir-path
        run: |
          echo "::set-output name=dir::$(yarn cache dir)"
          echo "::set-output name=version::$(yarn -v)"

      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v2
        with:
          node-version: ${{ matrix.node-version }}

      - uses: actions/cache@v2
        with:
          path: |
            ${{ steps.yarn-cache-dir-path.outputs.dir }}
            '**/node_modules'
            '**/.eslintcache'
          key: ${{ runner.os }}-yarn-${{ steps.yarn-cache-dir-path.outputs.version }}-${{ hashFiles('**/yarn.lock') }}

      - name: Install packages
        run: yarn install --frozen-lockfile

I receive that the cache is stored and re-used for the consecutive runs:

key: Linux-yarn-1.22.10-143fef95c7228810cf502305eff3be1cbc468dc8a3e0b153a4311c0250aaef6f
Received 158645465 of 175422681 (90.4%), 151.3 MBs/sec
Received 175422681 of 175422681 (100.0%), 138.1 MBs/sec
Cache Size: ~167 MB (175422681 B)
/usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/08363700-9a23-447e-a80e-6f3dbec6068f/cache.tzst -P -C /home/runner/work/path
Cache restored successfully
Cache restored from key: Linux-yarn-1.22.10-143fef95c7228810cf502305eff3be1cbc468dc8a3e0b153a4311c0250aaef6f

but yarn still tries to resolve the dependencies:

yarn install --frozen-lockfile
shell: /usr/bin/bash -e {0}
yarn install v1.22.10
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
.....
[4/4] Building fresh packages...
Done in 40.07s.

My expectations are that it should be working as on my local machine:

$ yarn --frozen-lockfile
yarn install v1.22.10
[1/4] 🔍  Resolving packages...
success Already up-to-date.
✨  Done in 0.72s.

Can I somehow improve my configuration to meet the expected result, or this GitHub Actions behaviour expected?


Update: When tried with the following paths:

          path: |
            '**/node_modules'
            '**/.eslintcache'

or:

          path: |
            'node_modules'
            '*/node_modules'
            '**/.eslintcache'

the cache size is 22 B. Probably not matching any of the node_modules dirs

like image 912
Teneff Avatar asked Apr 17 '21 09:04

Teneff


People also ask

Does GitHub Actions cache?

About caching workflow dependencies To help speed up the time it takes to recreate files like dependencies, GitHub can cache files you frequently use in workflows. To cache dependencies for a job, you can use GitHub's cache action. The action creates and restores a cache identified by a unique key.

Should you cache Node_modules?

As I understand the documentation about caching both in this repo and in the upstream actions/cache repo, the reason node_modules is not recommended to cache is because dependencies can break across versions of Node. js.

What is Node_modules cache?

node_modules/. cache is a community-standard cache folder for storing files.

Do GitHub Actions cost money?

GitHub Actions usage is free for standard GitHub-hosted runners in public repositories, and for self-hosted runners. For private repositories, each GitHub account receives a certain amount of free minutes and storage for use with GitHub-hosted runners, depending on the product used with the account.


3 Answers

After a lot of trial and error, removing the quotes from the paths seems to have fixed the problem. And the size of the cache have increased almost twice

  - uses: actions/cache@v2
    id: yarn-cache
    with:
      path: |
        **/node_modules
        **/.eslintcache
        ${{ steps.yarn-cache-dir-path.outputs.dir }}

      key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
      restore-keys: |
        ${{ runner.os }}-yarn-

Run actions/cache@v2

Received 213909504 of 305669200 (70.0%), 203.6 MBs/sec
Received 305669200 of 305669200 (100.0%), 185.6 MBs/sec
Cache Size: ~292 MB (305669200 B)
/usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/2e2d2a1d-04d7-44c3-829e-ec4e8faf394b/cache.tzst -P -C /home/runner/work/path
Cache restored successfully
Cache restored from key: Linux-yarn-143fef95c7228810cf502305eff3be1cbc468dc8a3e0b153a4311c0250aaef6f

Run yarn install --frozen-lockfile

yarn install v1.22.10
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.96s.
  • permalink to workflow
  • master version
like image 61
Teneff Avatar answered Oct 17 '22 03:10

Teneff


For anyone else looking for yarn caching implementation in gha, the following code is working best for me:

      - name: Get yarn cache directory path
        id: yarn-cache-dir-path
        run: echo "::set-output name=dir::$(yarn cache dir)" 

      - uses: actions/cache@v2
        id: yarn-cache
        with:
          path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-

      - name: Install dependencies
        run: yarn --prefer-offline
        if: steps.yarn-cache.outputs.cache-hit != 'true'


It is best practice to NOT cache the node_modules.

Caching node_modules to determine which packages to install could led to improper cache busting.

like image 2
TillyJonesy Avatar answered Oct 17 '22 03:10

TillyJonesy


You shouldn't be caching node_modules!
setup-node GitHub action, comes with built-in caching on its own. As written in docs, just add cache property to GitHub Action, where you define your package manager npm | yarn | pnpm

steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
  with:
    node-version: 16
    cache: 'npm' # or 'yarn' | 'pnpm'
- run: npm ci
- run: npm test
like image 1
marko424 Avatar answered Oct 17 '22 02:10

marko424