Github runner out of disk space after building docker image

Question

I have two github repos (call them A and B) that both have a .github/workflows/build_and_test.yml file that looks almost identical to this:

name: Build and test

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build-with-docker:
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v3
      - uses: docker/setup-buildx-action@v2
      - uses: docker/build-push-action@v3
        with:
          tags: my_tag:latest
          load: true
          context: .
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - run: docker run my_tag pytest

In repo A

• the base image in my Dockerfile is r-base:4.2.1, which is around 782MB
• the image that I build ends up being around 1.71GB
• the github workflow runs happily from beginning to end (the image builds, the docker run my_tag pytest step runs, all is well)

In repo B

• the base image in my Dockerfile is pytorch/pytorch, which is around 5.82GB
• the image that I build ends up being around 6.4GB
• the github workflow surfaces two problems, one warning and one error, which I'll describe below

The warning is You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 0 MB.

The error happens in the docker run my_tag pytest step: I get

Unable to find image 'my_tag:latest' locally
docker: Error response from daemon: pull access denied for my_tag, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.

which I suspect is a red herring that would go away if I fixed the disk space issue, although I am not certain. (I suspect Unable to find image is a red herring because I do not get this error in repo A.)

Is there a way to reduce the disk space used by my build-push-action step in repo B without changing its Dockerfile? Should I be using the no-cache-filters input described at https://github.com/docker/build-push-action#inputs?

(Possibly related: How to remove old docker images in continuous integration to save disk space, although I am looking for an answer that better fits my github workflow setup.)

According to https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources github's Linux runners have 14 GB of SSD space.

Update: I tried adding no-cache: true to docker/build-push-action@v3 in repo B, and I see the exact same running out of disk space warning and Unable to find image error as before.

Confusingly, the github UI has a check mark next to Run docker/build-push-action@v3, which indicates that that step ran successfully, so I'm not sure at what exact point in time the runner runs out of disk space:

Answer 1

The solution provided by the OP is good but it may not suffice. In my case since I had a big docker image (about ~20G), I had to delete many more APT software and further delete the .git/ directory created after the checkout (normally you don't need the .git/ into your docker image for production).

I used this and now it works just fine

name: Docker build and push

on:
  push:
    branches:
      - "main"

jobs:
  build:
    runs-on: ubuntu-20.04
    steps:
      - 
        name: Check disk space
        run: df . -h
      - 
        name: Free disk space
        run: |
          sudo docker rmi $(docker image ls -aq) >/dev/null 2>&1 || true
          sudo rm -rf \
            /usr/share/dotnet /usr/local/lib/android /opt/ghc \
            /usr/local/share/powershell /usr/share/swift /usr/local/.ghcup \
            /usr/lib/jvm || true
          echo "some directories deleted"
          sudo apt install aptitude -y >/dev/null 2>&1
          sudo aptitude purge aria2 ansible azure-cli shellcheck rpm xorriso zsync \
            esl-erlang firefox gfortran-8 gfortran-9 google-chrome-stable \
            google-cloud-sdk imagemagick \
            libmagickcore-dev libmagickwand-dev libmagic-dev ant ant-optional kubectl \
            mercurial apt-transport-https mono-complete libmysqlclient \
            unixodbc-dev yarn chrpath libssl-dev libxft-dev \
            libfreetype6 libfreetype6-dev libfontconfig1 libfontconfig1-dev \
            snmp pollinate libpq-dev postgresql-client powershell ruby-full \
            sphinxsearch subversion mongodb-org azure-cli microsoft-edge-stable \
            -y -f >/dev/null 2>&1
          sudo aptitude purge google-cloud-sdk -f -y >/dev/null 2>&1
          sudo aptitude purge microsoft-edge-stable -f -y >/dev/null 2>&1 || true
          sudo apt purge microsoft-edge-stable -f -y >/dev/null 2>&1 || true
          sudo aptitude purge '~n ^mysql' -f -y >/dev/null 2>&1
          sudo aptitude purge '~n ^php' -f -y >/dev/null 2>&1
          sudo aptitude purge '~n ^dotnet' -f -y >/dev/null 2>&1
          sudo apt-get autoremove -y >/dev/null 2>&1
          sudo apt-get autoclean -y >/dev/null 2>&1
          echo "some packages purged"
      - 
        name: Check disk space
        run: |
          sudo dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -nr | head
          df . -h
          sudo du /usr/ -hx -d 4 --threshold=1G | sort -hr | head
      -
        name: Checkout
        uses: actions/checkout@v3
      - 
        name: Check working space directory
        run: du ${GITHUB_WORKSPACE} -h -d 1
      - 
        name: Get more space
        run: |
          df . -h
          sudo rm -rf ${GITHUB_WORKSPACE}/.git
          df . -h
      -
        name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      -
        name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: ${{ secrets.DOCKER_USERNAME }}/my-app:latest

Answer 2

I got this to work thanks to https://github.com/actions/runner-images/issues/2840#issuecomment-1284059930.

My new github workflow yml file is

name: Build Docker image

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build-with-docker:
    runs-on: ubuntu-20.04
    steps:
      - name: Remove unnecessary files
        run: |
          sudo rm -rf /usr/share/dotnet
          sudo rm -rf "$AGENT_TOOLSDIRECTORY"
      - uses: actions/checkout@v3
      - uses: docker/setup-buildx-action@v2
      - uses: docker/build-push-action@v4
        with:
          context: .
          tags: my_tag:latest
          load: true
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Run pytest
        run: docker run my_tag pytest

The Remove unnecessary files step fixed the disk space warning and the Unable to find image error went away (so it was indeed a red herring).