Docker run results in error: layer does not exist

Question

I'm trying to setup redmine (with postgres) on my raspberry pi 3 using docker-compose. It already worked once, but then I tried to install plugins and somehow managed to bork my system.

Now it won't let me start my database container anymore. Even creating a new postgres:12.8 container, yields the error layer does not exist:

$ docker run --rm -it postgres:12.8 bash
docker: Error response from daemon: layer does not exist.

I've already restarted the system twice, which sadly didn't change anything. How can I get this to work again?

Additional info:

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 5
  Running: 4
  Paused: 0
  Stopped: 1
 Images: 65
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.66-v7+
 Operating System: Raspbian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 926.1MiB
 Name: raspberrypi
 ID: <redacted>
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

---

Update 1:

Removing the image (via docker rmi) doesn't stop the error.

$ docker run --rm -it postgres:12.8 bash
Unable to find image 'postgres:12.8' locally
12.8: Pulling from library/postgres
Digest: sha256:e10cd754296c1b3e93a121bcc64550d183df56f20bab47e08cacb123fb969b5e
Status: Downloaded newer image for postgres:12.8
docker: Error response from daemon: layer does not exist.
See 'docker run --help'.

Other images can be run as usual:

$ docker run --rm -it alpine bash
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
4ee0caa23b36: Pull complete
Digest: sha256:eb3e4e175ba6d212ba1d6e04fc0782916c08e1c9d7b45892e9796141b1d379ae
Status: Downloaded newer image for alpine:latest
/ # whoami
root
/ #

Answer 1

Nuking /var/lib/docker seems to get the system working again - this removes all images and lots of other docker-related data.

This doesn't feel like a great solution, but it'll have to do for now.

$ sudo service docker stop
$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
$ sudo rm -rf /var/lib/docker
$ sudo service docker start
$ docker run --rm -it postgres:12.8 bash
Unable to find image 'postgres:12.8' locally
12.8: Pulling from library/postgres
607f77084e8a: Pull complete
ce89a6184ac9: Pull complete
2a4e6e7ac863: Pull complete
f3b6f439172b: Pull complete
2e803a3461b4: Pull complete
467430f26cab: Pull complete
1a81afc37621: Pull complete
4ceea61b7b5c: Pull complete
1f5bfa96a004: Pull complete
b40859b8e537: Pull complete
c7b3bf3cab26: Pull complete
653c63793cd5: Pull complete
aad7328542d8: Pull complete
Digest: sha256:e10cd754296c1b3e93a121bcc64550d183df56f20bab47e08cacb123fb969b5e
Status: Downloaded newer image for postgres:12.8
root@8ba418515688:/# 

References:

• https://github.com/coreos/bugs/issues/1808
• https://github.com/coreos/bugs/issues/1313