Docker gets stuck on building and other commands

Question

There's a very strange issue, the docker daemon can't response the building request and other commands like docker info. But if I build image from an exist image Dockerfile FROM some/exist/image, it works well, docker version works well too. We have about 56 VMs to build images, the building command is just like this::

export DOCKER_HOST = '<my_host>'
docker build -t <tag> - < context.tar.gz

The client output below, it seems that it gets stuck on extrating:

[test]$ docker build -t test .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM busybox
latest: Pulling from library/busybox
7c9d20b9b6cd: Extracting  32.77kB/760.8kB
^C

But the cpu and memory are no problem, also there's still a lot of disk space.

[~]# top
top - 10:18:09 up 48 days, 20:20,  1 user,  load average: 1.00, 1.03, 1.05
Tasks: 167 total,   1 running, 166 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8006376 total,   603984 free,   612016 used,  6790376 buff/cache
KiB Swap:  4192928 total,  4186560 free,     6368 used.  6749588 avail Mem

Log:

I used systemctl to start dockerd and it's strange that I can't see recent log about dockerd, it seems that dockerd gets stuck from 16:04:08:

cat /var/log/messages|grep docker
2019-10-28 16:04:07 dockerd[1549]: time="2019-10-28T16:04:07+08:00" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/d584a3957292ac1d3edfea2ec57abcf2dbe79465f9bcd41183b0ccd9dac3bb01/shim.sock" debug=false pid=18014
2019-10-28 16:04:08 dockerd[1549]: time="2019-10-28T16:04:08+08:00" level=info msg="shim reaped" id=d584a3957292ac1d3edfea2ec57abcf2dbe79465f9bcd41183b0ccd9dac3bb01
2019-10-28 16:04:08 dockerd[1549]: time="2019-10-28T16:04:08.081553011+08:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2019-10-28 19:15:00 dockerd[1549]: time="2019-10-28T19:15:00.826993765+08:00" level=error msg="Not continuing with pull after error: context canceled"
2019-10-28 19:17:49 dockerd[1549]: time="2019-10-28T19:17:49.216471516+08:00" level=error msg="Handler for POST /v1.38/build returned error: Error processing tar file(exit status 1): unexpected EOF"
2019-10-28 19:42:18 dockerd[1549]: time="2019-10-28T19:42:18.954682094+08:00" level=error msg="Not continuing with pull after error: context canceled"
2019-10-28 19:51:51 dockerd[1549]: time="2019-10-28T19:51:51.611278576+08:00" level=info msg="Attempting next endpoint for pull after error: manifest unknown: manifest unknown"
2019-10-28 19:52:03 dockerd[1549]: time="2019-10-28T19:52:03.076089163+08:00" level=info msg="Attempting next endpoint for pull after error: manifest unknown: manifest unknown"
2019-10-28 19:55:41 dockerd[1549]: time="2019-10-28T19:55:41.605964950+08:00" level=info msg="Attempting next endpoint for pull after error: Get https://xx/centos/manifests/6: no basic auth credentials"
2019-10-28 19:55:44 dockerd[1549]: time="2019-10-28T19:55:44.298856115+08:00" level=info msg="Attempting next endpoint for pull after error: Get https://xx/centos/manifests/6: no basic auth credentials"

Docker processes:

[~]# ps -ef|grep docker
root      1549     1  6 Oct09 ?        1-05:45:10 /usr/bin/dockerd
root      1557  1549  0 Oct09 ?        00:42:20 docker-containerd --config /var/run/docker/containerd/containerd.toml
root      2574  1549  0 Oct09 ?        00:00:01 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8090 -container-ip x.x.x.x -container-port 8080
root      2581  1557  0 Oct09 ?        00:00:41 docker-containerd-shim -namespace moby -workdir /opt/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/56e9432a55304bc61d284d3a9da15272d292c1493d33d4499e481a0d64ff53e4 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
root     15630 15049  0 10:13 pts/0    00:00:00 grep --color=auto docker

Docker version:

[~]$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:23:03 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Proc Stack

[root@vm ~]# cat /proc/1549/stack
[<ffffffff810f8564>] futex_wait_queue_me+0xc4/0x120
[<ffffffff810f90d9>] futex_wait+0x179/0x280
[<ffffffff810fb1de>] do_futex+0xfe/0x5b0
[<ffffffff810fb710>] SyS_futex+0x80/0x180
[<ffffffff8169d53d>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

I’ve seen the github issue like https://github.com/docker/for-win/issues/813 and https://github.com/moby/moby/issues/12823 , but I can’t get a correct way to solve this. How can I solve this problem? Any help would be greatly appreciated.

System

Centos
Kernel: 3.10.0-514.41.1.el7 x86_64 GNU/Linux

Answer 1

Its hard to reproduce your problem. But here are some ideas, how you can solve the problem.

1. dockerd has a bad state
   Sometimes a restart via systemctl restart dockerd solves problems.
2. dockerd is corrupted
   I saw a problem of corrupted installation of docker. It leads to strange behaviors they never get 1:1 reproduced. Just reinstall docker on the maschine.
3. other
   Many other factors can produce this problem. Maybe there are problems with your centos. In one of your links, somebody discovered that a antivirus-system was the bad one (see https://github.com/docker/for-win/issues/813#issuecomment-431031402).