Can't run Docker container due device mapper error

Question

I just can't create and run new containers in Docker anymore. But in the same time a can run previously created containers.

When I try to do something like this:

[user@host ~ ] docker run --name=fpm-5.3 debian:jessie
2014/07/12 07:34:08 Error: Error running DeviceCreate (createSnapDevice) dm_task_run failed

From docker.log:

2014/07/12 05:57:11 POST /v1.12/containers/create?name=fpm-5.3
[f56fcb6f] +job create(fpm-5.3)
Error running DeviceCreate (createSnapDevice) dm_task_run failed
[f56fcb6f] -job create(fpm-5.3) = ERR (1)
[error] server.go:1025 Error: Error running DeviceCreate (createSnapDevice) dm_task_run failed
[error] server.go:90 HTTP Error: statusCode=500 Error running DeviceCreate (createSnapDevice) dm_task_run failed

dmsetup status

docker-8:1-1210426-pool: 0 209715200 thin-pool 352 2510/524288 205173/1638400 - ro discard_passdown queue_if_no_space 

But they are a lot of free space on disk.

dmsetup info

Name:              docker-8:1-1210426-pool
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      1
Major, minor:      252, 0
Number of targets: 1

docker info

Containers: 4
Images: 65
Storage Driver: devicemapper
 Pool Name: docker-8:1-1210426-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 12823.3 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 9.9 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.14.4

docker version

Client version: 1.0.0
Client API version: 1.12
Go version (client): go1.2.2
Git commit (client): 63fe64c
Server version: 1.0.0
Server API version: 1.12
Go version (server): go1.2.2
Git commit (server): 63fe64c

Answer 1

The following is for a Fedora/RHEL system, so you'll need to adjust for Debian...

# systemctl stop docker.service
# thin_check /var/lib/docker/devicemapper/devicemapper/metadata

If there were no errors then proceed with:

# thin_check --clear-needs-check-flag /var/lib/docker/devicemapper/devicemapper/metadata
# systemctl start docker.service
# docker run --name=fpm-5.3 debian:jessie

---

To install thin_check command, run:

# apt-get install -y thin-provisioning-tools

Answer 2

When the docker partition filled and docker would no longer start after reboot, I encountered this:

# thin_check /var/lib/docker/devicemapper/devicemapper/metadata
examining superblock
examining devices tree
  missing devices: [0, -]
    bad checksum in btree node
examining mapping tree
  thin device 72 is missing mappings [137494, 137594]
    bad checksum in btree node
  thin device 72 is missing mappings [137721, -]
    bad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree node

I was able to repair with this procedure:

# thin_dump -r /var/lib/docker/devicemapper/devicemapper/metadata -o /tmp/metadata.xml
# thin_restore -i /tmp/metadata.xml -o /var/lib/docker/devicemapper/devicemapper/metadata

Answer 3

In our case just need to start and stop the service.

Depending on your system you can run:

service docker stop
service docker start

or:

sudo systemctl stop docker.service
sudo systemctl start docker.service

or:

sudo /etc/init.d/docker restart

Answer 4

I had the same problem and was not able to fix it. I found something promising at: http://grokbase.com/t/gg/docker-user/1563fzdtm7/docker-docker-runs-out-of-space-when-trying-to-create-a-new-image 'The default docker storage driver allocates 10GB storage blocks for your images. Move to overlayfs and avoid this entirely. In the command that starts your docker daemon just add "-s overlay" '

this fixed my problem.

Answer 5

I've been fighting this issue with Debian 8.2. I had other problems because I run a 4.3.3 Kernel (default is 3.16) with grsec.

Despite the GRSEC issues (mount & chmod denied) I was able to run docker and create some image and container.

Then, I would reboot and docker would just spit out the error. I ran thin_check and what I found was this:

• metadata was fine (I don't use a thin pool right now since docker keeps on telling me that my thin pool isn't a thin pool...)
• data was corrupted.

Tried to repair it but thin_restore crashes.

I realized that: docker daemon ... was working but can't be stopped with systemctl stop docker.service. It says that the service is stopped but the daemon is still in memory (ps -elf | grep docker)

To fix the problem I had to change the DOCKER_STORAGE_OPTIONS in /etc/default/docker

rm -rf /var/lib/docker
reboot

At boot time, the service starts. docker info

Shows the info as expected. Built an image. Rebooted, the service starts fine again. I think that basically the docker daemon can't be stopped and killing it with a:

kill <pid>

Causes the data file to be corrupted and therefore the checksum won't match.

Bottom line is don't mix and match docker.service and docker daemon. At least on Debian/Ubuntu.