Why do my volumes sometime not get mounted in my Docker container when using fig?

Question

I'm seeing a strange problem in a Docker/Fig environment. My hypothesis is that it's due to a delay in mounting volumes to containers, but I'm not sure how to confirm that.

I have a container with the following:

Dockerfile

FROM busybox
MAINTAINER Dan Rumney <email>

ADD loadsnapshot.sh /loadsnapshot.sh
RUN ["chmod", "u+x", "/loadsnapshot.sh"]

VOLUME ["/snapshot"]

ENTRYPOINT ["/loadsnapshot.sh"]

loadsnapshot.sh

#!/bin/sh

if [ "$( ls -A /snapshot)" ]; then
  echo "Loading snapshot..."
  # Do stuff
else
  echo "No snapshot to load"
fi

In my fig.yml file I have:

pdsvol:
 image: busybox
 volumes:
 - "/opt/alfresco/alf_data"
 - "/data"
 - "/mysqlbackup"
 - "/ldapbackup"
loader:
 image: "docker.myregistry.com/snapshot.loader:3.5.0"
 volumes_from: 
 - pdsvol
 volumes:
 - "/opt/snapshots/my-data/:/snapshot/"

The goal here (which may be obvious) is to start up a data container (pdsvol) and then populate it with some data that's running on my machine. pdsvol is then shared by a bunch of other containers.

The way I run this is to call

fig up pdsvol

and then

fig run --rm loader

What I expect to see is

builder@beast:/opt/docker-vm$ fig run --rm loader
Loading snapshot...
... stuff ...
Removing dockervm_loader_run_1...

and, sometimes I do. However, sometimes I see:

builder@beast:/opt/docker-vm$ fig run --rm loader
No snapshot to load
Removing dockervm_loader_run_1...

I can run fig run --rm loader over and over and I will get one of the two outcomes.

My working theory was that there was some delay in mounting the volume and sometimes it happens before the ENTRYPOINT script is run and sometimes after. However, if I run:

 docker run --rm -v /opt/snapshots/my-data/:/snapshot/ busybox ls -A /snapshot

I consistently see the files I'm expecting... so this goes against that theory.

I know I could hack at loadsnapshot.sh and put in a delay and see if that helps, but I'd rather understand what's going on than kludge a fix.

Does anyone have any ideas what's going on here?

BTW: the host system is Linux, so we are using native containers here.

Update

I tried putting a 2s delay at the top of loadsnapshot.sh, but it did not help.

Update 2

I added some logging to fig to dump the configuration that is used to create the container and in every instance (fail or no), it's the same:

{
 'Env': None, 
 'Hostname': None, 
 'Entrypoint': None, 
 'Dns': None, 
 'Memory': 0, 
 'OpenStdin': True, 
 'User': None, 
 'CpuShares': None, 
 'AttachStdout': True, 
 'NetworkDisabled': False, 
 'WorkingDir': None, 
 'Cmd': None, 
 'StdinOnce': True, 
 'AttachStdin': True, 
 'Volumes': {u'/snapshot/': {}}, 
 'MemorySwap': 0, 
 'VolumesFrom': None, 
 'Tty': True, 
 'AttachStderr': True, 
 'Domainname': None, 
 'Image': 'docker.myregistry.com/snapshot.loader:3.5.0', 
 'ExposedPorts': None
}

Answer 1

I can reproduce the issue with Docker 1.4.0/Fig 1.0.1.

Rolling back to Docker 1.3.1 fixed the issue for me.

It still appears to be an open issue affecting many folks.

Although #443 is closed, there are related open issues:

• Issue #622: Since docker 1.3.0 / fig 1.0.0 volumes are not mounted when fig up'ing existing containers
• Issue #723: File volumes are broken for docker 1.4.0
• Issue #443: Local directories periodically fail to be mounted.
• Pull Request #711: Fix volumes on recreate #711