Docker container's sshfs mount freezes, but only when mounted by Python

Question

I have a development laptop (Mint 19.3), and a test server (Ubuntu 18.04.4 LTS).

The laptop is Docker version 19.03.5, build 633a0ea838, the server is Docker version 19.03.12, build 48a66213fe

I'm running Python 3.6 code inside the container, which uses subprocess (code below) to create an sshfs mount to a third server, after which the python code walks through the mounted directory.

Everything works fine on my development laptop. But on the server, the directory mounts (and is seen with the mount command) however cd'ing into the directory just hangs, and the Python code's subsequent walk just hangs. (NOTE: The python code never crashes or errors out. It just hangs forever.)

HOWEVER, if I manually use the same sshfs command at the container's command line, the directory works fine.

I'm at a loss as to how to troubleshoot this.

===2020-09-25 UPDATE===

OK. Since the Python code uses subprocess, the sshfs mount is obviously available to any terminal windows that wants to use it.

I have tried accessing the mount from a new terminal window inside the container, but when I cd to the mount - the window just freezes.

Well, I left everything sitting overnight - and now when I try to cd into the mount ... it works. It's like the mount has to sit for hours before it will work.

Any ideas?

Python code

@target_dir.setter    
def target_dir(self, value):
    if value is None:
        _tmp = tempfile.mkdtemp()
        self._TARGETDIR = _tmp

    else:
        self._TARGETDIR = value


def mount(self):
    # sshfs username@server-worker01:/test_project/ /mnt/test3 -o IdentityFile=/home/username/.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
    command = "sshfs {U}@{S}:{RD} {LD}".format(
        U  = self.user, 
        S  = self.server, 
        RD = self.remote_dir, 
        LD = self.target_dir)
    command += " -o IdentityFile={IDF},auto_cache,reconnect,transform_symlinks,follow_symlinks,allow_other".format(IDF = key)

    cp = subprocess.getoutput(command)
    if cp != "":
        err = "Something appears to be wrong with the mount. ({})".format(cp)
        raise RuntimeError(err) 

Mounting manually inside container works

root@328100dd78be:/# mkdir /mnt/test
root@328100dd78be:/# sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=/etc/ssl/certs/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
root@328100dd78be:/# cd /mnt/test
root@328100dd78be:/mnt/test# ls -la
total 40
drwxr-s--- 1 16543 41500  210 Aug  6 07:07 .
drwxr-xr-x 1 root  root  4096 Sep 24 14:27 ..
drwxr-s--- 1 16543 41500  187 Jul 27 11:16 configFiles
(...snip...)
drwx--S--- 1 16543 41500  104 May  6 14:03 submission

But if I let the code mount it, the dir hangs

root@328100dd78be:# ODCFFileCrawl.py 
2020-09-24 14:31:16,722 - root - INFO - logger started! with:{'loglevel': '10'}
sshfs username@server-worker01:/test_project/ /tmp/tmp0wwjcg_6 -o IdentityFile=/home/username/.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks,allow_other,StrictHostKeyChecking=no

2020-09-24 14:31:16,797 - root - WARNING - Mounted remote filesystem 'server-worker01:/test_project/' to '/tmp/tmp7fig76gx'.

(change to new console)
root@328100dd78be:/# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/RWED5GN6GYRRGECDFN4LLIVVU7:/var/lib/docker/overlay2/l/7MTLBYSZ4ARQHKQ65TYBY5VSHW:/var/lib/docker/overlay2/l/EJX3L6YHRYB5IICXKAIH4PHT5I:/var/lib/docker/overlay2/l/AFJWYKSAAP6E7RMIU3H2PUMRSN:/var/lib/docker/overlay2/l/CU3AWZFBMNMQLQSBUQSVCHOKWC:/var/lib/docker/overlay2/l/RBJZT34EXSVZD3UYORTLM6L5UX:/var/lib/docker/overlay2/l/OJ7YFP5Z4SEETA2UVO35HQ2K53:/var/lib/docker/overlay2/l/CBU6F6UI47VHQ2BW5NMGJE64UA:/var/lib/docker/overlay2/l/62ZEX3XIFI7VYTLYYJX6XNLJTM:/var/lib/docker/overlay2/l/NBRXSW53NRLL5KRGYQ2BDL2ICZ:/var/lib/docker/overlay2/l/W5DJJWYP2VAYHTNDDHPY2KK2N3:/var/lib/docker/overlay2/l/ZOR2MOYW2NICOX3YVUUY6VXKVW:/var/lib/docker/overlay2/l/WLZEM73R55XKTPNVFCC5TRDCKT:/var/lib/docker/overlay2/l/PARGLNJTE4CM7PTTS5UOLXAIOC:/var/lib/docker/overlay2/l/JDA4NCWU5RN5YNVC63USV4O3VC:/var/lib/docker/overlay2/l/CEGTXJOQH65FTIZLAJNTAPF24R:/var/lib/docker/overlay2/l/B4VV7H463RL3THLES637O57DUO:/var/lib/docker/overlay2/l/KCENZYGKRWT7BRZC2HRUKSB4C5:/var/lib/docker/overlay2/l/EOYAIHOSSFBSX7X5HEWWZQGXS4:/var/lib/docker/overlay2/l/T2ZFALNJHY37UVCYHXD6GC36R5:/var/lib/docker/overlay2/l/4ACYIA6XBKYGFV5BUHYWTOALAY:/var/lib/docker/overlay2/l/PWZLNT3WAZPWPPYFOHZQ2SBHMG:/var/lib/docker/overlay2/l/W7UH56VKQDQH65GOWIGCYAMR3U,upperdir=/var/lib/docker/overlay2/9c2385aa1221d4fbcac321bfb7862dae23677bdf594ddea803315144b5124617/diff,workdir=/var/lib/docker/overlay2/9c2385aa1221d4fbcac321bfb7862dae23677bdf594ddea803315144b5124617/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/vda1 on /etc/resolv.conf type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/hostname type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/hosts type ext4 (rw,relatime,data=ordered)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/vda1 on /etc/ssl/certs type ext4 (rw,relatime,data=ordered)
/dev/vdb1 on /opt/vep/.vep type ext4 (rw,relatime,data=ordered)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime)
tmpfs on /sys/firmware type tmpfs (ro,relatime)
username@server-worker01:/test_project/ on /mnt/test type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0)
username@server-worker01:/test_project/ on /tmp/tmp7fig76gx type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

root@328100dd78be:# cd /tmp/tmp7fig76gx
(hangs forever)

===2020-12-04 UPDATE FOR @colidyre===

Try this...

mount.py

import tempfile
import subprocess

# target_dir = The local full path to which the remote directory will be mounted
target_dir = tempfile.mkdtemp() 
user = "<Your username on the remote server >"
server = "<The server name for the remote server>"
remote_dir = "<Full path of the remote directory>"
key = "<Full path of the ssh private key for the remote server>"
options = " -o IdentityFile={IDF},auto_cache,reconnect,transform_symlinks,follow_symlinks,allow_other".format(IDF = key)

command = "sshfs {U}@{S}:{RD} {LD}".format(
    U  = user, 
    S  = server, 
    RD = remote_dir, 
    LD = target_dir)
command += options
cp = subprocess.getoutput(command)
if cp != "":
    err = "Something appears to be wrong with the mount. ({})".format(cp)
    raise RuntimeError(err) 

Code for Dockerfile NOTE: This is just the factory VEP Docker container from https://hub.docker.com/r/ensemblorg/ensembl-vep/ with lines added in the middle to copy the mount.py command into /usr/bin, and the entrypoint chnaged to mount.py. See section ...

##########################################
# Setup environment for mount.py
##########################################

Dockerfile

###################################################
# Stage 1 - docker container to build ensembl-vep #
###################################################
FROM ubuntu:18.04 as builder

# Update aptitude and install some required packages
# a lot of them are required for Bio::DB::BigFile
RUN apt-get update && apt-get -y install \
    build-essential \
    apt-utils \
      python3 \
      python3-pip \
    git \
    libpng-dev \
    zlib1g-dev \
    libbz2-dev \
    liblzma-dev \
    perl \
    perl-base \
    unzip \
    wget
#    wget && \
#    rm -rf /var/lib/apt/lists/*

# Setup VEP environment
ENV OPT /opt/vep
ENV OPT_SRC $OPT/src
ENV HTSLIB_DIR $OPT_SRC/htslib
ENV BRANCH release/101

# Working directory
WORKDIR $OPT_SRC

# Clone/download repositories/libraries
RUN if [ "$BRANCH" = "master" ]; \
    then export BRANCH_OPT=""; \
    else export BRANCH_OPT="-b $BRANCH"; \
    fi && \
    # Get ensembl cpanfile in order to get the list of the required Perl libraries
    wget -q "https://raw.githubusercontent.com/Ensembl/ensembl/$BRANCH/cpanfile" -O "ensembl_cpanfile" && \
    # Clone ensembl-vep git repository
    git clone $BRANCH_OPT --depth 1 https://github.com/Ensembl/ensembl-vep.git && chmod u+x ensembl-vep/*.pl && \
    # Clone ensembl-variation git repository and compile C code
    git clone $BRANCH_OPT --depth 1 https://github.com/Ensembl/ensembl-variation.git && \
    mkdir var_c_code && \
    cp ensembl-variation/C_code/*.c ensembl-variation/C_code/Makefile var_c_code/ && \
    rm -rf ensembl-variation && \
    chmod u+x var_c_code/* && \
    # Clone bioperl-ext git repository - used by Haplosaurus
    git clone --depth 1 https://github.com/bioperl/bioperl-ext.git && \
    # Download ensembl-xs - it contains compiled versions of certain key subroutines used in VEP
    wget https://github.com/Ensembl/ensembl-xs/archive/2.3.2.zip -O ensembl-xs.zip && \
    unzip -q ensembl-xs.zip && mv ensembl-xs-2.3.2 ensembl-xs && rm -rf ensembl-xs.zip && \
    # Clone/Download other repositories: bioperl-live is needed so the cpanm dependencies installation from the ensembl-vep/cpanfile file takes less disk space
    ensembl-vep/travisci/get_dependencies.sh && \
    # Only keep the bioperl-live "Bio" library
    mv bioperl-live bioperl-live_bak && mkdir bioperl-live && mv bioperl-live_bak/Bio bioperl-live/ && rm -rf bioperl-live_bak && \
    ## A lot of cleanup on the imported libraries, in order to reduce the docker image ##
    rm -rf Bio-HTS/.??* Bio-HTS/Changes Bio-HTS/DISCLAIMER Bio-HTS/MANIFEST* Bio-HTS/README Bio-HTS/scripts Bio-HTS/t Bio-HTS/travisci \
           bioperl-ext/.??* bioperl-ext/Bio/SeqIO bioperl-ext/Bio/Tools bioperl-ext/Makefile.PL bioperl-ext/README* bioperl-ext/t bioperl-ext/examples \
           ensembl-vep/.??* ensembl-vep/docker \
           ensembl-xs/.??* ensembl-xs/TODO ensembl-xs/Changes ensembl-xs/INSTALL ensembl-xs/MANIFEST ensembl-xs/README ensembl-xs/t ensembl-xs/travisci \
           htslib/.??* htslib/INSTALL htslib/NEWS htslib/README* htslib/test && \
    # Only keep needed kent-335_base libraries for VEP - used by Bio::DB::BigFile (bigWig parsing)
    mv kent-335_base kent-335_base_bak && mkdir -p kent-335_base/src && \
    cp -R kent-335_base_bak/src/lib kent-335_base_bak/src/inc kent-335_base_bak/src/jkOwnLib kent-335_base/src/ && \
    cp kent-335_base_bak/src/*.sh kent-335_base/src/ && \
    rm -rf kent-335_base_bak

# Setup bioperl-ext
WORKDIR bioperl-ext/Bio/Ext/Align/
RUN perl -pi -e"s|(cd libs.+)CFLAGS=\\\'|\$1CFLAGS=\\\'-fPIC |" Makefile.PL

# Install htslib binaries (for 'bgzip' and 'tabix')
# htslib requires the packages 'zlib1g-dev', 'libbz2-dev' and 'liblzma-dev'
WORKDIR $HTSLIB_DIR
RUN make install && rm -f Makefile *.c

# Compile Variation LD C scripts
WORKDIR $OPT_SRC/var_c_code
RUN make && rm -f Makefile *.c


###################################################
# Stage 2 - docker container to build ensembl-vep #
###################################################
FROM ubuntu:18.04

# Update aptitude and install some required packages
# a lot of them are required for Bio::DB::BigFile
RUN apt-get update && apt-get -y install \
    build-essential \
    python3 \
    python3-pip \
    git \
    cpanminus \
    curl \
    libmysqlclient-dev \
    libpng-dev \
    libssl-dev \
    zlib1g-dev \
    libbz2-dev \
    liblzma-dev \
    locales \
    openssl \
    perl \
    perl-base \
    unzip \
    sshfs \
    vim && \
    apt-get -y purge manpages-dev && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# gitpython must be installed before setup.py is run
RUN pip3 install gitpython

##########################################
# Setup environment for mount.py
##########################################

# Takes the local FUSE conf as the base. This should be properly configured
#  with "user_allow_other" enabled
COPY mount.py /usr/bin/mount.py
RUN chown root:root /usr/bin/mount.py
RUN chmod 755 /usr/bin/mount.py

# To hold the GIT downloads prior to install
RUN mkdir /src
RUN chmod 777 /src

##########################################
# Setup VEP environment
##########################################

ENV OPT /opt/vep
ENV OPT_SRC $OPT/src
ENV PERL5LIB_TMP $PERL5LIB:$OPT_SRC/ensembl-vep:$OPT_SRC/ensembl-vep/modules
ENV PERL5LIB $PERL5LIB_TMP:$OPT_SRC/bioperl-live
ENV KENT_SRC $OPT/src/kent-335_base/src
ENV HTSLIB_DIR $OPT_SRC/htslib
ENV MACHTYPE x86_64
ENV DEPS $OPT_SRC
ENV PATH $OPT_SRC/ensembl-vep:$OPT_SRC/var_c_code:$PATH
ENV LANG_VAR en_US.UTF-8

# Create vep user
RUN useradd -r -m -U -d "$OPT" -s /bin/bash -c "VEP User" -p '' vep && usermod -a -G sudo vep && mkdir -p $OPT_SRC
USER vep

# Copy downloaded libraries (stage 1) to this image (stage 2)
COPY --chown=vep:vep --from=builder $OPT_SRC $OPT_SRC
#############################################################

# Change user to root for the following complilations/installations
USER root

# Install bioperl-ext, faster alignments for haplo (XS-based BioPerl extensions to C libraries)
WORKDIR $OPT_SRC/bioperl-ext/Bio/Ext/Align/
RUN perl Makefile.PL && make && make install && rm -f Makefile*

# Install ensembl-xs, faster run using re-implementation in C of some of the Perl subroutines
WORKDIR $OPT_SRC/ensembl-xs
RUN perl Makefile.PL && make && make install && rm -f Makefile* cpanfile

WORKDIR $OPT_SRC
# Install/compile more libraries
RUN ensembl-vep/travisci/build_c.sh && \
    # Remove unused Bio-DB-HTS files
    rm -rf Bio-HTS/cpanfile Bio-HTS/Build.PL Bio-HTS/Build Bio-HTS/_build Bio-HTS/INSTALL.pl && \
    # Install ensembl perl dependencies (cpanm)
    cpanm --installdeps --with-recommends --notest --cpanfile ensembl_cpanfile . && \
    cpanm --installdeps --with-recommends --notest --cpanfile ensembl-vep/cpanfile . && \
    # Delete bioperl and cpanfiles after the cpanm installs as bioperl will be reinstalled by the INSTALL.pl script
    rm -rf bioperl-live ensembl_cpanfile ensembl-vep/cpanfile && \
    # Configure "locale", see https://github.com/rocker-org/rocker/issues/19
    echo "$LANG_VAR UTF-8" >> /etc/locale.gen && locale-gen en_US.utf8 && \
    /usr/sbin/update-locale LANG=$LANG_VAR && \
    # Copy htslib executables. It also requires the packages 'zlib1g-dev', 'libbz2-dev' and 'liblzma-dev'
    cp $HTSLIB_DIR/bgzip $HTSLIB_DIR/tabix $HTSLIB_DIR/htsfile /usr/local/bin/

ENV LC_ALL $LANG_VAR
ENV LANG $LANG_VAR

# Switch back to vep user
USER vep
ENV PERL5LIB $PERL5LIB_TMP

# Final steps for VEP
WORKDIR $OPT_SRC/ensembl-vep
# Update bash profile
RUN echo >> $OPT/.profile && \
    echo PATH=$PATH:\$PATH >> $OPT/.profile && \
    echo export PATH >> $OPT/.profile && \
    # Run INSTALL.pl and remove the ensemb-vep tests and travis
    ./INSTALL.pl -a a -l -n && rm -rf t travisci .travis.yml

ENTRYPOINT ["mount.py"]
# CMD tail -f /dev/null

Answer 1

I am assuming you want to mount some server's directory to container's filesystem using SSHFS. You could add that instruction to the Dockerfile:

FROM ubuntu:18.04
RUN apt-get update && apt-get -y install stuff
COPY sshprivkey .ssh/sshprivkey
RUN mkdir /mnt/test
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks

You can add other stuff that you have in your Dockerfile below this. You should add sshprivkey file to your current directory.

Now if you build it:

docker build -t your_desired_image_name .

And then run a container using that image and check whether it worked:

docker run --privileged -it your_desired_image_name 

root@container_id$ ls /mnt/test

Remember to add that --privileged flag for it to work. I found it out from here.

Answer 2

Server Directory/Directories can be mounted directly to container's filesystem using SSHFS utility.

Dockerfile

FROM ubuntu:18.04
RUN apt-get update
COPY sshprivkey .ssh/sshprivkey
RUN mkdir /mnt/test
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks

Container is created using ubuntu:18.04 base image.

FROM ubuntu:18.04

Update package list. This command will be changed according to base image. here command can be updated to install required packages

RUN apt-get update

Copied sshprivkey to use with sshfs for permissions to mount server directly to container

COPY sshprivkey .ssh/sshprivkey

Where server directory should be mounted for that test directory is created

RUN mkdir /mnt/test

Run sshfs command with sshprivkey to mount server directory in previous created test directory

RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks

To Build image from Dockerfile in current directory following command is used:

docker build -t your_desired_image_name .

Running interactive session container from previously created image following command it used:

docker run --cap-add SYS_ADMIN -it your_desired_image_name 

After container interactive session starts we can check that if it worked:

root@container_id$ ls /mnt/test

Here i have used --cap-add to give container permission to mount directory

Note: It is good to add limited permission as required using --cap-add and not using --privileged until nothing else works for security purpose has it is not good to give too much permissions to that container

You can read more about docker permissions here