Docker follow symlink outside context

Question

That is not possible and will not be implemented. Please have a look at the discussion on github issue #1676:

We do not allow this because it's not repeatable. A symlink on your machine is the not the same as my machine and the same Dockerfile would produce two different results. Also having symlinks to /etc/paasswd would cause issues because it would link the host files and not your local files.

If anyone still has this issue I found a very nice solution on superuser.com:

https://superuser.com/questions/842642/how-to-make-a-symlinked-folder-appear-as-a-normal-folder

It basically suggests using tar to dereference the symlinks and feed the result into docker build:

$ tar -czh . | docker build -

One possibility is to run the build in the parent directory, with:

$ docker build [tags...] -f dir1/Dockerfile .

(Or equivalently, in child directory,)

$ docker build  [tags...] -f Dockerfile ..

The Dockerfile will have to be configured to do copy/add with appropriate paths. Depending on your setup, you might want a .dockerignore in the parent to leave out things you don't want to be put into the context.

I know that it breaks portability of docker build, but you can use hard links instead of symbolic:

ln /some/file ./hardlink

I just had to solve this issue in the same context. My solution is to use hierarchical Docker builds. In other words:

parent_dir
  - common_files
    - Dockerfile
    - file.txt

- dir1
    - Dockerfile (FROM common_files:latest)

The disadvantage is that you have to remember to build common_files before dir1. The advantage is that if you have a number of dependant images then they are all a bit smaller due to using a common layer.

I got frustrated enough that I made a small NodeJS utility to help with this: file-syncer

Given the existing directory structure:

parent_dir
    - common_files
        - file.txt
    - my-app
        - Dockerfile
        - common_files -> symlink to ../common_files

Basic usage:

cd parent_dir

// starts live-sync of files under "common_files" to "my-app/HardLinked/common_files"
npx file-syncer --from common_files --to my-app/HardLinked

Then in your Dockerfile:

[regular commands here...]

# have docker copy/overlay the HardLinked folder's contents (common_files) into my-app itself
COPY HardLinked /

Q/A

• How is this better than just copying parent_dir/common_files to parent_dir/my-app/common_files before Docker runs?

That would mean giving up the regular symlink, which would be a loss, since symlinks are helpful and work fine with most tools. For example, it would mean you can't see/edit the source files of common_files from the in-my-app copy, which has some drawbacks. (see below)

• How is this better than copying parent_dir/common-files to parent_dir/my-app/common_files_Copy before Docker runs, then having Docker copy that over to parent_dir/my-app/common_files at build time?

There are two advantages:

1. file-syncer does not "copy" the files in the regular sense. Rather, it creates hard links from the source folder's files. This means that if you edit the files under parent_dir/my-app/HardLinked/common_files, the files under parent_dir/common_files are instantly updated, and vice-versa, because they reference the same file/inode. (this can be helpful for debugging purposes and cross-project editing [especially if the folders you are syncing are symlinked node-modules that you're actively editing], and ensures that your version of the files is always in-sync/identical-to the source files)
2. Because file-syncer only updates the hard-link files for the exact files that get changed, file-watcher tools like Tilt or Skaffold detect changes for the minimal set of files, which can mean faster live-update-push times than you'd get with a basic "copy whole folder on file change" tool would.

• How is this better than a regular file-sync tool like Syncthing?

Some of those tools may be usable, but most have issues of one kind or another. The most common one is that the tool either cannot produce hard-links of existing files, or it's unable to "push an update" for a file that is already hard-linked (since hard-linked files do not notify file-watchers of their changes automatically, if the edited-at and watched-at paths differ). Another is that many of these sync tools are not designed for instant responding, and/or do not have run flags that make them easy to use in restricted build tools. (eg. for Tilt, the --async flag of file-syncer enables it to be used in a local(...) invokation in the project's Tiltfile)

instead of using simlinks it is possible to solve problem administratively by just moving files from sites_available to sites_enabled instead of copying or making simlinks

so your site config will be in one copy only in site_available folder if it stopped or something or in sites_enabled if it should be used