How do I use command substition in Dockerfile

Question

In my Dockerfile I need to use command substition to add some environment variables. I want to set

ENV PYTHONPATH /usr/local/$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')

but it doesn't work. The result is

foo@bar:~$ echo $PYTHONPATH
/usr/local/$(python3 -c from distutils import sysconfig; print(sysconfig.get_python_lib()))

What's wrong?

Answer 1

What went wrong

The $( ... ) command substitution you attempted is for Bash, whereas the Dockerfile is not Bash. So docker doesn't know what to do with that, it's just plain text to docker, docker just spews out what you wrote as-is.

Recommendation

To avoid hard-coding values into a Dockerfile, and instead, to dynamically change a setting or custom variable as PYTHONPATH during the build, perhaps the ARG ... , --build-arg docker features might be most helpful, in combination with ENV ... to ensure it persists.

Within your Dockerfile:

ARG PYTHON_PATH_ARG

ENV PYTHONPATH ${PYTHON_PATH_ARG}

In Bash where you build your container:

python_path="/usr/local$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"

docker build --build-arg PYTHON_PATH_ARG=$python_path .

Explanation

According to documentation, ARG:

The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag.

So, in Bash we first:

python_path="/usr/local$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"

• $(...) Bash command substitution is used to dynamically put together a Python path value
• this value is stored temporarily in a Bash variable $python_path for clarity

docker build --build-arg PYTHON_PATH_ARG=$python_path .

• Bash variable $python_path value is passed to docker's --build-arg PYTHON_PATH_ARG

Within the Dockerfile:

ARG PYTHON_PATH_ARG

• so PYTHON_PATH_ARG stores the value from --build-arg PYTHON_PATH_ARG...

ARG variables are not equivalent to ENV variables, so we couldn't merely do ARG PYTHONPATH and be done with it. According to documentation about Using arg variables:

ARG variables are not persisted into the built image as ENV variables are.

So finally:

ENV PYTHONPATH ${PYTHON_PATH_ARG}

• We use Dockerfile's ${...} convention to get the value of PYTHON_PATH_ARG, and save it to your originally named PYTHONPATH environment variable

Differences from original code

You originally wrote:

ENV PYTHONPATH /usr/local/$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')

I re-wrote the Python path finding portion as a Bash command, and tested on my machine:

$ python_path="/usr/local/$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"

$ echo $python_path
/usr/local//usr/lib/python3/dist-packages

Notice there is a double forward slash ... local//usr ... , not sure if that will break anything for you, depends on how you use it in your code.

Instead, I changed it to:

$ python_path="/usr/local$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"

Result:

$ echo $python_path
/usr/local/usr/lib/python3/dist-packages

So this new code will have no double forward slashes.

Answer 2

You should use ARG if possible. But sometimes you really need to use command substitution for a dynamic variable. As long as you put all the commands in the same RUN statement, then you can still access the value.

RUN foo=$(date) && \
    echo $foo