Why might the host behave more deterministic than a docker container?

Question

We use Docker to well define the build environment and help with deterministic builds but on my machine I get a tiny change in the build results using Docker but not when not using Docker.

I did pretty extensive testing and am out of ideas :(

I tested on the following systems:

• A: My new PC without Docker
• AD1: My new PC with Docker, using our Dockerfile based on ubuntu:18.04 compiled "a year ago"
• AD2: My new PC with Docker, using our Dockerfile based on ubuntu:19:10 compiled now
• B: My laptop (that I had copied the disk from to my new PC) without Docker
• BD: My laptop with Docker
• CD1: Co-worker's laptop with Docker, using our Dockerfile based on ubuntu:18.04 compiled "a year ago"
• CD2: Co-worker's laptop with Docker, using our Dockerfile based on ubuntu:19:10 compiled now
• DD: A Digital Ocean VPS with our Dockerfile based on ubuntu:18.04 compiled now

In all scenarios we got either of two build results I will name variant X and Y.

• We got variant X using A, B, CD1, CD2 and DD.
• We got variant Y using AD1, AD2 and BD.

The issue keeps being 100% reproducible since several releases of our Android app. It did not go away when I updated my Docker from 19.03.6 to 19.03.8 to match my co-worker's version. We both had Ubuntu 19.10 back then and I now keep getting the issue with Ubuntu 20.04.

I always freshly cloned our project into a new folder, used disorderfs to eliminate file system sorting issues and mounted the folder into the docker container.

I doubt it's relevant but we are using this Dockerfile:

FROM ubuntu:18.04

RUN dpkg --add-architecture i386 && \
    apt-get update -y && \
    apt-get install -y software-properties-common && \
    apt-get update -y && \
    apt-get install -y wget \
            openjdk-8-jre-headless=8u162-b12-1 \
            openjdk-8-jre=8u162-b12-1 \
            openjdk-8-jdk-headless=8u162-b12-1 \
            openjdk-8-jdk=8u162-b12-1 \
            git unzip && \
    rm -rf /var/lib/apt/lists/* && \
    apt-get autoremove -y && \
    apt-get clean

# download and install Android SDK
ARG ANDROID_SDK_VERSION=4333796
ENV ANDROID_HOME /opt/android-sdk
RUN mkdir -p /opt/android-sdk && cd /opt/android-sdk && \
    wget -q https://dl.google.com/android/repository/sdk-tools-linux-${ANDROID_SDK_VERSION}.zip && \
    unzip *tools*linux*.zip && \
    rm *tools*linux*.zip && \
    yes | $ANDROID_HOME/tools/bin/sdkmanager --licenses

Also here are the build instructions I run and get different results. The diff itself is can be found here.

---

Edit: I also filed it as a bug on the docker repo.

Answer 1

Docker is not fully architecture-independent. For different architecture you may have more or less minute differences. Usually it should not affect anything important but may change some optimisation decisions of a compiler and such things. It is more visible if you try very different CPUs like AMD64 vs ARM. For Java it should not matter but it seems that at least sometimes it matters.

Another thing is network and DNS. When you do apt-get, wget and other such things then it downloads code or binary from network. It may differ depending on which DNS you use (which may lead to different server or different repo url) and there can be some minute differences. Theoretically there should be no difference but practically there can be difference sometimes like when they roll out new version and it's visible only on some nodes or something bad happened or you have some cache/proxy in between and connect through that and it caches etc.

Also the latter can create differences that appear in time. Like app is compiled on one month and someone tries to verify few weeks or months later and apt-get installs other versions of libraries and in effect there are minute differences.

I'm not sure which applies here but I have some ideas:

• may try to make some small changes to the app so in effect it will again build same on most of popular CPU's, do extensive testing, and then list architectures on which it can be verified

• make verification process a little more complex and non-free so users should have to run a server instance (on AWS or Google or Azure or Rackspace or other) with specified architecture and build and verify there - may try and specify on which types of machines exacly result will be the same and what are minimal requirements (as it may or may not run on free-plan instances)

• check diff of created images content (not only apk but full system image), maybe there will be something important that differs between docker images on different machines producing different results

• try to find as small as possible initial image and don't allow apt-get or other things automatically install dependencies with newest version but specify all dependencies and their versions