standard_init_linux.go:219: exec user process caused: no such file or directory

Question

I am trying to move my rust server from Heroku to Google Cloud or AWS. Even though I like the simplicity of having a git push build and deploy to Heroku with just a buildpack specified, the service is not cost effective for me.

I identified Google Cloud Run and AWS Elastic Beanstalk as potential alternatives.

First, I need to build a docker image with a static binary.

Thus, I added this Dockerfile:

FROM rust AS build
WORKDIR /usr/src
RUN rustup target add x86_64-unknown-linux-musl
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential git clang llvm-dev libclang-dev libssl-dev pkg-config libpq-dev musl-tools brotli
RUN USER=root cargo new loxe-api
WORKDIR /usr/src/loxe-api
COPY Cargo.toml Cargo.lock ./
COPY data ./data
COPY migrations ./migrations
ENV RUSTFLAGS="-C target-feature=+crt-static" # this should be set by the target, but just to be sure
RUN cargo build --release
COPY src ./src
ENV PKG_CONFIG_ALLOW_CROSS=1
ENV OPENSSL_INCLUDE_DIR="/usr/include/openssl"
RUN cargo install --target x86_64-unknown-linux-musl --path .

FROM scratch
COPY --from=build /usr/local/cargo/bin/loxe-api .
COPY data ./data
COPY migrations ./migrations
USER 1000
CMD ["./loxe-api"]

The image gets build without errors, but if I run it via docker run, I'm getting this error:

standard_init_linux.go:219: exec user process caused: no such file or directory

By replacing the base image of the final step with rust, I checked whether the binary and other files are in fact in the image. They are, I can see them via ls, but I also cannot execute loxe-api when entering the shell.

dockerd logs this:

INFO[2020-07-05T13:04:42.368119033-07:00] shim containerd-shim started                  address=/containerd-shim/bf85e63468a9c1b3b9fe418b5a186673f0609bfff20c4832789ae87433e82473.sock debug=false pid=27032
INFO[2020-07-05T13:04:42.913438974-07:00] shim reaped                                   id=8cadeee800649ceca8a52d9a75cc9071b923d01a5d2a37497bf8b9a6e719267a
INFO[2020-07-05T13:04:42.925442900-07:00] ignoring event                                module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

Here's dependencies-section of the Cargo.toml:

[dependencies]
actix = "0.9"
actix-cors = "0.2"
actix-identity = "0.2"
actix-multipart = "0.2"
actix-rt = "1.0"
actix-web = "2.0"
argonautica = "0.2"
brotli = "3.3"
bytes = { version = "0.5", features = ["serde"]  }
chrono = { version = "0.4", features = ["serde"]  }
derive_more = "0.99"
diesel = { version = "1.4", features = ["postgres", "uuidv07", "r2d2", "chrono", "serde_json"]  }
diesel_migrations = "1.4"
dotenv = "0.15"
env_logger = "0.7"
futures = "0.3"
indexmap = { version = "1.3", features = ["serde-1"] }
lazy_static = "1.4"
log = "0.4"
openssl = { version = "0.10", features = ["vendored"] }
openssl-probe = "0.1.2"
percent-encoding = "2.1"
r2d2 = "0.8"
rand = "0.7"
redis = "0.15"
rusoto_core = { version = "0.44" }
rusoto_s3 = { version = "0.44" }
sanitize-filename = "0.2"
sendgrid = { version = "0.10", features = ["rustls"] }
serde = { version = "1.0", features = ["derive"]  }
serde_json = "1.0"
stripe-rust = "0.12"
uuid = { version = "0.8", features = ["serde", "v4"]  }
wana_kana = "2.0"

Further investigation from inside the container:

$ ldd /loxe-api
        linux-vdso.so.1 (0x00007ffcc219d000)
        libpq.so.5 => /usr/lib/x86_64-linux-gnu/libpq.so.5 (0x00007f2d3792d000)
        libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f2d3789b000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f2d375b2000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f2d37565000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f2d37511000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2d374f0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2d3732d000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2d37328000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f2d37248000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f2d37214000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f2d3720e000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f2d371ff000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f2d371f6000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f2d371dc000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f2d371cb000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f2d371ae000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f2d37002000)
        /lib/ld64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007f2d37983000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f2d36ed1000)
        libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f2d36eb2000)
        libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f2d36d2e000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f2d36b1b000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f2d36ae3000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f2d36aaa000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f2d36a25000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f2d36a1b000)

I also tried to build and execute from my host system. I ran cargo run --target x86_64-unknown-linux-musl --release which built the binary, but ultimately resulted in this error:

error: could not execute process target/x86_64-unknown-linux-musl/release/loxe-api (never executed)

I checked for the existence of the binary via du -h target/x86_64-unknown-linux-musl/release/loxe-api:

35M     target/x86_64-unknown-linux-musl/release/loxe-api

Further unsuccessful attempts with out-of-the-box and some modified cross, clux/muslrust and emk/rust-musl-builder repos have been made.

Building and running a new cargo default project with the x86_64-unknown-linux-musl target works. I assume, libclang, brotli, or some of the argonautica libraries may make this not work.

---

Here's a simplified Dockerfile that yields the same result.

FROM rust AS build
WORKDIR /usr/src

RUN rustup target add x86_64-unknown-linux-musl
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential git clang llvm-dev libclang-dev libssl-dev pkg-config libpq-dev musl-tools brotli

RUN USER=root cargo new loxe-api
WORKDIR /usr/src/loxe-api
COPY Cargo.toml Cargo.lock ./
COPY data ./data
COPY migrations ./migrations
COPY src ./src
ENV PKG_CONFIG_ALLOW_CROSS=1
ENV OPENSSL_INCLUDE_DIR="/usr/include/openssl"
ENV RUSTFLAGS="-C target-feature=+crt-static"
RUN cargo install --target x86_64-unknown-linux-musl --path .

FROM debian
COPY --from=build /usr/local/cargo/bin/loxe-api .
COPY .env ./.env
COPY data ./data
COPY migrations ./migrations
USER 1000
CMD ["./loxe-api"]

Answer 1

While still a little bloated, at least now I have my service running on GCP Cloud Run. This is how I created a 241 mb docker image that I can ship to different services.

First, I replaced the aronautica crate with rust-argon2. Second, I modified the Dockerfile:

FROM rust AS build
WORKDIR /usr/src
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential git clang llvm-dev libclang-dev libssl-dev pkg-config libpq-dev brotli
RUN USER=root cargo new loxe-api
WORKDIR /usr/src/loxe-api
COPY Cargo.toml Cargo.lock ./
COPY data ./data
COPY migrations ./migrations
RUN cargo build --release
# Copy the source and build the application.
COPY src ./src
ENV PKG_CONFIG_ALLOW_CROSS=1
ENV OPENSSL_INCLUDE_DIR="/usr/include/openssl"
RUN cargo install --path .

FROM debian:buster-slim
COPY --from=build /usr/local/cargo/bin/loxe-api .
# standard env
COPY .env ./.env
COPY data ./data
COPY migrations ./migrations
RUN apt-get update && apt-get install -y libssl-dev pkg-config libpq-dev brotli
CMD ["/loxe-api"]

And that's essentially it. The produced Docker image now runs without problems on Google Cloud Run.

Answer 2

I cannot build your simplified Dockerfile as-is because I do not have the source files you reference in COPY statements, so I get "COPY failed" errors. You say "Building and running a new cargo default project with the x86_64-unknown-linux-musl target works" and indeed this Dockerfile (your simplified Dockerfile with the COPY commands removed) works fine for me:

FROM rust AS build
WORKDIR /usr/src

RUN rustup target add x86_64-unknown-linux-musl
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential git clang llvm-dev libclang-dev libssl-dev pkg-config libpq-dev musl-tools brotli

RUN USER=root cargo new loxe-api
WORKDIR /usr/src/loxe-api

ENV PKG_CONFIG_ALLOW_CROSS=1
ENV OPENSSL_INCLUDE_DIR="/usr/include/openssl"
ENV RUSTFLAGS="-C target-feature=+crt-static"
RUN cargo install --target x86_64-unknown-linux-musl --path .

FROM debian
COPY --from=build /usr/local/cargo/bin/loxe-api .

USER 1000
CMD ["./loxe-api"]

However, when I build that, I get a statically linked executable, which is what I expect, but not what your ldd output shows. My ldd:

$ ldd loxe-api 
    not a dynamic executable

Similarly, I can build your larger Dockerfile after removing the COPY commands and the comment, and it builds fine for me.

Are you perhaps using Windows? You may be suffering from line-ending problems, and updating your git configuration and re-cloning your git repo could help.

# update git to automatically set line ending to LF
git config --global core.eol lf
git config --global core.autocrlf input

After that, you'll need to delete and re-clone your git repo. See this helpful doc from GitHub for more.

If that does not help, please post some code that reproduces the problem.