How to store artifacts on a server running MLflow

Question

I define the following docker image:

FROM python:3.6

RUN pip install --upgrade pip
RUN pip install --upgrade mlflow

ENTRYPOINT mlflow server --host 0.0.0.0 --file-store /mnt/mlruns/

and build an image called mlflow-server. Next, I start this server from a local machine:

docker run --rm -it -p 5000:5000 -v ${PWD}/mlruns/:/mnt/mlruns mlflow-server

Next, I define the following function:

def foo(x, with_af=False):
    mlflow.start_run()
    mlflow.log_param("x", x)
    print(x)
    if with_af:
        with open(str(x), 'wb') as fout:
            fout.write(os.urandom(1024))
        mlflow.log_artifact(str(x))
        mlflow.log_artifact('./foo.data')
    mlflow.end_run()

From the same directory I run foo(10) and the parameter is logged correctly. However, foo(10, True) yields the following error: PermissionError: [Errno 13] Permission denied: '/mnt'. Seems like log_artifact tries to save the file on the local file system directly.

Any idea what am I doing wrong?

Answer 1

Good question. Just to make sure, sounds like you're already configuring MLflow to talk to your tracking server when running your script, e.g. via MLFLOW_TRACKING_URI=http://localhost:5000 python my-script.py.

Artifact Storage in MLflow

Artifacts differ subtly from other run data (metrics, params, tags) in that the client, rather than the server, is responsible for persisting them. The current flow (as of MLflow 0.6.0) is:

• User code calls mlflow.start_run
• MLflow client makes an API request to the tracking server to create a run
• Tracking server determines an appropriate root artifact URI for the run (currently: runs' artifact roots are subdirectories of their parent experiment's artifact root directories)
• Tracking server persists run metadata (including its artifact root) & returns a Run object to the client
• User code calls log_artifact
• Client logs artifacts under the active run's artifact root

The issue

When you launch an MLflow server via mlflow server --host 0.0.0.0 --file-store /mnt/mlruns/, the server logs metrics and parameters under /mnt/mlruns in the docker container, and also returns artifact paths under /mnt/mlruns to the client. The client then attempts to log artifacts under /mnt/mlruns on the local filesystem, which fails with the PermissionError you encountered.

The fix

The best practice for artifact storage with a remote tracking server is to configure the server to use an artifact root accessible to both clients and the server (e.g. an S3 bucket or Azure Blob Storage URI). You can do this via mlflow server --default-artifact-root [artifact-root].

Note that the server uses this artifact root only when assigning artifact roots to newly-created experiments - runs created under existing experiments will use an artifact root directory under the existing experiment's artifact root. See the MLflow Tracking guide for more info on configuring your tracking server.

Answer 2

I had the same issue, try:

sudo chmod 755 -R /mnt/mlruns
docker run --rm -it -p 5000:5000 -v /mnt/mlruns:/mnt/mlruns mlflow-server

I had to create a folder with the exact path of the docker and change the permissions.

I did the same inside docker.

FROM python:3.6

RUN pip install --upgrade pip
RUN pip install --upgrade mlflow
RUN mkdir /mnt/mlruns/
RUN chmod 777 -R /mnt/mlruns/

ENTRYPOINT mlflow server --host 0.0.0.0 --file-store /mnt/mlruns/