Using a custom docker with Azure ML

Question

I'm following the guidelines (https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-environments) to use a custom docker file on Azure. My script to create the environment looks like this:

from azureml.core.environment import Environment

myenv = Environment(name = "myenv")
myenv.docker.enabled = True
dockerfile = r"""
FROM mcr.microsoft.com/azureml/base:intelmpi2018.3-ubuntu16.04
RUN apt-get update && apt-get install -y libgl1-mesa-glx
RUN echo "Hello from custom container!"
"""
myenv.docker.base_image = None
myenv.docker.base_dockerfile = dockerfile

Upon execution, this is totally ignored and libgl1 is not installed. Any ideas why?

EDIT: Here's the rest of my code:

est = Estimator(
    source_directory = '.',
    script_params = script_params,
    use_gpu = True,
    compute_target = 'gpu-cluster-1',
    pip_packages = ['scipy==1.1.0', 'torch==1.5.1'],
    entry_script = 'AzureEntry.py',
    )

run = exp.submit(config = est)
run.wait_for_completion(show_output=True)

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-environments

Answer 1

Have no issues installing the lib. First, please dump your dockerfile content into a file, easier to maintain and read ;)

e = Environment("custom")
e.docker.base_dockerfile = "path/to/your/dockerfile"

will set the content of the file into a string prop.

e.register(ws).build(ws).wait_for_completion()

step 2/16 will have your apt update and libgl1 install

Note, that should work with >=1.7 SDK

Answer 2

This should work :

from azureml.core import Workspace
from azureml.core.environment import Environment
from azureml.train.estimator import Estimator
from azureml.core.conda_dependencies import CondaDependencies
from azureml.core import Experiment

ws = Workspace (...)
exp = Experiment(ws, 'test-so-exp')

myenv = Environment(name = "myenv")
myenv.docker.enabled = True
dockerfile = r"""
FROM mcr.microsoft.com/azureml/base:intelmpi2018.3-ubuntu16.04
RUN apt-get update && apt-get install -y libgl1-mesa-glx
RUN echo "Hello from custom container!"
"""
myenv.docker.base_image = None
myenv.docker.base_dockerfile = dockerfile

## You need to instead put your packages in the Environment definition instead... 
## see below for some changes too

myenv.python.conda_dependencies = CondaDependencies.create(pip_packages = ['scipy==1.1.0', 'torch==1.5.1'])

Finally you can build your estimator a bit differently :

est = Estimator(
    source_directory = '.',
#     script_params = script_params,
#     use_gpu = True,
    compute_target = 'gpu-cluster-1',
#     pip_packages = ['scipy==1.1.0', 'torch==1.5.1'],
    entry_script = 'AzureEntry.py',
    environment_definition=myenv
    )

And submit it :

run = exp.submit(config = est)
run.wait_for_completion(show_output=True)

Let us know if that works.