Docker build from Dockerfile with more memory

Question

How to docker build from Dockerfile with more memory?

This is a different question from this Allow more memory when docker build a Dockerfile

When installing the software natively, there is enough memory to successfully build and install the marian tool

But when building the Docker image using the Dockerfile https://github.com/marian-nmt/marian/blob/master/scripts/docker/Dockerfile.cpu , it fails with multiple memory exhausted errors

virtual memory exhausted: Cannot allocate memory 

[out]:

Step : RUN cmake $MARIANPATH && make -j  ---> Running in 4867d166d17a -- The C compiler identification is GNU 5.4.0 -- The CXX compiler identification is GNU 5.4.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done CUDA_TOOLKIT_ROOT_DIR not found or specified -- Cannot find CUDA libraries. Compiling without them. -- Could NOT find CUDA (missing:  CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY)  -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE   -- Boost version: 1.58.0 -- Found the following Boost libraries: --   system --   filesystem --   program_options --   timer --   iostreams --   python --   thread --   chrono --   regex --   date_time --   atomic -- Found Python -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.12", minimum required is "2.7")  -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.8")  -- Found Git: /usr/bin/git (found version "2.7.4")  -- Git version: 5abc774 -- Found SparseHash: /usr/include   -- Configuring done -- Generating done -- Build files have been written to: /marian/build Scanning dependencies of target fast_align Scanning dependencies of target extract_lex Scanning dependencies of target libcnpy Scanning dependencies of target atools Scanning dependencies of target libyaml-cpp-amun [  2%] Building CXX object src/amun/3rd_party/extract_lex/CMakeFiles/extract_lex.dir/exception.cpp.o [  2%] Building CXX object src/amun/3rd_party/extract_lex/CMakeFiles/extract_lex.dir/utils.cpp.o [  3%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/fast_align.dir/src/ttables.cc.o [  6%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/fast_align.dir/src/fast_align.cc.o [  7%] Building CXX object src/amun/3rd_party/extract_lex/CMakeFiles/extract_lex.dir/extract-lex-main.cpp.o [ 10%] Building CXX object src/amun/3rd_party/CMakeFiles/libcnpy.dir/cnpy/cnpy.cpp.o [ 10%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/atools.dir/src/atools.cc.o [ 10%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/atools.dir/src/alignment_io.cc.o [ 11%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/regex_yaml.cpp.o [ 12%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scanner.cpp.o [ 14%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/convert.cpp.o [ 15%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/stream.cpp.o [ 16%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/memory.cpp.o [ 17%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/node.cpp.o [ 19%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/directives.cpp.o [ 20%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/null.cpp.o [ 21%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitfromevents.cpp.o [ 23%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/singledocparser.cpp.o [ 24%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitterstate.cpp.o [ 25%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scantag.cpp.o [ 26%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/simplekey.cpp.o [ 28%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/binary.cpp.o [ 29%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/parser.cpp.o [ 30%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/nodeevents.cpp.o [ 32%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emit.cpp.o [ 33%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/exp.cpp.o [ 35%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/nodebuilder.cpp.o [ 38%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scanscalar.cpp.o [ 39%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitter.cpp.o [ 41%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/tag.cpp.o [ 42%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scantoken.cpp.o [ 43%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/ostream_wrapper.cpp.o Scanning dependencies of target libcommon [ 37%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/node_data.cpp.o [ 44%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/contrib/graphbuilder.cpp.o [ 46%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitterutils.cpp.o [ 34%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/contrib/graphbuilderadapter.cpp.o [ 47%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/parse.cpp.o [ 48%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/scorer.cpp.o [ 50%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/git_version.cpp.o [ 51%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/base_matrix.cpp.o [ 52%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/sentence.cpp.o [ 53%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/config.cpp.o [ 55%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/utils.cpp.o [ 56%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/vocab.cpp.o [ 65%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/filter.cpp.o [ 66%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/god.cpp.o [ 67%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/exception.cpp.o [ 69%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/search.cpp.o [ 69%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/hypothesis.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/translation_task.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/printer.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/sentences.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/history.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/logging.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/processor/bpe.cpp.o [ 71%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/output_collector.cpp.o [ 73%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/types.cpp.o [ 74%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/loader.cpp.o virtual memory exhausted: Cannot allocate memory virtual memory exhausted: Cannot allocate memory virtual memory exhausted: Cannot allocate memory 

And eventually it leads to:

src/amun/CMakeFiles/libcommon.dir/build.make:254: recipe for target 'src/amun/CMakeFiles/libcommon.dir/common/loader.cpp.o' failed virtual memory exhausted: Cannot allocate memory make[2]: *** [src/amun/CMakeFiles/cpumode.dir/cpu/decoder/encoder_decoder.cpp.o] Error 1 src/amun/CMakeFiles/cpumode.dir/build.make:110: recipe for target 'src/amun/CMakeFiles/cpumode.dir/cpu/decoder/encoder_decoder.cpp.o' failed [ 79%] Built target libcnpy virtual memory exhausted: Cannot allocate memory src/amun/CMakeFiles/libcommon.dir/build.make:326: recipe for target 'src/amun/CMakeFiles/libcommon.dir/common/printer.cpp.o' failed make[2]: *** [src/amun/CMakeFiles/libcommon.dir/common/printer.cpp.o] Error 1 CMakeFiles/Makefile2:340: recipe for target 'src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/all' failed make[1]: *** [src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/all] Error 2 CMakeFiles/Makefile2:182: recipe for target 'src/amun/CMakeFiles/libcommon.dir/all' failed make[1]: *** [src/amun/CMakeFiles/libcommon.dir/all] Error 2 make[1]: *** [src/amun/CMakeFiles/cpumode.dir/all] Error 2 CMakeFiles/Makefile2:110: recipe for target 'src/amun/CMakeFiles/cpumode.dir/all' failed make: *** [all] Error 2 

Looking at the docker build --help, there are options to control memory usage:

$ docker build --help  Usage:  docker build [OPTIONS] PATH | URL | -  Build an image from a Dockerfile  Options:       --build-arg list             Set build-time variables (default [])       --cache-from stringSlice     Images to consider as cache sources       --cgroup-parent string       Optional parent cgroup for the container       --compress                   Compress the build context using gzip       --cpu-period int             Limit the CPU CFS (Completely Fair Scheduler) period       --cpu-quota int              Limit the CPU CFS (Completely Fair Scheduler) quota   -c, --cpu-shares int             CPU shares (relative weight)       --cpuset-cpus string         CPUs in which to allow execution (0-3, 0,1)       --cpuset-mems string         MEMs in which to allow execution (0-3, 0,1)       --disable-content-trust      Skip image verification (default true)   -f, --file string                Name of the Dockerfile (Default is 'PATH/Dockerfile')       --force-rm                   Always remove intermediate containers       --help                       Print usage       --isolation string           Container isolation technology       --label list                 Set metadata for an image (default [])   -m, --memory string              Memory limit       --memory-swap string         Swap limit equal to memory plus swap: '-1' to enable unlimited swap       --network string             Set the networking mode for the RUN instructions during build (default "default")       --no-cache                   Do not use cache when building the image       --pull                       Always attempt to pull a newer version of the image   -q, --quiet                      Suppress the build output and print image ID on success       --rm                         Remove intermediate containers after a successful build (default true)       --security-opt stringSlice   Security options       --shm-size string            Size of /dev/shm, default value is 64MB   -t, --tag list                   Name and optionally a tag in the 'name:tag' format (default [])       --ulimit ulimit              Ulimit options (default []) 

But I couldn't figure out the correct syntax of where exactly to put the -m option -_-|||

It isn't before the Dockerfile:

# Before Docker file. $ docker build -m 4g Dockerfile.cpu -t ibot-cpu .  "docker build" requires exactly 1 argument(s). See 'docker build --help'.  Usage:  docker build [OPTIONS] PATH | URL | -  Build an image from a Dockerfile 

It isn't after Dockerfile before -t

# Before -t $ docker build Dockerfile.cpu -m 4g -t ibot-cpu .  "docker build" requires exactly 1 argument(s). See 'docker build --help'.  Usage:  docker build [OPTIONS] PATH | URL | -  Build an image from a Dockerfile 

It isn't after -t before the local path

# Before local path $ docker build Dockerfile.cpu -t ibot-cpu -m 4g .  "docker build" requires exactly 1 argument(s). See 'docker build --help'.  Usage:  docker build [OPTIONS] PATH | URL | -  Build an image from a Dockerfile 

It isn't after the local path at the end too...

# At the end... $ docker build Dockerfile.cpu -t ibot-cpu . -m 4g  "docker build" requires exactly 1 argument(s). See 'docker build --help'.  Usage:  docker build [OPTIONS] PATH | URL | -  Build an image from a Dockerfile 

How to docker build from Dockerfile with more memory?

My docker version:

docker version Client:  Version:      17.03.1-ce  API version:  1.27  Go version:   go1.7.5  Git commit:   c6d412e  Built:        Tue Mar 28 00:40:02 2017  OS/Arch:      darwin/amd64  Server:  Version:      17.04.0-ce  API version:  1.28 (minimum version 1.12)  Go version:   go1.7.5  Git commit:   4845c56  Built:        Wed Apr  5 18:45:47 2017  OS/Arch:      linux/amd64  Experimental: false 

Answer 1

It is not something about order. The Dockerfile must be specified with -f

docker build -f Dockerfile.cpu -t ibot-cpu -m 4g . 

However, take into account that by default docker does not limit the container memory. It can take the whole free memory.

As I can see that you are on OSX, which runs docker over a Linux VM. Configure the max memory clicking the whale icon in the task bar. It's 2GB by default.

For further information please see my other answer: How to assign more memory to docker container