Error: Maximal number of DLLs reached

Question

I'm writing an R package which depends upon many other packages. When I load too many packages into the session I frequently got this error:

Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/Library/Frameworks/R.framework/Versions/3.2/Resources/library/proxy/libs/proxy.so':
  `maximal number of DLLs reached...

This post Exceeded maximum number of DLLs in R pointed out that the issue is with the Rdynload.c of the base R code: #define MAX_NUM_DLLS 100

Is there any way to bypass this issue except modifying and building from source?

Answer 1

As of R 3.4, you can set a different max number of DLLs using and environmental variable R_MAX_NUM_DLLS. From the release notes:

The maximum number of DLLs that can be loaded into R e.g. via dyn.load() can now be increased by setting the environment variable R_MAX_NUM_DLLS before starting R.

Answer 2

Increasing that number is of course "possible"... but it also costs a bit (adding to the fixed memory footprint of R).

I did not set that limit, but I'm pretty sure it was also meant as reminder for the useR to "clean up" a bit in her / his R session, i.e., not load package namespaces unnecessarily. I cannot yet imagine that you need > 100 packages | namespaces loaded in your R session. OTOH, some packages nowadays have a host of dependencies, so I agree that this at least may happen accidentally more frequently than in the past.

The real solution of course would be a code improvement that starts with a relatively small number of "DLLinfo" structures (say 32), and then allocates more batches (of size say 32) if needed.

Patches to the R sources (development trunk in subversion at https://svn.r-project.org/R/trunk/ ) are very welcome!

---- added Jan.26, 2017: In the mean time, we've had a public bug report about this, a proposed patch (which was not good enough: There is always an OS dependent limit on the number of open files), and today that bug report has been closed by R core member @TomasKalibera who implemented new code where the maximal number of loaded DLLs is set at

pmax(100, pmin(1000, 0.6* OS_dependent_getrlimit_or_equivalent()))

and so on Windows and Linux (and not yet tested, but "almost surely" macOS), the limit should be considerably higher than previously.

----- Update #2 (written Jan.5, 2018):
In Oct'17, the above change was made more automatic with the following commit to the sources (of the development version of R - only!)

r73545 | kalibera | 2017-10-12 14:41:20

Increase the number of DLLs that can be loaded by default. If needed, increase the soft limit on open files.

and on the help page ?dyn.load (https://stat.ethz.ch/R-manual/R-devel/library/base/html/dynload.html) the ulimit -n <num_open_files> is now mentioned (section Note close to bottom).

So you might consider using R's development version till that becomes "main stream" in April.
Alternatively, you do (in a terminal / shell)

ulimit -n 2048

and then start R from that terminal. Tomas Kalibera mentioned this to work on macOS.

Answer 3

I had this issue with the simpleSingleCell library in bioconductor

On the macOS you can't exceed 256. So I set my .Renviron in my home dir R_MAX_NUM_DLLS=150