Julia: limit on the number of files open while using readdlm

Question

I have some Julia code where I open a very large number of files using the readdlm function. My code looks like this:

for file in large_list_of_files
    open(file) do filehandle
        data = readdlm(filehandle)
    end
    #Do some data processing and store results
end

When I run this I get the error:

LoadError: SystemError: opening file <filename>: Too many open files
while loading In[28], in expression starting on line 28

in open at /Applications/Julia-0.4.0.app/Contents/Resources/juli/lib/julia/sys.dylib
in open at iostream.jl:102
in open at iostream.jl:112
in process_data at In[13]:11
[inlined code] from In[28]:31
in anonymous at no file:30

This doesn't make sense, since the do block should be closing the file every time I go around the loop, therefore the file should only be open in the do block. Also, if anyone knows a function in Julia that can count the number of files that are open, then that would be great :)

System info: Mac OSX, Yosemite

Julia Version 0.4.0
Commit 0ff703b* (2015-10-08 06:20 UTC)
Platform Info:
System: Darwin (x86_64-apple-darwin13.4.0)
CPU: Intel(R) Core(TM) i7-3720QM CPU @ 2.60GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.3

Also, I am running this code using Jupyter with a Julia 0.4.0 kernel, but I doubt that that would make any difference.

EDIT: I noticed this issue on Github, ViralBShah says that the problem is fixed but I don't think that it is.

https://github.com/JuliaLang/julia/issues/8891

EDIT EDIT: I have tried using the gc() function and that doesn't fix the problem either.

Answer 1

Seems to be solved:

Tested under

• MacOS HighSierra 10.13.6
• julia version 1.5.1
  

using DelimitedFiles

# 329571 files:
for file in split(String(read(a)),"\n")[1:end-1]
  open(file) do fh
    try readdlm(fh)
    catch ff
    end
  end
end