Increasing maximum number of open file descriptors in Matlab in Windows using fopen

Question

I have a program which needs to hold approximately 3000 open file descriptors in Matlab. The reason for this is that if I dont keep them open I need to open and close them over 100 000 times which means 300 million open close operations. Considering that each file is appended to each time and fopen and fclose can take upwards of a second each (files are large i.e. 100mb+) it should be clear this situation is unacceptable.

I know that the Windows limit for file handles is set at 10000 but Matlab refuses to open more than 512 files with fopen. I cant figure out how to force it to increase that number.

Does someone know how to change the 512 limit? Where is it defined? Is it even Matlab related?

Answer 1

FWIW, below is a bit of code to reproduce this problem:

fids = zeros(1,513);
for ix = 1:length(fids)
   fids(ix) = fopen(sprintf('testfile_%03d.tmp',ix),'w');
end
fids(507:end)

(After this, basic commands like "help" fail, you need to run fclose all).

A little bit of web searching turns up other people (on inferior Q&A forums) with the same problems, but no easy solutions (e.g. this Mathworks forum post.)

---

My first instinct when I run into Matlab limitations is always to turn to Java. For example:

streamwriters = cell(1,513);
for ix = 1:length(streamwriters)
    strName = sprintf('testfile_2_%03d.tmp',ix);
    streamwriters{ix} = java.io.FileOutputStream(strName);
end
streamwriters{513}.write(uint8('Some data to write'))

There is a cost (I think a few msec) every time you make a java call from within Matlab, so you you are really doing 1,000,000's of writes, I would profile your code, and look for ways to collect the code in memory and perform fewer, larger batched writes if needed.

Also remember, you need to close these individually, e.g.

for ix = 1:length(streamwriters)
    streamwriters{ix}.close();
end