"RuntimeError: fdopen() failed unexpectedly" when reading too many gml files

Question

I am reading too many GML files (few thousands) using igraph with python. At some point of running the code, I got the following run time error:

RuntimeError: fdopen() failed unexpectedly

I spent a lot of time trying to understand the causes, but I did not find any thing helpful.

The root cause was produced from the code here https://github.com/igraph/python-igraph/blob/master/src/filehandle.c#L231

The code I used is bellow. It breaks at the line of reading the GML after reading only few hundreds of the files.

gmls= []
for f in sorted(glob.glob('path_to_gmls'), key=os.path.getsize):
    g = Graph.Read_GML(f)
    gmls.append(g)

UPDATE: I tried the same code on Mac, and it works just fine. The problems is on Windows.

UPDATE2: I tested the package using the following code, and it runs without problems.

import igraph.test
igraph.test.run_tests()

Answer 1

Are you running out of memory? (I'm not a python programmer). If multiple files are being opened at the same time then the buffers for those open handles might be taking up a considerable amount of memory. Also, it appears that you are appending the contents of the file to an in-memory array. If the contents are large again it could be a memory issue.