Pygraphviz crashes after drawing 170 graphs

Question

I am using pygraphviz to create a large number of graphs for different configurations of data. I have found that no matter what information is put in the graph the program will crash after drawing the 170th graph. There are no error messages generated the program just stops. Is there something that needs to be reset if drawing this many graphs?

I am running Python 3.7 on a Windows 10 machine, Pygraphviz 1.5, and graphviz 2.38

    for graph_number in range(200):
        config_graph = pygraphviz.AGraph(strict=False, directed=False, compound=True, ranksep='0.2', nodesep='0.2')

        # Create Directory
        if not os.path.exists('Graph'):
            os.makedirs('Graph')

        # Draw Graph      
        print('draw_' + str(graph_number))
        config_graph.layout(prog = 'dot')
        config_graph.draw('Graph/'+str(graph_number)+'.png') 

Answer 1

I was able to constantly reproduce the behavior with:

1. Python 3.7.6 (pc064 (64bit), then also with pc032)
2. PyGraphviz 1.5 (that I built - available for download at [GitHub]: CristiFati/Prebuilt-Binaries - Various software built on various platforms. (under PyGraphviz, naturally) - might also want to check [SO]: Installing pygraphviz on Windows 10 64-bit, Python 3.6 (@CristiFati's answer))
3. Graphviz 2.42.2 ((pc032) same as #2.)

I suspected an Undefined Behavior (UB) somewhere in the code, even if the behavior was precisely the same:

• OK for 169 graphs
• Crash for 170

Did some debugging (added some print(f) statements in agraph.py, and cgraph.dll (write.c)).
PyGraphviz invokes Graphviz's tools (.exes) for many operations. For that, it uses subprocess.Popen and communicates with the child process via its 3 available streams (stdin, stdout, stderr).

From the beginning I noticed that 170 * 3 = 510 (awfully close to 512 (0x200)), but didn't pay as much attention as I should have til later (mostly because the Python process (running the code below) had no more than ~150 open handles in Task Manager (TM) and also Process Explorer (PE)).

However, a bit of Googleing revealed:

• [SO]: Is there a limit on number of open files in Windows (@stackprogrammer's answer) (and from here)

• [MS.Docs]: _setmaxstdio (which states (emphasis is mine)):

  C run-time I/O now supports up to 8,192 files open simultaneously at the low I/O level. This level includes files opened and accessed using the _open, _read, and _write family of I/O functions. By default, up to 512 files can be open simultaneously at the stream I/O level. This level includes files opened and accessed using the fopen, fgetc, and fputc family of functions. The limit of 512 open files at the stream I/O level can be increased to a maximum of 8,192 by use of the _setmaxstdio function.

• [SO]: Python: Which command increases the number of open files on Windows? (@NorthCat's answer)

Below is your code that I modified for debugging and reproducing the error. It needs (for code shortness' sake, as same thing can be achieved via CTypes) the PyWin32 package (python -m pip install pywin32).

code00.py:

#!/usr/bin/env python

import sys
import os
#import time
import pygraphviz as pgv
import win32file as wfile


def handle_graph(idx, dir_name):
    graph_name = "draw_{0:03d}".format(idx)
    graph_args = {
        "name": graph_name,
        "strict": False,
        "directed": False,
        "compound": True,
        "ranksep": "0.2",
        "nodesep": "0.2",
    }
    graph = pgv.AGraph(**graph_args)
    # Draw Graph      
    img_base_name = graph_name + ".png"
    print("  {0:s}".format(img_base_name))
    graph.layout(prog="dot")
    img_full_name = os.path.join(dir_name, img_base_name)
    graph.draw(img_full_name)
    graph.close()  # !!! Has NO (visible) effect, but I think it should be called anyway !!!


def main(*argv):

    print("OLD max open files: {0:d}".format(wfile._getmaxstdio()))
    # 513 is enough for your original code (170 graphs), but you can set it up to 8192
    wfile._setmaxstdio(513)  # !!! COMMENT this line to reproduce the crash !!!
    print("NEW max open files: {0:d}".format(wfile._getmaxstdio()))

    dir_name = "Graph"
    # Create Directory
    if not os.path.isdir(dir_name):
        os.makedirs(dir_name)

    #ts_global_start = time.time()
    start = 0
    count = 169
    #count = 1
    step_sleep = 0.05
    for i in range(start, start + count):
        #ts_local_start = time.time()
        handle_graph(i, dir_name)
        #print("  Time: {0:.3f}".format(time.time() - ts_local_start))
        #time.sleep(step_sleep)
    handle_graph(count, dir_name)
    #print("Global time: {0:.3f}".format(time.time() - ts_global_start - step_sleep * count))


if __name__ == "__main__":
    print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(item.strip() for item in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    main(*sys.argv[1:])
    print("\nDone.")

Output:

e:\Work\Dev\StackOverflow\q060876623>"e:\Work\Dev\VEnvs\py_pc064_03.07.06_test0\Scripts\python.exe" code00.py
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)] 64bit on win32

OLD max open files: 512
NEW max open files: 513
  draw_000.png
  draw_001.png
  draw_002.png

...

  draw_167.png
  draw_168.png
  draw_169.png

Done.

Conclusions:

• Apparently, some file handles (fds) are open, although they are not "seen" by TM or PE (probably they are on a lower level). However I don't know why this happens (is it a MS UCRT bug?), but from what I am concerned, once a child process ends, its streams should be closed, but I don't know how to force it (this would be a proper fix)
• Also, the behavior (crash) when attempting to write (not open) to a fd (above the limit), seems a bit strange
• As a workaround, the max open fds number can be increased. Based on the following inequality: 3 * (graph_count + 1) <= max_fds, you can get an idea about the numbers. From there, if you set the limit to 8192 (I didn't test this) you should be able handle 2729 graphs (assuming that there are no additional fds opened by the code)

Side notes:

• While investigating, I ran into or noticed several adjacent issues, that I tried to fix:

  • Graphviz:
    • [GitLab]: graphviz/graphviz - [Issue #1481]: MSB4018 The NativeCodeAnalysis task failed unexpectedly. (merged on 20200406)
  • PyGraphviz:
    • [GitHub]: pygraphviz/pygraphviz - AGraph Graphviz handle close mechanism

• There's also an issue open for this behavior (probably the same author): [GitHub]: pygraphviz/pygraphviz - Pygraphviz crashes after drawing 170 graphs