OSError: [WinError 6] The handle is invalid when calling subprocess from Python 3.6

Question

I'm porting a project to Python3 and I'm running into a unexpected error on Windows:

Basically on Python 3.6 on Windows, each time a process is being created with subprocess, I have this exception:

d:\temp\backpack\venv\myvenv_py3.6\lib\site-packages\git\cmd.py:1011: in _call_process
    return self.execute(call, **exec_kwargs)
d:\temp\backpack\venv\myvenv_py3.6\lib\site-packages\git\cmd.py:732: in execute
    **subprocess_kwargs
D:\temp\cpython-3.6.3\Lib\subprocess.py:611: in __init__
    _cleanup()
D:\temp\cpython-3.6.3\Lib\subprocess.py:220: in _cleanup
    res = inst._internal_poll(_deadstate=sys.maxsize)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <subprocess.Popen object at 0x0000000004FD53C8>
_deadstate = 9223372036854775807
_WaitForSingleObject = <built-in function WaitForSingleObject>
_WAIT_OBJECT_0 = 0, _GetExitCodeProcess = <built-in function GetExitCodeProcess>

    def _internal_poll(self, _deadstate=None,
            _WaitForSingleObject=_winapi.WaitForSingleObject,
            _WAIT_OBJECT_0=_winapi.WAIT_OBJECT_0,
            _GetExitCodeProcess=_winapi.GetExitCodeProcess):
        """Check if child process has terminated.  Returns returncode
                attribute.

                This method is called by __del__, so it can only refer to objects
                in its local scope.

                """
        _log.debug('proc._internal_poll  self.pid=%s  self._handle=%s  self.returncode=%s  self=%s', self.pid, self._handle, self.returncode, self)
        if self.returncode is None:
>           if _WaitForSingleObject(self._handle, 0) == _WAIT_OBJECT_0:
E           OSError: [WinError 6] The handle is invalid

D:\temp\cpython-3.6.3\Lib\subprocess.py:1051: OSError

It doesn't matter where the subprocess call is being created from (in this project it's a lot from the GitPython package and from plumbum).

This execution occurs under this umbrella: build script which invokes venv/Scripts/coverage run -m pytest -v tests/. But I've also tried pytest-cov with venv/Scripts/pytest --cov=mypackage --cov-config .coveragerc - tests/ In terms of reproduction:

• on my Win 7 PC it always passes :(
• on a Win 7 virtual machine:
  • it always fails when [<build script> invokes venv/Scripts/coverage run -m pytest]
  • it always fails when coverage run -m pytest is invoked directly from the venv
  • but it always passes when pytest is invoked directly from the virtualenv
• on a Win 10 PC it will always fails, no matter if the the <build script> is invoked, or if coverage or pytest are invoked directly from the venv

The only clue that I've got so far is from this StackOverflow thread: https://stackoverflow.com/a/43975118 and to be more precise, it's the comments from the first accepted answer that point to something useful related to the _cleanup method from the subprocess module.

Answer 1

After a bit of pause, I quickly found the reason behind this.

It was because of the GitPython usage in the project which didn't call git.Repo.close() or used git.Repo as a context manager. There is a warning about this on GitPython's readme.

It was helpful to add logging in subprocess's _internal_poll method to know which process (args) is the culprit. For GitPython it was git cat-file ...