OSError: [Errno 22] Invalid argument in subprocess

Question

Python 3.3.3 Windows 7

Here is the full stack:
Traceback (most recent call last):
  File "Blah\MyScript.py", line 578, in Call
    output = process.communicate( input=SPACE_KEY, timeout=600 )
  File "C:\Python33\lib\subprocess.py", line 928, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "C:\Python33\lib\subprocess.py", line 1202, in _communicate
    self.stdin.write(input)
OSError: [Errno 22] Invalid argument

The code looks like this:

process = subprocess.Popen( arguments,
                stdin=subprocess.PIPE,
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
                universal_newlines=True,
                env=environment )

output = process.communicate( input=SPACE_KEY, timeout=600 )

This code runs hundreds of times a day without problems. But if more than one script is running on the same machine (the same script, but sometimes from different folders) I get the error. The scripts are not executing the same thing (i.e.: the other script is not executing a subprocess when I get this error).

The subProcess code raises the error with many different command lines fed to it.

So, anyone has an idea as to what is happening? Does the interpreter have a problem with multiple execution (in different processes)? Same code that normally works perfectly fine, craps out if the interpreter is running the same (or very similar) scripts. But they are usually executing different parts of the script.

I'm at a loss: Using a single processor on an 8 core machine is annoying.

Answer 1

Previously communicate only ignored an EPIPE error when writing to the process stdin. Starting with 3.3.5, per issue 19612, it also ignores EINVAL (22) if the child has already exited (see Lib/subprocess.py line 1199).

Background:

process.communiciate calls process.stdin.write, which calls io.FileIO.write, which on Windows calls the C runtime _write, which calls Win32 WriteFile (which in this case calls NtWriteFile, which dispatches to the NamedPipe filesystem, as either an IRP_MJ_WRITE or FastIoWrite).

If the latter fails, it sets a Windows system error code in the thread. In this case the underlying Windows error is probably ERROR_NO_DATA (232) because the child process has already exited. The C runtime maps this to an errno value of EINVAL (22). Then since _write failed, FileIO.write raises OSError based on the current value of errno.

---

Addendum:

There wouldn't have been a problem at all if the CRT instead mapped ERROR_NO_DATA to EPIPE. Python's own Windows error translation generally follows the CRT's, but per issue 13063, it makes an exception to map ERROR_NO_DATA to EPIPE (32). Thus if the child has already exited, _winapi.WriteFile raises BrokenPipeError.

The following example replicates the EINVAL error given the child process has already exited. It also shows how _winapi.WriteFile (3.3.3 source link) would instead map this error to EPIPE. IMO, this should be considered a bug in Microsoft's CRT.

>>> cmd = 'reg query hkcu'                                                
>>> process = Popen(cmd, stdin=PIPE, stdout=PIPE, universal_newlines=True)
>>> process.stdin.write(' ')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument

>>> hstdin = msvcrt.get_osfhandle(process.stdin.fileno())
>>> _winapi.WriteFile(hstdin, b' ')                                       
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
BrokenPipeError: [WinError 232] The pipe is being closed

Answer 2

@eryksun very well analyzed the core of the issue, but I think the bigger picture (the obvious things, probably) is missing in his answer.

So, anyone has an idea as to what is happening?

Your code suffers from a race condition. Sometimes your child process does not behave the way you think it behaves.

OSError: [Errno 22] Invalid argument is raised in your case when the child exits before communicate attempts to write to the named pipe, which subprocess has established between your parent and stdin of your child process.

Popen() does a lot under the hood. In your case, it first creates three named pipes (in _get_handles()), via _winapi.CreatePipe(), one for each of stdin/out/err. Then, it spawns the child process (in _execute_child()), using _winapi.CreateProcess().

_execute_child() finishes with cleanup procedures. Remember: all of this happens within Popen().

Only after Popen() returns, your Python VM in the parent process is about to proceed with the invocation of output = process.communicate(input=SPACE_KEY, timeout=600)

Given that you are on a multi core system, your system has time slices available to let the child process do some work while your Python interpreter is still within execution of Popen().

That is, there is a narrow time window between _winapi.CreateProcess() (after which the child process does some work) and Python's attempt to write to the child's stdin via communicate() (which makes a call to Windows' WriteFile, as eryksun nicely explained).

When your child exits in that time window you retrieve named error.

Why your child process exits earlier than expected? Only you can tell. Obviously, it does not always wait for data coming from stdin.