Python Popen hangs, yet same command in Windows cmd window runs fine

Question

I am pulling my hair out here. I am spawning a process which I need the feedback from in Python.

When I run the command in the cmd window it runs fine, but when I try to run it via Python the terminal hangs.

p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(out, err) = p.communicate()

Where startcmd is a string which when printed in the Python console looks like this:

"C:/Program Files/GRASS GIS 7.2.1/grass72.bat" --version

If I copy and paste this into a Windows cmd, it shows the version information and returns control to the command prompt about a second later, but in Python it freezes up.

I should point out, if I replace the startcmd string with something like "dir" or even "python --version", it works fine!

Additional: I have tried shell=True, this has the same result.

Additional: I have tried sending the cmd and arguments through as an array as suggested in an answer below given that shell=False, but this also hangs the same.

Additional: I have added the GRASS path to the system PATH, so that now I can simply call grass72 --version in the cmd window to get a result, however this also still freezes in Python but works fine in cmd.

Additional: I have created a basic .bat file to test if .bat files run ok via Python, here is what I created:

@echo off
title Test Batch Script
echo I should see this message

This runs fine both in cmd, and in Python.

Problem found but not solved!

So, I'm running the script which spawns the process using subprocess.Popen using Python 3.6. The .bat file which is spawned launches a Python script using a version of Python (based on 2.7) which comes shipped with GRASS:

%GRASS_PYTHON% "\BLAH\BLAH\grass72.py"

What is interesting, is that if I launch the subprocess.Popen script with Python 2.7, it works fine. Ahah, you may think, solved! But this doesn't solve my problem - because I really need Python 3.6 to be launching the process, also why does it matter what version of Python launches the batch file? The new Python script which is spawned is launched with Python 2.7 anyway.

Since I started re-directing stdout I can see that there is an error when I use Python 3.6 to launch the process:

 File "C:\ProgramData\Anaconda3\lib\site.py", line 177
    file=sys.stderr)
        ^
SyntaxError: invalid syntax

Notice its reverting to Anaconda3! Even though it is launched using python.exe from 2.7!

Answer 1

I experienced the same issue with Python 3.6 and 3.7 on Windows hanging for subprocess calls:

p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(out, err) = p.communicate()

Upon closer investigation I noticed this occurs only if the process writes more than about 4 KB (4096 bytes) of output which might explain why your short script does not reproduce this.

A workaround I found is using tempfile in the standard library:

# Write to a temporary file because pipe redirection seems broken
with tempfile.NamedTemporaryFile(mode="w+") as tmp_out,
        tempfile.NamedTemporaryFile(mode="w+") as tmp_err:

    p = subprocess.Popen(startcmd, stdout=tmp_out, stderr=tmp_err,
        universal_newlines=True)

    # `run` waits for command to complete, `Popen` continues Python program
    while p.poll() is None:
        time.sleep(.1)

    # Cursor is after the last write call, reset to read output
    tmp_out.seek(0)
    tmp_err.seek(0)
    out = tmp_out.read()
    err = tmp_err.read()