Why does Popen fail on Windows if the "env" parameter contains a unicode object?

Question

Consider this example:

>>> import subprocess as sp
>>> sp.Popen("notepad2.exe",env={"PATH":"C:\\users\\guillermo\\smallapps\\bin"})
<subprocess.Popen object at 0x030DF430>
>>> sp.Popen("notepad2.exe",env={"PATH":u"C:\\users\\guillermo\\smallapps\\bin"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python26\lib\subprocess.py", line 633, in __init__
    errread, errwrite)
 File "C:\Python26\lib\subprocess.py", line 842, in _execute_child
   startupinfo)
TypeError: environment can only contain strings

I've traced back the error to this CPython code:

http://hg.python.org/cpython/file/ca54c27a9045/Modules/_winapi.c#l511

I'm unable to udnerstand what PyUnicode_Check does, though:

http://hg.python.org/cpython/file/26af48f65ef3/Objects/unicodeobject.c#l73

Answer 1

I had a similar situation when configured jupyter kernel for pyspark: the kernel.json is read as unicode, and then packaged to **kwargs passed to proc = Popen(cmd, **kwargs) in the C:\Anaconda2\Lib\site-packages\jupyter_client\launcher.py

Therefore, I had to modify the launcher.py like this:

 try:
        # Ihor Bobak:  fix to convert all env keys and values to str
        klist = kwargs['env'].keys()[:]
        for key in klist:
            value = kwargs['env'][key]
            if isinstance(key, unicode) or isinstance(value, unicode):
                newkey = key.encode('ascii','ignore')
                newvalue = value.encode('ascii','ignore')
                del kwargs['env'][key]
                kwargs['env'][newkey] = newvalue
        # regular code
        proc = Popen(cmd, **kwargs) 
    except Exception as exc:
        msg = (
            "Failed to run command:\n{}\n"
            "    PATH={!r}\n"
            "    with kwargs:\n{!r}\n"
        )

Answer 2

As the error message says, the environment must only contain strings. Your first Popen call satisfies this condition, but the second one doesn't because you are mapping PATH to a Unicode object created with the u"..." syntax. Use only byte strings when providing environment dicts to Popen and you will not get this error.

Note that, judging by the traceback, you are using Python 2.6, so the linked code does not in fact apply, because it comes from Python 3.3.0 beta2. PyUnicode_Check checks that the object is a unicode object, which makes sense in Python 3, where strings are (internally implemented as) unicode objects. In Python 2.6, however, the equivalent line is using PyString_Check, which would make it fail in your second example.