python: getting actual environment variables to modify and pass to subprocess

Question

Well, it seems situation with environment variables is not consistent in python.

It's not a secret that reading environment variables using os.environ or os.getenv returns the state of env at the moment os module was imported. It's still possible to update environment using assignment to os.environ keys.

But once I used os.putenv or run any ctypes code that has modified the environment I get inconsistency between actual process environment and os.environ. Nuff said, this actual environment will be preserved for subprocess, no matter created with os.system or subprocess library. In my case it's desired behavior.

Now I want to review and change the environment passed to subprocesses. Usually it's suggested to get copy of os.environ, modify it and pass as a parameter to subprocess.Popen call. But in this case updates made to environment made by ctypes code will be lost.

Is there any way to overcome this issue? Strictly speaking is there a way to reload os.environ or get a copy with actual environment using other facilities?

Answer 1

os.putenv() does not update os.environ as its docs say explicitly. C putenv() (in a CPython extension module) does not update os.environ too (as documented: changes in the environment after os import are not reflected in os.environ).

os.getenv(var) is just os.environ.get(var). There is related Python issue as @ShadowRanger has mentioned.

If you need it; you could access C environ from Python using ctypes e.g. (tested on Ubuntu, it might work on OS X (you might need to call _NSGetEnviron() there), it is unlikely to work on Windows (use _wenviron there)):

import ctypes

libc = ctypes.CDLL(None)
environ = ctypes.POINTER(ctypes.c_char_p).in_dll(libc, 'environ')

environ is a pointer to an array of C (NUL-terminated) strings (char*) where the last item is NULL. To enumerate values in Python 2:

for envvar in iter(iter(environ).next, None):
    print envvar

Output

LC_PAPER=en_GB.UTF-8
LC_ADDRESS=en_GB.UTF-8
CLUTTER_IM_MODULE=xim
LC_MONETARY=en_GB.UTF-8
VIRTUALENVWRAPPER_PROJECT_FILENAME=.project
SESSION=ubuntu
...

To get it as a dictionary that you could modify and pass to a child process:

env = dict(envvar.split(b'=', 1) for envvar in iter(iter(environ).next, None))

To synchronize with os.environ:

os.environ.clear() # NOTE: it clears C environ too!
getattr(os, 'environb', os.environ).update(env) # single source Python 2/3 compat.

Here're several convenience functions:

#!/usr/bin/env python
import ctypes
import functools
import os


_environ = None


def get_libc_environb_items():
    """Get envvars from C environ as bytestrings (unsplit on b'=')."""
    global _environ
    if _environ is None:
        libc = ctypes.CDLL(None)
        _environ = ctypes.POINTER(ctypes.c_char_p).in_dll(libc, 'environ')
    return iter(functools.partial(next, iter(_environ)), None)


def get_libc_environb():
    """Get a copy of C environ as a key,value mapping of bytestrings."""
    return dict(k_v.split(b'=', 1) for k_v in get_libc_environb_items()
                if b'=' in k_v)  # like CPython



def get_libc_environ():
    """Get a copy of C environ as a key,value mapping of strings."""
    environb = get_libc_environb()
    # XXX sys.getfilesystemencoding()+'surrogateescape'
    fsdecode = getattr(os, 'fsdecode', None)
    if fsdecode is None:  # Python 2
        return environb  # keep bytestrings as is (`str` type)
    else:  # Python 3, decode to Unicode
        return {fsdecode(k): fsdecode(v) for k, v in environb.items()}


def synchronize_environ():
    """Synchronize os.environ with C environ."""
    libc_environ = get_libc_environ()
    os.environ.clear()
    os.environ.update(libc_environ)


def test():
    assert 'SPAM' not in os.environ
    assert 'SPAM' not in get_libc_environ()
    os.putenv('SPAM', 'egg')
    assert 'SPAM' not in os.environ
    assert os.getenv('SPAM') is None
    assert get_libc_environ()['SPAM'] == 'egg'
    assert os.popen('echo $SPAM').read().strip() == 'egg'
    synchronize_environ()
    assert os.environ['SPAM'] == 'egg'


if __name__ == "__main__":
    test()
    from pprint import pprint
    pprint(get_libc_environ())

It works on CPython 2, CPython 3, pypy. It doesn't work on Jython.

Answer 2

This is a known issue with Python, as yet unfixed. os.getenv reads from os.environ, and setting an item on os.environ implicitly does an os.putenv, deleting implicitly calls os.unsetenv, etc.

But even though os.getenv reads from os.environ, os.putenv doesn't write to it (and this behavior is documented). And there doesn't appear to be a way to make it reread the os.environ. Basically, if you want a consistent environment, you have to update os.environ only, not use os.putenv; if ctypes calls are updating the C level environ directly, you're going to need another ctypes call to read the C level environ and update os.environ to match.