Fix Import error on using environb in python

Question

from os import environb as environ

On typing the above command in Windows Anaconda python 3.6 installation, I get this error:

ImportError: cannot import name 'environb'.

This seems to be because,

environb is only available if supports_bytes_environ is True.

and

supports_bytes_environ is False on Windows.

Is there a way to get around this?

ref: https://docs.python.org/3/library/os.html#os.environb

Specifically, I found this error in the GRASS GIS Python scripting library.

Answer 1

First of all: you probably don't need os.environb, at all.

I'll cover why not when you are developing Python software, and at the end, also cover GRASS GIS and how to fix that project properly.

Why you won't need this object

On non-Windows systems, the os.environb mapping is only needed if you ever need to access the raw binary data from an environment, without it having being decoded to Unicode according to the current locale. You could want to have that access because the locale could be wrong, or you wanted to pass in binary data from an environment variable to your program without having to re-encode it with the locale and the surrogateescape error handler, or to pass data in a different encoding to another program, again without having to forcefully create a surrogateescape decoded string first. (I'm glossing over the fact that in POSIX you can't use nulls in environment variables but that's not relevant).

On Windows, you don't need this because on that OS the environment variables are already passed to Python as Unicode data. This also means that the Windows environment can't readily be used to pass binary data; you can't pass in data with a different encoding for child processes and you can't accept binary data from the environment without first bundling that data in some kind of binary-to-text encoding such as Base64. os.environb would not serve any purpose on Windows!

So, if you creating cross-platform software, you should use os.environ and require that the locale is correctly configured, and not worry about os.environb.

Code defensively instead

Sometimes need binary environment data access? Then next option could be to defensively code for the attribute missing with an ImportError guard, and just accept that it is missing:

try:
    from os import environb
except ImportError:
    environb = None

# ...

if environb is not None:
    # ... it is available, use it

Full os.environb replacement

A last option, for that case where some third party expects os.environb to be available anyway and you can't change that, or where you have a large codebase that is going to be hard to update, is to create the os.environb object just for Windows.

This is not that hard; just encode the data from the original os.environ as needed, and decode it again on setting new keys or values. The os.environ object for POSIX already does the same thing, except in the other direction, so we can re-use the same infrastructure:

import os
try:
    os.environb
except AttributeError:
    # Windows has no os.environb, create one that maps to os.environ._data
    import sys
    _encoding = sys.getfilesystemencoding()  # echos POSIX
    # note the inversion, we *decode* where encoding is expected, and vice versa
    def _encode(value, _encoding=_encoding):
        if not isinstance(value, bytes):
            raise TypeError("bytes expected, not %s" % type(value).__name__)
        return value.decode(_encoding, 'surrogateescape')
    def _decode(value, _encoding=_encoding):
        return value.encode(_encoding, 'surrogateescape')

    # reuse the Unicode mapping, putenv and unsetenv references from os.environ
    # but map binary data to unicode on setting, unicode to binary on getting
    os.environb = os._Environ(
        os.environ._data,
        _encode, _decode, _encode, _decode,
        os.environ.putenv, os.environ.unsetenv)

    del _encoding, _encode, _decode

This creates the same type of mapping object, which fully supports getting and setting environment variables, and the changes to that object will be visible in os.environ, and vice versa:

>>> os.environ
environ({'FOO': 'bar baz', 'SPAM': 'håm'})
>>> os.environb
environ({b'FOO': b'bar baz', b'SPAM': b'h\xc3\xa5m'})
>>> os.environb[b'eric'] = 'Îdlé'.encode('utf8')
>>> os.environ
environ({'FOO': 'bar baz', 'SPAM': 'håm', 'eric': 'Îdlé'})
>>> del os.environ['FOO']
>>> os.environb
environ({b'SPAM': b'h\xc3\xa5m', b'eric': b'\xc3\x8edl\xc3\xa9'})

---

GRASS GIS specifically

In the comments you mention you are trying to get GRASS GIS to work. That project is simply made an incorrect choice to set an environment variable as bytes on both Python 2 and Python 3, and has issues not just for Windows but for all platforms that need addressing.

They try to use os.environb as a replacement for os.environ, and then use a naive quoting method to generate the value from sys.argv. At the same time, the same module uses os.environ for all other environment variable needs.

At the top of the lib/python/script/core.py they use

# python3
# ...
from os import environb as environ

and then store a single variable in that mapping (in the def parser(): function definition):

cmdline = [basename(encode(sys.argv[0]))]
cmdline += [b'"' + encode(arg) + b'"' for arg in sys.argv[1:]]
environ[b'CMDLINE'] = b' '.join(cmdline)

The b'"' + encode(arg) + b'"' is a naive method for quoting values to avoid problems with subshells, but this won't handle embedded quotes.

There is no reason for this to be a bytes value. sys.argv is a list of Unicode strings on Python 3, bytestrings on Python 2. This follows the os.environ data types on either Python version, so the data should just be handled as str types, on either Python version.

For quoting values against shell interpretation, Python has the shlex.quote() function, which happens to be available as pipes.quotes() on both Python 2 and Python 3 as well.

So the issue can be avoided entirely with a few changes to that file (the traceback you have for the os.environb import error will tell you where it is located on your computer):

# 1. at the top, add an import
import pipes

# 2. remove the `from os import environb as environ` line altogether

# 3. in def parse(), use
cmdline = [basename(sys.argv[0])]
cmdline += (pipes.quote(a) for a in sys.argv[1:])
os.environ['CMDLINE'] = ' '.join(cmdline)

I'm reporting this to the GRASS GIS project so that they can fix this for a future release.

Answer 2

On Windows, the os module does not have the attribute environb, so you will not be able to load it. However, you can add it manually:

First, load os into the global namespace:

import os

Overload it into the os module. This will modify the module that is already loaded:

os.environb = {bytes(k, encoding='utf-8'): bytes(env, encoding='utf8') for k, env in os.environ.items()}

Now, if you run from os import environb as environ, python will see that os is already imported and will not attempt to load it again.

environ[b'PATH']

>>> b'C:\\Windows\\System32'

If you also need to be able to set environment variables, you can provide a two-way mapping by using the following:

class Environb(object):
    def __init__(self):
        pass
    def __getitem__(self, item):
        return bytes(os.environ[item.decode('utf8')], encoding='utf-8')
    def __setitem__(self, key, item):
        os.environ[key.decode('utf8')] = item.decode('utf8')

os.environb = Environb()

os.environb[b'FOO'] = b'BAR'
print(os.environ['FOO']

>>> 'BAR'