Using __future__ style imports for module specific features in Python

Question

The Python future statement from __future__ import feature provides a nice way to ease the transition to new language features. Is it is possible to implement a similar feature for Python libraries: from myproject.__future__ import feature?

It's straightforward to set a module wide constants on an import statement. What isn't obvious to me is how you could ensure these constants don't propagate to code executed in imported modules -- they should also require a future import to enable the new feature.

This came up recently in a discussion of possible indexing changes in NumPy. I don't expect it will actually be used in NumPy, but I can see it being useful for other projects.

As a concrete example, suppose that we do want to change how indexing works in some future version of NumPy. This would be a backwards incompatible change, so we decide we to use a future statement to ease the transition. A script using this new feature looks something like this:

import numpy as np
from numpy.__future__ import orthogonal_indexing

x = np.random.randn(5, 5)
print(x[[0, 1], [0, 1]])  # should use the "orthogonal indexing" feature
# prints a 2x2 array of random numbers

# we also want to use a legacy project that uses indexing, but
# hasn't been updated to the use the "orthogonal indexing" feature
from legacy_project import do_something

do_something(x)  # should *not* use "orthogonal indexing"

If this isn't possible, what's the closest we can get for enabling local options? For example, is to possible to write something like:

from numpy import future
future.enable_orthogonal_indexing()

Using something like a context manager would be fine, but the problem is that we don't want to propagate options to nested scopes:

with numpy.future.enable_orthogonal_indexing():
    print(x[[0, 1], [0, 1]])  # should use the "orthogonal indexing" feature
    do_something(x)  # should *not* use "orthogonal indexing" inside do_something

Answer 1

The way Python itself does this is pretty simple:

In the importer, when you try to import from a .py file, the code first scans the module for future statements.

Note that the only things allowed before a future statement are strings, comments, blank lines, and other future statements, which means it doesn't need to fully parse the code to do this. That's important, because future statements can change the way the code is parsed (in fact, that's the whole point of having them…); strings, comments, and blank lines can be handled by the lexer step, and future statements can be parsed with a very simple special-purpose parser.

Then, if any future statements are found, Python sets a corresponding flag bit, then re-seeks to the top of the file and calls compile with those flags. For example, for from __future__ import unicode_literals, it does flags |= __future__.unicode_literals.compiler_flag, which changes flags from 0 to 0x20000.

In this "real compile" step, the future statements are treated as normal imports, and you will end up with a __future__._Feature value in the variable unicode_literals in the module's globals.

---

Now, you can't quite do the same thing, because you're not going to reimplement or wrap the compiler. But what you can do is use your future-like statements to signal an AST transform step. Something like this:

flags = []
for line in f:
    flag = parse_future(line)
    if flag is None:
        break
    flags.append(flag)
f.seek(0)
contents = f.read()
tree = ast.parse(contents, f.name)
for flag in flags:
    tree = transformers[flag](tree)
code = compile(tree, f.name)

Of course you have to write that parse_future function to return 0 for a blank line, comment, or string, a flag for a recognized future import (which you can look up dynamically if you want), or None for anything else. And you have to write the AST transformers for each flag. But they can be pretty simple—e.g., you can transform Subscript nodes into different Subscript nodes, or even into Call nodes that call different functions based on the form of the index.

To hook this into the import system, see PEP 302. Note that this gets simpler in Python 3.3, and simpler again in Python 3.4, so if you can require one of those versions, instead read the import system docs for your minimum version.

---

For a great example of import hooks and AST transformers being used in real life, see MacroPy. (Note that it's using the old 2.3-style import hook mechanism; again, your own code can be simpler if you can use 3.3+ or 3.4+. And of course your code isn't generating the transforms dynamically, which is the most complicated part of MacroPy…)

Answer 2

The __future__ in Python is both a module and also not. The Python __future__ is actually not imported from anywhere - it is a construct used by the Python bytecode compiler, deliberately chosen so that no new syntax needs to be created. There is also a __future__.py in the library directory; it can be imported as such: import __future__; and then you can for example access the __future__.print_function to find out which Python version makes the feature optionally available and in which version the feature is on by default.

---

It is possible to make a __future__ module that knows what is being imported. Here is an example of myproject/__future__.py that can intercept feature imports on per module basis:

import sys
import inspect

class FutureMagic(object):
    inspect = inspect

    @property
    def more_magic(self):
        importing_frame = self.inspect.getouterframes(
                  self.inspect.currentframe())[1][0]
        module = importing_frame.f_globals['__name__']
        print("more magic imported in %s" % module)

sys.modules[__name__] = FutureMagic()

On load time the module is replaced with a FutureMagic() instance. Whenever more_magic is imported from myproject.FutureMagic, the more_magic property method will be called, and it will print out the name of the module that imported the feature:

>>> from myproject.__future__ import more_magic
more magic imported in __main__

Now, you could have a bookkeeping of the modules that have imported this feature. Doing import myproject.__future__; myproject.__future__.more_magic would trigger the same machinery, but you could also ensure that the more_magic import be at the beginning of the file - its global variables at that point shouldn't contain anything else except values returned from this fake module; otherwise the value is being accessed for inspection only.

However the real question is: how could you use this - to find out from which module the function is being called is quite expensive, and would limit the usefulness of this feature.

---

Thus a possibly more fruitful approach could be to use import hooks to do source translation on abstract syntax trees on modules that do from mypackage.__future__ import more_magic, possibly changing all object[index] into __newgetitem__(operand, index).