How __future__ imports work under the hood

Question

I've been fascinated by the __future__ module - in particular, its ability to change the way statements are parsed in python.

What's most interesting is how doing something like

from __future__ import print_function

Enables you to use print (and not print_function, like you would expect any other normal import to do).

I have read What is __future__ in Python used for and how/when to use it, and how it works thoroughly and in particular came across a particular line:

A future statement is a directive to the compiler that a particular module should be compiled using syntax or semantics that will be available in a specified future release of Python.

I would love to know the intricacies of what exactly makes this possible. In particular, how something like

from __future__ import division

Can enable true division on python2, while

from __future__ import barry_as_FLUFL

Can enable the <> syntax on python3 (what I find most funny is that you have to import a feature from "__future__" for backward compatibility).

Anyway, to summarise, I would like to know how the directive is understood and executed by the compiler when __future__ or its artefacts are imported.

Answer 1

from __future__ import print_function tells the parser to not treat print as a keyword (leaving it as a name instead). That way the compiler treats it as the function and not a statement.

To track this, the compiler struct has a c_future field that holds a PyFutureFeatures object that tracks which future directives have been enabled. Various parts of the parser and compiler check the flags and alter behaviour.

This is mostly handled in the future.c source file, which has a future_parse() function that checks for import from AST objects with the module parameter set to __future__, and sets flags based on what is found.

For example, for the barry_as_FLUFL 'feature', the parser refuses != as syntax but accepts <> instead:

if (type == NOTEQUAL) {
    if (!(ps->p_flags & CO_FUTURE_BARRY_AS_BDFL) &&
                    strcmp(str, "!=")) {
        PyObject_FREE(str);
        err_ret->error = E_SYNTAX;
        break;
    }
    else if ((ps->p_flags & CO_FUTURE_BARRY_AS_BDFL) &&
                    strcmp(str, "<>")) {
        PyObject_FREE(str);
        err_ret->text = "with Barry as BDFL, use '<>' "
                        "instead of '!='";
        err_ret->error = E_SYNTAX;
        break;
    }
}

You can find the other examples by grepping for the FUTURE_* flags listed in compile.h.

Note that there is a __future__ Python module, but it is not directly involved in the parsing and compilation of code; it is merely there to give Python code easy access to metadata about directives (including bitfield values to pass to the flags argument of the compile() function), nothing more.