Python: Why should 'from <module> import *' be prohibited?

Question

If you happen to have

from <module> import * 

in the middle of your program (or module), you would get the warning:

/tmp/foo:100: SyntaxWarning: import * only allowed at module level 

I understand why import * is discouraged in general (namespace invisibility), but there are many situations where it would prove convenient, especially where code is not shared with anyone.

So, can anyone explain precisely in detail why from <module> import * should be prohibited in all possible cases?

Answer 1

I believe by "in the middle of your program" you are talking about an import inside a function definition:

def f():     from module import *    # not allowed 

This is not allowed because it would make optimizing the body of the function too hard. The Python implementation wants to know all of the names of function-local variables when it byte-compiles a function, so that it can optimize variable references into operations on the (CPython) virtual machine's operand stack, or at least to local variable-slot operations rather than lookups in outer namespaces. If you could dump the entire contents of a module into a function's local namespace, then the compiler would have to assume that any name in the function might possibly refer to a module global, because the list of names brought in by from module import * is only known at runtime.

Putting from module import * in between top-level declarations is poor style, but it's allowed:

def f():     ...  from module import *  def g():     ... 

EDIT April 2013: While looking into something else, I discovered that this restriction was introduced in Python 2.1, as a consequence of the "Nested Scopes" feature (PEP 227). Quoting from the link:

One side effect of the change is that the from module import * and exec statements have been made illegal inside a function scope under certain conditions. The Python reference manual has said all along that from module import * is only legal at the top level of a module, but the CPython interpreter has never enforced this before. As part of the implementation of nested scopes, the compiler which turns Python source into bytecodes has to generate different code to access variables in a containing scope. from module import * and exec make it impossible for the compiler to figure this out, because they add names to the local namespace that are unknowable at compile time. Therefore, if a function contains function definitions or lambda expressions with free variables, the compiler will flag this by raising a SyntaxError exception.

This clarifies the Python 3.x vs 2.x behavior discussed in the comments. It is always contrary to the language specification, but CPython 2.1 through 2.7 only issue an error for from module import * within a function if it might affect the compiler's ability to know whether a variable binds locally or in a containing scope. In 3.x it has been promoted to an unconditional error.

SON OF EDIT: ... and apparently flashk pointed this out years ago in another answer, quoting the same paragraph of "What's New in Python 2.1" yet. Y'all go upvote that now.

Answer 2

At any lexical level, from amodule import * is a "seemed a good idea at the time" design decision that has proven a real disaster in real life, with the possible exception of handy exploration at the interactive interpreter prompt (even then, I'm not too hot on it -- import module as m forces only two extra characters to use qualified names instead [[just an m. prefix]], and qualified names are always sharper and more flexible than barenames, not to mention the great usefulness in exploratory interactive situations of having m available for help(m), reload(m), and the like!).

This bedraggled construct makes it very hard, for the poor person reading the code (often in a doomed attempt to help debug it) to understand where mysteriously-appearing names are coming from -- impossible, if the construct is used more than once on a lexical level; but even when used just once, it forces laborious re-reading of the whole module every time before one can convince oneself that, yep, that bedraggled barename must come from the module.

Plus, module authors usually don't go to the extreme trouble needed to "support" the horrid construct in question. If somewhere in your code you have, say, a use of sys.argv (and an import sys at the very top of your module, of course), how do you know that sys is the module it should be... or some completely different one (or a non-module) coming from the ... import *?! Multiply that by all the qualified names you're using, and misery is the only end result -- that, and mysterious bugs requiring long, laborious debugging (usually with the reluctant help of somebody who does "get" Python...!-).

Within a function, a way to add and override arbitrary local names would be even worse. As an elementary but crucial optimization, the Python compiler looks around the function's body for any assignment or other binding statements on each barename, and deems "local" those names it sees thus assigned (the others must be globals or built-ins). With an import * (just like with an exec somestring without explicit dicts to use as namespaces), suddenly it becomes a total mystery which names are local, which names are global -- so the poor compiler would have to resort to the slowest possible strategy for each name lookup, using a dict for local variables (instead of the compact "vector" it normally uses) and performing up to three dict look-ups for each barename referenced, over and over.

Go to any Python interactive prompt. Type import this. What do you see? The Zen of Python. What's the last and probably greatest bit of wisdom in that text...?

Namespaces are one honking great idea -- let's do more of those!

By forcing the use of barenames where qualified names are so vastly preferable, you're essentially doing the very opposite of this wise recommendation: instead of admiring the greatness and honkingtude of namespaces, and doing more of those, you're breaking down two perfectly good and ready-to-use namespaces (that of the module you're importing, and that of the lexical scope you're importing it in) to make a single, unholy, buggy, slow, rigid, unusable mess.

If I could go back and change one early design decision in Python (it's a hard choice, because the use of def and especially lambda for what Javascript so much more readably calls function is a close second;-), I would retroactively wipe out the import * idea from Guido's mind. No amount of alleged convenience for exploration at the interactive prompt can balance the amount of evil it's wrought...!-)