How do I import an external module when a local module masks a standard library module imported by the external module?

Question

I have a local module named tokenize.py, masks a standard library module of the same name. I only discovered this when I tried to import an external module (sklearn.linear_model), which in turn does import tokenize and expects to get the standard library module, but gets my local module instead.

This is related to How to access a standard-library module in Python when there is a local module with the same name?, but the setting is different, because applying the above solution would require modifying the external module.

An option would be to rename the local tokenize.py, but I would prefer not to do so as "tokenize" best expresses the module's role.

To illustrate the problem, here is a sketch of the module structure:

   \my_module
      \__init__.py
      \tokenize.py
      \use_tokenize.py

In use_tokenize.py, there is the following import:

import sklearn.linear_model

Which results in the following error when invoking python my_module/use_tokenize.py:

Traceback (most recent call last):
  File "use_tokenize.py", line 1, in <module>
    import sklearn.linear_model
  <...>
  File "<EDITED>/lib/python2.7/site-packages/sklearn/externals/joblib/format_stack.py", line 35, in <module>
    generate_tokens = tokenize.tokenize
AttributeError: 'module' object has no attribute 'tokenize'

Is there any way to suppress local modules when importing an external module?

edit: Added python2.7 as a tag due to comments that the solution varies by Python version

Answer 1

The problem is not so much the module name, but that you're running a module like it were a script. When Python runs a script, it adds the script's containing directory as the first element in sys.path, so all module lookups from anywhere will search that directory first.

To avoid this, ask Python to execute it as a module instead:

python -m my_module.use_tokenize

Or, of course, you could just keep executable scripts out of your module hierarchy.