When can import find a module when nothing else can?

Question

In short, how can this happen?

cternus@astarael:~⟫ python
Python 2.7.12 (default, Jun 29 2016, 14:05:02)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import backports
>>> import imp
>>> imp.find_module('backports')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named backports

The imp module claims to be "an interface to the mechanisms used to implement the import statement." If this is so, why can the import statement find backports, but imp.find_module() can't?

For some background: backports claims to be a "namespace package," not a package in its own right; other modules, such as backports.shutil_get_terminal_size, are situated in this namespace. This formed the basis of an ultimately-rejected PEP. I'm asking this question because I'm having a variant of this issue and am trying to track down the cause.

For more weirdness:

>>> backports.__file__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '__file__'
>>> dir(backports)
['__doc__', '__name__', '__path__']
>>> backports.__path__
['/Library/Python/2.7/site-packages/backports']
>>> import os; os.path.exists(backports.__path__[0])
False

(And no, I have no files or directories named backports or backports.py anywhere on my system.)

Edited to clarify: I am aware that this probably represents a strange configuration state of my system. My question is not "how can I fix this" but "how is it possible?"

Answer 1

This module can be brought by python-configparser APT package.

how is it possible?

This is possible because python-configparser uses path configuration file (.pth file):

[email protected]:/# dpkg -L python-configparser | head | tail -n 1
/usr/lib/python2.7/dist-packages/configparser-3.5.0b2-nspkg.pth

This file is automatically picked up by python's site module on interpreter startup because it is located in /usr/lib/python2.7/dist-packages/ and has .pth extension. As docs say:

A path configuration file is a file whose name has the form name.pth and exists in one of the four directories mentioned above... Lines starting with import (followed by space or tab) are executed.

The file /usr/lib/python2.7/dist-packages/configparser-3.5.0b2-nspkg.pth contains the following:

import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('backports',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('backports', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('backports', [os.path.dirname(p)])));m = m or sys.modules.setdefault('backports', types.ModuleType('backports'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)

So, this code is auto-executed on python startup. Slightly beautified, it looks like this:

import sys, types, os

has_mfs = sys.version_info > (3, 5)

p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('backports',))
importlib = has_mfs and __import__('importlib.util')
has_mfs and __import__('importlib.machinery')

m = has_mfs and sys.modules.setdefault('backports', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('backports', [os.path.dirname(p)])))

m = m or sys.modules.setdefault('backports', types.ModuleType('backports'))

mp = (m or []) and m.__dict__.setdefault('__path__',[])

(p not in mp) and mp.append(p)

What it does (at least on python 2) is: it manually creates a module object by invoking types.ModuleType constructor (that's why it looks like <module 'backports' (built-in)>) and puts it to sys.modules with sys.modules.setdefault('backports', types.ModuleType('backports')). After it was added to sys.modules, import backports will just return that object.

__path__ gives a hint

[email protected]:/# python -c 'import backports; print backports.__path__'
['/usr/lib/python2.7/dist-packages/backports']
[email protected]:/# dpkg -S /usr/lib/python2.7/dist-packages/backports
python-configparser: /usr/lib/python2.7/dist-packages/backports

I have no files or directories named backports

On Ubuntu, there is /usr/lib/python2.7/dist-packages/backports brought by this package as shown above, so I'm not sure why you don't have it. Maybe it's another package behaving similarly/variant of it for MacOS is different/you just deleted that dir while debugging the issue?