importing a package that doesn't exist

Question

I've never seen an import issue like this before. I removed a directory from site-packages and the corresponding package is still importable.

python2
> import google
> print(google.__path__)
['/home/bamboo/.local/lib/python2.7/site-packages/google']

However this directory doesn't actually exist

ls: cannot access /home/bamboo/.local/lib/python2.7/site-packages/google: No such file or directory

I've removed everything that I'm aware of that is related to it, but there must still be something hanging around.

Digging another level deeper I tried to reload google.

python2
> import google;
> reload(google);
ImportError: No module named google

So apparently it recognizes it is gone on reload.

Checking out sys.modules you get

python2
> import sys
> print(sys.modules)
{'google': <module 'google' (built-in)>, 'copy_reg': <module 'copy_reg' from '/usr/lib/python2.7/copy_reg.pyc'> ...

which indicates that apparently google is a built-in.

Note on motivation: Usually this sort of issue would be weird, but not a show stopper. The problem for me is that the google package is masking a different package of the same name.

Answer 1

tl,dr: use pip to uninstall Google packages completely.

There are two issues here:

• strange import/reload behaviour of the google package
• removal of the google package

import/reload behavior

I can reproduce the import/reload behaviour by installing the (Google) protobuf package (many Google packages will behave in the same way).

$ mktmpenv -p $(which python2)
...
$ python --version
Python 2.7.13
$ pip install protobuf
...
Installing collected packages: six, protobuf
Successfully installed protobuf-3.5.1 six-1.11.0

>>> import google
>>> print google.__path__
['~/virtual-envs/tmp-66cd9b4d01a8dec6/lib/python2.7/site-packages/google']
>>> import sys
>>> print sys.modules['google']
<module 'google' (built-in)>
>>> reload(google)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named google

I suspect what's going on here is that Google prefer to have all Google packages installed under a single google package, but this package is not designed to be importable, hence the unexpected reload behaviour. However importing subpackages by name works as expected:

>>> import protobuf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named protobuf

>>> from google import protobuf
>>> protobuf.__path__
['~/virtual-envs/tmp-66cd9b4d01a8dec6/lib/python2.7/site-packages/google/protobuf']
>>> reload(protobuf)
<module 'google.protobuf' from '~/virtual-envs/tmp-66cd9b4d01a8dec6/lib/python2.7/site-packages/google/protobuf/__init__.pyc'>
>>> 

Removal of the google package

The question states:

I removed a directory from site-packages and the corresponding package is still importable.

This can also be reproduced:

($ rm -rf ~/virtual-envs/tmp-66cd9b4d01a8dec6/lib/python2.7/site-packages/google
$  python
>>> import google
>>> print google.__path__
['~/virtual-envs/tmp-66cd9b4d01a8dec6/lib/python2.7/site-packages/google']
>>> 

The problem here is that simply removing the google directory and its contents is not enough to completely uninstall whatever Google packages are present.

The site-packages directory still contains the file protobuf-3.5.1-py2.7-nspkg.pth, which contains this code (split into separate lines for readability, the original is a single line of semi-colon separated statements):

import sys, types, os
has_mfs = sys.version_info > (3, 5)
p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',))
importlib = has_mfs and __import__('importlib.util')
has_mfs and __import__('importlib.machinery')
m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])))
m = m or sys.modules.setdefault('google', types.ModuleType('google'))
mp = (m or []) and m.__dict__.setdefault('__path__',[])
(p not in mp) and mp.append(p)

The line

m = m or sys.modules.setdefault('google', types.ModuleType('google'))

is creating the google module in sys.modules if it doesn't already exist - this is why the google module is importable even after the directory has been deleted.

The correct way to remove the google module is by uninstalling google packages using pip:

pip uninstall protobuf

If pip isn't available in the build environment the it's a case of identifying any related files and folders (*dist-info/, *.pth) in site-packages and removing them manually.