Running without Python source files in Python 3.4

Question

I'm trying to run a Python application without keeping the .py source files around, and only relying on the .pyc compiled files. However, I am getting import errors when I remove the .py source files. This functionality is working in Python 2.7, but not in 3.4 (with the new __pycache__ structure).

Here's a sample directory structure:

package/
  __init__.py
  module.py

Python 2.7

First let's see what happens when I use Python 2.7 (this is the desired behavior)

$ python2 -c "from package import module"
$ find package -name "*.py" -delete
$ python2 -c "from package import module"

It's all good and no errors are thrown. The directory structure after doing this would look like this, with the .pyc files along side the original .py files:

package/
  __init__.pyc
  module.pyc

Python 3.4

Now, let's do the same thing with Python 3.4, starting with our original directory structure again

$ python3 -c "from package import module"
$ find package -name "*.py" -delete
$ python3 -c "from package import module"
 Traceback (most recent call last):
   File "<string>", line 1, in <module>
 ImportError: cannot import name 'module'

Uh oh, it can't import the module. Interestingly, I could still safely run python3 -c "import package" at this point, but it can't grab any modules out of there. At this point, the directory structure looks a little different than it did in 2.7, specifically like this:

package/
  __pycache__/
      __init__.cpython-34.pyc
      module.cpython-34.pyc

So the question is this: why can't Python 3.4 import/execute properly given only .pyc files? Is this desired behavior, meaning that the source has to be kept around in all situations? Or am I missing something stupid?

Answer 1

No enough reputation to add comment to BrenBarn's answer. So here is some complement.

According to the compileall doc:

-b

Write the byte-code files to their legacy locations and names, which may overwrite byte-code files created by another version of Python. The default is to write files to their PEP 3147 locations and names, which allows byte- code files from multiple versions of Python to coexist.

So you could run python -m compileall -b . to compile all the code files in this directory recursively.

Answer 2

According to the PEP:

It's possible that the foo.py file somehow got removed, while leaving the cached pyc file still on the file system. If the __pycache__/foo.<magic>.pyc file exists, but the foo.py file used to create it does not, Python will raise an ImportError when asked to import foo. In other words, Python will not import a pyc file from the cache directory unless the source file exists.

But:

In order to continue to support source-less distributions though, if the source file is missing, Python will import a lone pyc file if it lives where the source file would have been.

So it would seem that __pycache__ and sourceless distributions are mutually exclusive. If you want to remove the source, you need to move the .pyc files out into the directory where the source would have been.