Why does Python compile modules but not the script being run?

Question

Why does Python compile libraries that are used in a script, but not the script being called itself?

For instance,

If there is main.py and module.py, and Python is run by doing python main.py, there will be a compiled file module.pyc but not one for main. Why?

Edit

Adding bounty. I don't think this has been properly answered.

1. If the response is potential disk permissions for the directory of main.py, why does Python compile modules? They are just as likely (if not more likely) to appear in a location where the user does not have write access. Python could compile main if it is writable, or alternatively in another directory.

2. If the reason is that benefits will be minimal, consider the situation when the script will be used a large number of times (such as in a CGI application).

Answer 1

Files are compiled upon import. It isn't a security thing. It is simply that if you import it python saves the output. See this post by Fredrik Lundh on Effbot.

>>>import main # main.pyc is created 

When running a script python will not use the *.pyc file. If you have some other reason you want your script pre-compiled you can use the compileall module.

python -m compileall . 

compileall Usage

python -m compileall --help option --help not recognized usage: python compileall.py [-l] [-f] [-q] [-d destdir] [-x regexp] [directory ...] -l: don't recurse down -f: force rebuild even if timestamps are up-to-date -q: quiet operation -d destdir: purported directory name for error messages    if no directory arguments, -l sys.path is assumed -x regexp: skip files matching the regular expression regexp    the regexp is searched for in the full path of the file 

Answers to Question Edit

1. If the response is potential disk permissions for the directory of main.py, why does Python compile modules?

   Modules and scripts are treated the same. Importing is what triggers the output to be saved.

2. If the reason is that benefits will be minimal, consider the situation when the script will be used a large number of times (such as in a CGI application).

   Using compileall does not solve this. Scripts executed by python will not use the *.pyc unless explicitly called. This has negative side effects, well stated by Glenn Maynard in his answer.

   The example given of a CGI application should really be addressed by using a technique like FastCGI. If you want to eliminate the overhead of compiling your script you may want eliminate the overhead of starting up python too, not to mention database connection overhead.

   A light bootstrap script can be used or even python -c "import script", but these have questionable style.

Glenn Maynard provided some inspiration to correct and improve this answer.

Answer 2

Nobody seems to want to say this, but I'm pretty sure the answer is simply: there's no solid reason for this behavior.

All of the reasons given so far are essentially incorrect:

• There's nothing special about the main file. It's loaded as a module, and shows up in sys.modules like any other module. Running a main script is nothing more than importing it with a module name of __main__.
• There's no problem with failing to save .pyc files due to read-only directories; Python simply ignores it and moves on.
• The benefit of caching a script is the same as that of caching any module: not wasting time recompiling the script every time it's run. The docs acknowledge this explicitly ("Thus, the startup time of a script may be reduced ...").

Another issue to note: if you run python foo.py and foo.pyc exists, it will not be used. You have to explicitly say python foo.pyc. That's a very bad idea: it means Python won't automatically recompile the .pyc file when it's out of sync (due to the .py file changing), so changes to the .py file won't be used until you manually recompile it. It'll also fail outright with a RuntimeError if you upgrade Python and the .pyc file format is no longer compatible, which happens regularly. Normally, this is all handled transparently.

You shouldn't need to move a script to a dummy module and set up a bootstrapping script to trick Python into caching it. That's a hackish workaround.

The only possible (and very unconvincing) reason I can contrive is to avoid your home directory from being cluttered with a bunch of .pyc files. (This isn't a real reason; if that was an actual concern, then .pyc files should be saved as dotfiles.) It's certainly no reason not to even have an option to do this.

Python should definitely be able to cache the main module.