Why are .pyc files created on import?

Question

I've seen several resources describing what .pyc files are and when they're created. But now I'm wondering why they're created when .py files are imported?

Also, why not create a .pyc file for the main Python file doing the importing?

I'm guessing it has to do with performance optimization and learning this has encouraged me to break out my files since the built-in compilation seems nice to take advantage of. But I'm not sure if this is the case, and I'm also curious if anyone has stats for the difference between running programs with and without the .pyc files if it is indeed for speed.

I'd run them myself but I don't have a good, large Python codebase to test it on. :(

Answer 1

Python source code is compiled to bytecode, and it is the bytecode that is run. A .pyc file contains a copy of that bytecode, and by caching that Python doesn't have to re-compile the Python code each time it needs to load the module.

You can get an idea of how much time is saved by timing the compile() function:

>>> import urllib2 >>> import timeit >>> urllib2_source = open(urllib2.__file__.rstrip('c')).read() >>> timeit.timeit("compile(source, '', 'exec')", 'from __main__ import urllib2_source as source', number=1000) 6.977046966552734 >>> _ / 1000.0 0.006977046966552734 

So it takes 7 milliseconds to compile the urllib2.py source code. That doesn't sound like much, but this adds up quickly as Python loads a lot of modules in its lifetime. Just run an average script with the -v command-line switch; here I run the help output for the pydoc tool:

$ bin/python -v -m pydoc -h # installing zipimport hook import zipimport # builtin # installed zipimport hook # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site.py import site # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/os.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/os.py import os # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/os.pyc import errno # builtin import posix # builtin # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/posixpath.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/posixpath.py import posixpath # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/posixpath.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/stat.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/stat.py import stat # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/stat.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/genericpath.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/genericpath.py import genericpath # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/genericpath.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/warnings.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/warnings.py import warnings # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/warnings.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/linecache.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/linecache.py import linecache # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/linecache.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/types.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/types.py import types # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/types.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/UserDict.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/UserDict.py import UserDict # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/UserDict.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/_abcoll.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/_abcoll.py import _abcoll # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/_abcoll.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/abc.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/abc.py import abc # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/abc.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/_weakrefset.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/_weakrefset.py import _weakrefset # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/_weakrefset.pyc import _weakref # builtin # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/copy_reg.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/copy_reg.py import copy_reg # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/copy_reg.pyc import encodings # directory /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/__init__.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/__init__.py import encodings # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/__init__.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/codecs.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/codecs.py import codecs # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/codecs.pyc import _codecs # builtin # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/aliases.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/aliases.py import encodings.aliases # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/aliases.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/utf_8.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/utf_8.py import encodings.utf_8 # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/utf_8.pyc Python 2.7.8 (default, Sep  9 2014, 11:33:29)  [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin Type "help", "copyright", "credits" or "license" for more information. # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/runpy.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/runpy.py import runpy # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/runpy.pyc import imp # builtin # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/pkgutil.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/pkgutil.py import pkgutil # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/pkgutil.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/re.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/re.py import re # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/re.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/sre_compile.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/sre_compile.py import sre_compile # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/sre_compile.pyc import _sre # builtin # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/sre_parse.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/sre_parse.py import sre_parse # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/sre_parse.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/sre_constants.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/sre_constants.py import sre_constants # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/sre_constants.pyc dlopen("/Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/_locale.so", 2); import _locale # dynamically loaded from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/_locale.so # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/inspect.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/inspect.py import inspect # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/inspect.pyc # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/string.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/string.py import string # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/string.pyc dlopen("/Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/strop.so", 2); import strop # dynamically loaded from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/strop.so # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/dis.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/dis.py import dis # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/dis.pyc # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/opcode.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/opcode.py import opcode # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/opcode.pyc # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/tokenize.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/tokenize.py import tokenize # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/tokenize.pyc dlopen("/Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/itertools.so", 2); import itertools # dynamically loaded from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/itertools.so # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/token.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/token.py import token # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/token.pyc dlopen("/Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/operator.so", 2); import operator # dynamically loaded from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/operator.so # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/collections.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/collections.py import collections # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/collections.pyc dlopen("/Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/_collections.so", 2); import _collections # dynamically loaded from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/_collections.so # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/keyword.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/keyword.py import keyword # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/keyword.pyc # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/heapq.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/heapq.py import heapq # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/heapq.pyc dlopen("/Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/_heapq.so", 2); import _heapq # dynamically loaded from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/_heapq.so import thread # builtin # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/repr.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/repr.py import repr # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/repr.pyc # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/traceback.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/traceback.py import traceback # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/traceback.pyc # /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/locale.pyc matches /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/locale.py import locale # precompiled from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/locale.pyc # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/functools.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/functools.py import functools # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/functools.pyc dlopen("/Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/_functools.so", 2); import _functools # dynamically loaded from /Users/mpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/lib-dynload/_functools.so # /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/getopt.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/getopt.py import getopt # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/getopt.pyc pydoc - the Python documentation tool  pydoc.py <name> ...     Show text documentation on something.  <name> may be the name of a     Python keyword, topic, function, module, or package, or a dotted     reference to a class or function within a module or module in a     package.  If <name> contains a '/', it is used as the path to a     Python source file to document. If name is 'keywords', 'topics',     or 'modules', a listing of these things is displayed.  pydoc.py -k <keyword>     Search for a keyword in the synopsis lines of all available modules.  pydoc.py -p <port>     Start an HTTP server on the given port on the local machine.  pydoc.py -g     Pop up a graphical interface for finding and serving documentation.  pydoc.py -w <name> ...     Write out the HTML documentation for a module to a file in the current     directory.  If <name> contains a '/', it is treated as a filename; if     it names a directory, documentation is written for all the contents.  # clear __builtin__._ # clear sys.path # clear sys.argv # clear sys.ps1 # clear sys.ps2 # clear sys.exitfunc # clear sys.exc_type # clear sys.exc_value # clear sys.exc_traceback # clear sys.last_type # clear sys.last_value # clear sys.last_traceback # clear sys.path_hooks # clear sys.path_importer_cache # clear sys.meta_path # clear sys.flags # clear sys.float_info # restore sys.stdin # restore sys.stdout # restore sys.stderr # cleanup __main__ # cleanup[1] _collections # cleanup[1] locale # cleanup[1] functools # cleanup[1] encodings # cleanup[1] site # cleanup[1] runpy # cleanup[1] operator # cleanup[1] supervisor # cleanup[1] _heapq # cleanup[1] abc # cleanup[1] _weakrefset # cleanup[1] sre_constants # cleanup[1] collections # cleanup[1] _codecs # cleanup[1] opcode # cleanup[1] _warnings # cleanup[1] mpl_toolkits # cleanup[1] inspect # cleanup[1] encodings.utf_8 # cleanup[1] repr # cleanup[1] codecs # cleanup[1] getopt # cleanup[1] pkgutil # cleanup[1] _functools # cleanup[1] thread # cleanup[1] keyword # cleanup[1] strop # cleanup[1] signal # cleanup[1] traceback # cleanup[1] itertools # cleanup[1] posix # cleanup[1] encodings.aliases # cleanup[1] exceptions # cleanup[1] _weakref # cleanup[1] token # cleanup[1] dis # cleanup[1] tokenize # cleanup[1] heapq # cleanup[1] string # cleanup[1] imp # cleanup[1] zipimport # cleanup[1] re # cleanup[1] _locale # cleanup[1] sre_compile # cleanup[1] _sre # cleanup[1] sre_parse # cleanup[2] copy_reg # cleanup[2] posixpath # cleanup[2] errno # cleanup[2] _abcoll # cleanup[2] types # cleanup[2] genericpath # cleanup[2] stat # cleanup[2] warnings # cleanup[2] UserDict # cleanup[2] os.path # cleanup[2] linecache # cleanup[2] os # cleanup sys # cleanup __builtin__ # cleanup ints: 21 unfreed ints # cleanup floats 

That's 53 imports:

$ bin/python -v -m pydoc -h 2>&1 | egrep ^import | wc -l       53 

rather than load the (larger) source file each time and compiling it, a smaller bytecode file can be read and used immediately. That easily adds up to 1/3rd or 1/2 a second just to print some help information for a command-line tool.

Python does not create a cache file for the main script; that's because that would clutter up your scripts directory with files that are not going to be loaded nearly as often as modules are loaded.

If you run a script that often that the compile time for that one file affects you, you can always either move the majority of the code to a module (and avoid having to compile a large script) or you can use the compileall tool to create a .pyc cache file for the script, then run that .pyc file directly. Note that Python then will not recompile that file if you changed the script!