What is the main use of the Python built-in 'compile'?

Question

When looking through the list of Python built-in functions, I struggle with understanding the usefulness the method compile. All of the examples I could find point to a simple "hello world". It make sense what it does, but not when to use it.

Is this the same method Python uses to generate the .pyc files?

Can this be used to remove some of the dynamic nature of Python to improve performance on certain blocks of code? (Knowing full well that a module in C is the way to go precompiled modules.)

Answer 1

To answer the dynamic nature thing, no, not really. The resulting code object is still interpreted; the same bytecode still runs.

The most useful/impressive use of compile() (and code generation) that I've seen is in Jinja2. It actually generates Python code from your templates, then uses compile so it runs at 'native' (== interpreter) speed, if that makes sense.

https://github.com/mitsuhiko/jinja2/blob/master/jinja2/environment.py#L506

In contrast with, say, django.template, which does variable lookups, etc in "userspace" (again, so to speak - metaphor is kind of weird).

https://github.com/django/django/blob/master/django/template/base.py#L752

Answer 2

From here: What's the difference between eval, exec, and compile in Python?:

compile is a lower level version of exec and eval. It does not execute or evaluate your statements or expressions, but returns a code object that can do it. The modes are as follows:

1. compile(string, '', 'eval') returns the code object that would have been executed had you done eval(string). Note that you cannot use statements in this mode; only a (single) expression is valid.
2. compile(string, '', 'exec') returns the code object that would have been executed had you done exec(string). You can use any number of statements here.
3. compile(string, '', 'single') is like the exec mode, but it will ignore everything except for the first statement. Note that an if/else statement with its results is considered a single statement.

UPDATE:

When to compile Python?

Generally you compile Python to take advantage of performance. Compiled code has a much faster startup time since it doesn't have to be compiled, but it doesn't run any faster.

Most notably, you would use compile if you want to convert code into bytecode by hand. This brings up another important, but pertinent question why do this?

As referenced in this magnificent article:

if you want to use exec and you plan on executing that code more than once, make sure you compile it into bytecode first and then execute that bytecode only and only in a new dictionary as namespace.

Of particular note is this:

Now how much faster is executing bytecode over creating bytecode and executing that?:

$ python -mtimeit -s 'code = "a = 2; b = 3; c = a * b"' 'exec code' 10000 loops, best of 3: 22.7 usec per loop

$ python -mtimeit -s 'code = compile("a = 2; b = 3; c = a * b",
"", "exec")' 'exec code' 1000000 loops, best of 3: 0.765 usec per loop

32 times as fast for a very short code example. It becomes a lot worse the more code you have. Why is that the case? Because parsing Python code and converting that into Bytecode is an expensive operation compared to evaluating the bytecode. That of course also affects execfile which totally does not use bytecode caches, how should it. It's not gonna magically check if there is a .pyc file if you are passing the path to a foo.py file.