What is this Python list containing all tokens from my code?

Question

If I run this code in Python 3.10:

import gc


def main():
    a = 23764723
    ref = gc.get_referrers(a)[0]
    print(ref)


if __name__ == "__main__":
    main()

I get the following output:

['gc', 'main', 'a', 23764723, 'ref', 'gc', 'get_referrers', 'a', 0, 'print', 'ref', '__main__', '__name__', 'main']

What is this list, that seems to contain all of the literals(?) from my code? Is there an explanation in the Python docs anywhere?

Answer 1

First off, that doesn't look like Python 3.10 output. That output should only happen on Python 3.8 or lower, due to parser changes. 3.10 produces a very different list.

The list you're seeing is the list of objects associated with the arena used to produce your code's abstract syntax tree. Quoting Python/pyarena.c:

/* The arena manages two kinds of memory, blocks of raw memory
   and a list of PyObject* pointers.  PyObjects are decrefed
   when the arena is freed.
*/

Most of the memory used by Python/ast.c is handled by an arena. In particular, every time it needs a new identifier:

static identifier
new_identifier(const char *n, struct compiling *c)
{
    ...
    if (PyArena_AddPyObject(c->c_arena, id) < 0) {

numeric constant:

    case NUMBER: {
        ...
        if (PyArena_AddPyObject(c->c_arena, pynum) < 0) {

or string literal:

/* Make a Constant node, but decref the PyUnicode object being added. */
static expr_ty
make_str_node_and_del(PyObject **str, struct compiling *c, const node* n)
{
    ...
    if (PyArena_AddPyObject(c->c_arena, s) < 0) {

it calls PyArena_AddPyObject to add the identifier, number, or string literal to the arena's list of objects.

When you run your file, execution goes through the pyrun_file function, which looks like this:

static PyObject *
pyrun_file(FILE *fp, PyObject *filename, int start, PyObject *globals,
           PyObject *locals, int closeit, PyCompilerFlags *flags)
{
    PyArena *arena = PyArena_New();
    if (arena == NULL) {
        return NULL;
    }

    mod_ty mod;
    mod = PyParser_ASTFromFileObject(fp, filename, NULL, start, 0, 0,
                                     flags, NULL, arena);
    if (closeit) {
        fclose(fp);
    }

    PyObject *ret;
    if (mod != NULL) {
        ret = run_mod(mod, filename, globals, locals, flags, arena);
    }
    else {
        ret = NULL;
    }
    PyArena_Free(arena);

    return ret;
}

First, it parses your code into an AST, populating the list you see as a side effect:

    mod = PyParser_ASTFromFileObject(fp, filename, NULL, start, 0, 0,
                                     flags, NULL, arena);

Then it calls run_mod to compile the AST and execute the resulting code object:

    if (mod != NULL) {
        ret = run_mod(mod, filename, globals, locals, flags, arena);
    }
    else {
        ret = NULL;
    }

And only after your code has been executed, the arena is freed, freeing the associated list:

    PyArena_Free(arena);