Limit on symbol table?

Question

I've been looking into some corner cases for loading python (2.7 on osx) files as configuration files. I wanted to see what the behavior was if I circularly ran execfile. I expected a out of memory error or a lot of swapping to occur, but I was rather surprised when I got a different result.

I setup a test scenario as follows:

'd' python script with:

#!/usr/bin/python
x = 0
execfile("d1")

'd1' python script with:

#!/usr/bin/python
x += 1
print "x = %d" % x
execfile("d2")

'd2' python script with:

#!/usr/bin/python
x += 1
print "x = %d" % x
execfile("d1")

The result:

$ ./d
x = 1
x = 2
x = 3
... removed for brevity ...
x = 997
x = 998
x = 999
Traceback (most recent call last):
  File "./d", line 5, in <module>
    execfile("d1")
  File "d1", line 5, in <module>
    execfile("d2")
  File "d2", line 5, in <module>
    execfile("d1")
... removed for brevity ...
  File "d1", line 5, in <module>
    execfile("d2")
  File "d2", line 5, in <module>
    execfile("d1")
  File "d1", line 5, in <module>
    execfile("d2")
KeyError: 'unknown symbol table entry'

I was just curious if there was someone who could explain what is happening here? Why does it stop after executing execfile ~1000 times?

Answer 1

From the Python source code, Objects/dictobject.c:

/* Note that, for historical reasons, PyDict_GetItem() suppresses all errors
 * that may occur (originally dicts supported only string keys, and exceptions
 * weren't possible).  So, while the original intent was that a NULL return
 * meant the key wasn't present, in reality it can mean that, or that an error
 * (suppressed) occurred while computing the key's hash, or that some error
 * (suppressed) occurred when comparing keys in the dict's internal probe
 * sequence.  A nasty example of the latter is when a Python-coded comparison
 * function hits a stack-depth error, which can cause this to return NULL
 * even if the key is present.
 */

So, PyDict_GetItem() does not always report errors correctly. Interesting... so in the following code in Python/symtable.c,

v = PyDict_GetItem(st->st_blocks, k);
if (v) {
    assert(PySTEntry_Check(v));
    Py_INCREF(v);
}
else {
    PyErr_SetString(PyExc_KeyError,
                    "unknown symbol table entry");
}

Any error that occurs when looking up a symbol (including out-of-memory errors) will be turned into a KeyError. This is probably a bug.