Creating dynamically named variables in a function in python 3 / Understanding exec / eval / locals in python 3

Question

First of all, let me say that I read the many threads with similar topics on creating dynamically named variables, but they mostly relate to Python 2 or they assume you are working with classes. And yes, I read Behavior of exec function in Python 2 and Python 3.

I'm also aware that creating dynamically named variables is a bad idea in 99% of time and dictionaries are the way to got, but I just want to know whether it is still possible and how exactly exec and locals work in python 3.

I'd like to show a bit of sample code illustrating my question (fibonacci calculates fibonacci numbers, ListOfLetters provides ["A", "B", ...]):

def functionname():
    for index, buchstabe in enumerate(ListOfLetters.create_list("A", "K"), 1): 
        exec("{} = {}".format(buchstabe, fibonacci(index)) ) #A = 1, B = 1, C = 2, D = 3, E = 5,...
        print(index, buchstabe, eval(buchstabe)) #works nicely, e.g. prints "4 D 3"
    print(locals()) #pritns all locals: {'B': 1, 'A': 1, 'index': 11, 'C': 2, 'H': 21, 'K': 89, ...
    print(locals()['K']) #prints 89 as it should
    print(eval("K")) #prints 89 as it should
    print(K) #NameError: name 'K' is not defined

So at least at my current understanding, there is some inconsistency in the behaviour of locals(), since it contains the variable names added by exec() but the variables are not available in the function.

I would be greatful if someone could explain this and tell whether this is by design or if it is a real inconsistency in the language. Yes, I know that locals should not be modified , but I'm not modifying it, I'm calling exec()...

Answer 1

When you're not sure why something works the way it does in Python, it often can help to put the behavior that you're confused by in a function and then disassemble it from the Python bytecode with the dis module.

Lets start with a simpler version of your code:

def foo():
    exec("K = 89")
    print(K)

If you run foo(), you'll get the same exception you're seeing with your more complicated function:

>>> foo()
Traceback (most recent call last):
  File "<pyshell#167>", line 1, in <module>
    foo()
  File "<pyshell#166>", line 3, in foo
    print(K)
NameError: name 'K' is not defined

Lets disassemble it and see why:

>>> import dis
>>> dis.dis(foo)
  2           0 LOAD_GLOBAL              0 (exec)
              3 LOAD_CONST               1 ('K = 89')
              6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
              9 POP_TOP

  3          10 LOAD_GLOBAL              1 (print)
             13 LOAD_GLOBAL              2 (K)
             16 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             19 POP_TOP
             20 LOAD_CONST               0 (None)
             23 RETURN_VALUE

The operation that you need to pay attention to is the one labeled "13". This is where the compiler handles looking up K within the last line of the function (print(K)). It is using the LOAD_GLOBAL opcode, which fails because "K" is not a global variable name, rather it's a value in our locals() dict (added by the exec call).

What if we persuaded the compiler to see K as a local variable (by giving it a value before running the exec), so it will know not to look for a global variable that doesn't exist?

def bar():
    K = None
    exec("K = 89")
    print(K)

This function won't give you an error if you run it, but you won't get the expected value printed out:

>>> bar()
None

Lets disassemble to see why:

>>> dis.dis(bar)
  2           0 LOAD_CONST               0 (None)
              3 STORE_FAST               0 (K)

  3           6 LOAD_GLOBAL              0 (exec)
              9 LOAD_CONST               1 ('K = 89')
             12 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             15 POP_TOP

  4          16 LOAD_GLOBAL              1 (print)
             19 LOAD_FAST                0 (K)
             22 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             25 POP_TOP
             26 LOAD_CONST               0 (None)
             29 RETURN_VALUE

Note the opcodes used at "3" and "19". The Python compiler uses STORE_FAST and LOAD_FAST to put the value for the local variable K into slot 0 and later fetch it back out. Using numbered slots is significantly faster than inserting and fetching values from a dictionary like locals(), which is why the Python compiler does it for all local variable access in a function. You can't overwrite a local variable in a slot by modifying the dictionary returned by locals() (as exec does, if you don't pass it a dict to use for its namespace).

Indeed, lets try a third version of our function, where we peek into locals again when we have K defined as a regular local variable:

def baz():
    K = None
    exec("K = 89")
    print(locals())

You won't see 89 in the output this time either!

>>> baz()
{"K": None}

The reason you see the old K value in locals() is explained in the function's documentation:

Update and return a dictionary representing the current local symbol table.

The slot that the local variable K's value is stored in was not changed by the exec statement, which only modifies the locals() dict. When you call locals() again, Python "update[s]" the dictionary with the value from the slot, replacing the value stored there by exec.

This is why the docs go on to say:

Note: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter.

Your exec call is modifying the locals() dict, and you're finding how its changes are not always seen by your later code.