Dictionary changed size during iteration - Code works in Py2 Not in Py3

Question

I have the following sample code:

k_list = ['test', 'test1', 'test3']

def test(*args, **kwargs):
    for k, value in kwargs.items():
        if k in k_list:
            print("Popping k = ", k)
            kwargs.pop(k, None)
    print("Remaining KWARGS:", kwargs.items())

test(test='test', test1='test1', test2='test2', test3='test3')

In Python 2.7.13 this prints exactly what I expect and still has an item left in the kwargs:

('Popping k = ', 'test')
('Popping k = ', 'test1')
('Popping k = ', 'test3')
('Remaining KWARGS:', [('test2', 'test2')])

In Python 3.6.1, however, this fails:

Popping k =  test
Traceback (most recent call last):
  File "test1.py", line 11, in <module>
    test(test='test', test1='test1', test2='test2', test3='test3')
  File "test1.py", line 5, in test
    for k, value in kwargs.items():
RuntimeError: dictionary changed size during iteration

What do I need to adjust to maintain the Python 2 compatibility but work correctly in Python 3.6? The remaining kwargs will be used for later logic in my script.

Answer 1

The reason that it works in python2.x is because kwargs.items() creates a list -- You can think of it as a snapshot of the dictionary's key-value pairs. Since it is a snapshot, you can change the dictionary without modifying the snapshot that you're iterating over and everything is OK.

In python3.x, kwargs.items() creates a view into the dictionary's key-value pairs. Since it is a view, you can no longer change the dictionary without also changing the view. This is why you get an error in python3.x

One resolution which will work on both python2.x and python3.x is to always create a snapshot using the list builtin:

for k, value in list(kwargs.items()):
    ...

Or, alternatively, create a snapshot by copying the dict:

for k, value in kwargs.copy().items():
    ...

This will work. In a very unscientific experiement that I did in my interactive interpreter, the first version is a fair amount faster than the second on python2.x. Also note that this whole thing will be slightly inefficient on python2.x because you'll be creating an addition copy of something (either a list or dict depending on which version you reference). Based on your other code, that doesn't look like too much of a concern. If it is, you can use something like six for compatibility:

for k, value in list(six.iteritems(kwargs)):
    ...

Answer 2

In Python 3, dict.items was changed into a view, where previously it returned a copy. To work with a copy again, and get cross-compat code, you may change this line:

for k, value in kwargs.items():

To this:

for k, value in list(kwargs.items()):

Answer 3

I would simply not use a loop, but sets:

# Your setup
kwargs = dict(test='test', test1='test1', test2='test2', test3='test3')
k_list = ['test', 'test1', 'test3']

# How to extract unwanted keys:
common_keys = set(kwargs) & set(k_list)
kwargs = { k: kwargs[k] for k in kwargs if k not in common_keys }

In the last line I did not use kwargs.items() or .iteritems(), so the example works well in both versions. But you might want to use .items() in production if you only target Python 3.