How to change all the dictionary keys in a for loop with d.items()?

Question

I would like some help with understanding why this code is not working as expected.

If one wants to change the key of a dictionary but keep the value, he/she might use:

d[new_key] = d.pop[old_key]

I want to modify all the keys (and keep the values in place) but the code below skips certain lines - ("col2") remains untouched. Is it because dictionaries are unordered and I keep changing the values in it?

How would I go about changing the keys and keep the values without creating a new dictionary?

import time
import pprint

name_dict = {"col1": 973, "col2": "1452 29th Street",
             "col3": "Here is a value", "col4" : "Here is another value",
             "col5" : "NULL", "col6": "Scottsdale",
             "col7": "N/A", "col8" : "41.5946922",
             "col9": "Building", "col10" : "Commercial"}


for k, v in name_dict.items():
    print("This is the key: '%s' and this is the value '%s'\n" % (k, v) )
    new_key = input("Please enter a new key: ")
    name_dict[new_key] = name_dict.pop(k)
    time.sleep(4)

pprint.pprint(name_dict)

Answer 1

It's never a good idea to change the object you're iterating over. Normally dict even throws an exception when you attempt it:

name_dict = {1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6}

for k, v in name_dict.items():
    name_dict.pop(k)

RuntimeError: dictionary changed size during iteration

However in your case you add one item for every removed item. That makes it more convolved. To understand what's happening you need to know that a dictionary is somewhat like a sparse table. For example a dictionary like {1: 1, 3: 3, 5: 5} could look like this (this changed in Python 3.6, for 3.6 and newer the following isn't correct anymore):

hash    key    value
   -      -        - 
   1      1        1
   -      -        - 
   3      3        3
   -      -        - 
   5      5        5
   -      -        - 
   -      -        - 
   -      -        - 

That's also the order in which it is iterated. So in the first iteration it will go to the second item (where the 1: 1 is stored). Let's assume you change the key to 2 and remove the key 1 the dict would look like this:

hash    key    value
   -      -        - 
   -      -        - 
   2      2        1
   3      3        3
   -      -        - 
   5      5        5
   -      -        - 
   -      -        - 
   -      -        - 

But we're still at the second line, so the next iteration it will go to the next "not-empty" entry which is 2: 1. Oups ...

It's even more complicated with strings as keys because string hashes are randomized (on a per session basis) so the order inside the dictionary is unpredictable.

In 3.6 the internal layout was changed a bit but something similar happens here.

Assuming you have this loop:

name_dict = {1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6}

for k, v in name_dict.items():
    # print(k, k+6, name_dict.__sizeof__())
    name_dict[k+6] = name_dict.pop(k)
    # print(name_dict)

The initial layout is like this:

key   value
  1       1
  2       2
  3       3
  4       4
  5       5
  6       1

The first loop removes 1 but adds 7. Because dictionaries are ordered in 3.6 this inserts a placeholder where 1 had been:

key   value
  -       -
  2       2
  3       3
  4       4
  5       5
  6       1
  7       2

This goes on until you replace 4 with 10.

key   value
  -       -
  -       -
  -       -
  -       -
  5       5
  6       1
  7       2
  8       3
  9       4
 10       5

But when you replace 5 with 11 the dictionary will need to increase it's size. Then something special happens: The placeholders are removed:

key   value
  6       6
  7       1
  8       2
  9       3
 10       4
 11       5

So, we were at position 5 in the last iteration and now we change line 6. But line 6 contains 11: 5 right now. Oups...

Never change the object you're iterating over: Don't mess with the keys during iteration (values are okay)!

You could instead keep a "translation table" (don't know if that violates your "without creating a new dict" requirement but you need some kind of storage to make your code work correctly) and do the renaming after the loop:

translate = {}
for k, v in name_dict.items():
    print("This is the key: '%s' and this is the value '%s'\n" % (k, v) )
    new_key = input("Please enter a new key: ")
    translate[k] = new_key
    time.sleep(4)

for old, new in translate.items():
    name_dict[new] = name_dict.pop(old)

Answer 2

in python3 dict.items() is just a view on the dict. as you are not allowed to modify an iterable while iterating you are not allowed to modify a dict while iterating over dict.items(). you have to copy items() to a list before iterating

for k, v in list(name_dict.items()):
    ...
    name_dict[new_key] = name_dict.pop(k)

this does fulfill your 'no new dict' requirement, although the list holds in fact a full copy of all your data.

you could relax the memory footprint a little by copying just the keys

for k in list(name_dict):
    v = name_dict.pop(k)
    ...
    name_dict[new_key] = v

---

EDIT: credits to Sven Krüger, he raised the possibility of a old-key-new-key collision problem. in that case you have to go for

kv = list(name_dict.items())
name_dict.clear()
for k, v in kv :
    ...
    name_dict[new_key] = v

by the way, there is a use case for not creating a new dict, the current one might be referenced somwhere else.