Python dictionary eating up ram

Question

I'm pretty new to programming and made a program to fetch inventory data from Team Fortress 2 players and put the inventory items into a dictionary with the steamid as the key and the list of items as the value.

The problem I'm running into is that after about 6000 entries into the dictionary the program has sucked up essentially all of the RAM on my system and shuts down.

I'm guessing the dictionary simply becomes too large but by what I've read from similar questions a dict of 6000 entries shouldn't take up that much of my RAM.

I've been looking into other solutions but I could use some concrete examples for my code.

import re, urllib.request, urllib.error, gzip, io, json, socket, sys

with open("index_to_name.json", "r", encoding=("utf-8")) as fp:
    index_to_name=json.load(fp)

with open("index_to_quality.json", "r", encoding=("utf-8")) as fp:
    index_to_quality=json.load(fp)

with open("index_to_name_no_the.json", "r", encoding=("utf-8")) as fp:
    index_to_name_no_the=json.load(fp)

with open("steamprofiler.json", "r", encoding=("utf-8")) as fp:
    steamprofiler=json.load(fp)

inventory=dict()
playerinventories=dict()
c=0

for steamid in steamprofiler:
    emptyitems=[]
    items=emptyitems
    try:
        url=urllib.request.urlopen("http://api.steampowered.com/IEconItems_440/GetPlayerItems/v0001/?key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&steamid="+steamid+"&format=json")
        inv=json.loads(url.read().decode("utf-8"))
        url.close()
    except (urllib.error.HTTPError, urllib.error.URLError, socket.error) as e:
        c+=1
        print("URL/HTTP error, continuing")
        continue
    try:
        for r in inv["result"]["items"]:
            inventory[r["id"]]=r["quality"], r["defindex"]
    except KeyError:
        c+=1
        print(steamid, "didn't have an inventory")
        continue
    for key in inventory:
        try:
            if index_to_quality[str(inventory[key][0])]=="":
                items.append(
                    index_to_quality[str(inventory[key][0])]
                    +""+
                    index_to_name[str(inventory[key][1])]
                    )
            else:
                items.append(
                    index_to_quality[str(inventory[key][0])]
                    +" "+
                    index_to_name_no_the[str(inventory[key][1])]
                    )
        except KeyError:
            print("Key error, uppdate def_to_index")
            c+=1
            continue
playerinventories[int(steamid)]=items
items=emptyitems
c+=1
print(c, "inventories fetched")

I don't really know of any other way to do it while retaining the dictionary appearence, which is quite important as I'd like to be able to tell whose inventory it is. If I've been unclear in any of this, just say so and I'll try to explain

Answer 1

I think you have some logic errors in your code. For instance, you're adding each player's inventory items to the inventory dictionary, then iterating over it to fill in other stuff.

However, you're never resetting the inventory dictionary, so it continues to accumulate items (so the second player will appear to have the first person's inventory in addition to their own).

You've got a similar issue with the items dictionary that you're using a bit later on. You reset it to emptyitems which was originally an empty list, but because assignment in Python is by reference, this has no effect (items was already the same object as emptyitems).

With those two fixes you may have a better chance at not using all your system's memory.

Another miscellaneous code improvement (probably not related to memory usage):

In your loop over inventory, you're repeatedly accessing the same two values and not using the key for anything. Instead of for key in inventory try for value1, value2 in inventory.itervalues() (or in inventory.values() if you're using Python 3). Then use value1 in place of inventory[key][0] and value2 in place of inventory[key][1] (or even better, give them more meaningful names).

Edit: Here's how the loop might look (I'm sort of guessing at names for the two values that were previously in inventory[key][0] and inventory[key][1]):

for quality, name in inventory.itervalues():
    try:
        if index_to_quality[str(quality)]=="":
            items.append(
                index_to_quality[str(quality)]
                +""+
                index_to_name[str(name)]
                )
        else:
            items.append(
                index_to_quality[str(quality)]
                +" "+
                index_to_name_no_the[str(name)]
                )