python should I use generator for this case?

Question

I have a list of almost 2k dictionaries inside it. And I am using the list several times. For instance:

c = myClass()
c.create(source) # where source is a text of approximately 50k chars
                 # this method creates the list that has approximately 2k dictionaries
item = c.get(15012) # now, this one loops thru the list to find an item
                    # whenever the condition is matched, the for loop is broken and the value is returned
item2 = c.prevItem(item) # this one also loops thru the list by reversing it and bringing the next item

Now, imagine this scenario where I have the use the same list over and over again. Since the list is large I'd like to use a generator but as far as I've understood, generators have to be recreated when they throw StopIteration. So basically, in this scenario, is it convenient to use a generator or is there a more efficient way in terms of speed?

Answer 1

It sounds to me like you have to decide which you'd rather do:

1) Save the values so you don't have to recalculate them, but use more space to do so.

2) Recalculate them each time, but save on space because you don't have to store them.

If you think about it, no matter what kind of generator/list/whatever you're using, one of those two things has to happen. And I don't think there's a simple hard rule to say which is better. (Personally I'd say pick one and don't look back. You have your whole life ahead of you.)

Answer 2

If you frequently get an item at a known offset from a previously retrieved item, is to change .get to return not only the item, but it's position in the list. Then you could implement prevItem as:

def previtem(self, pos):
    return self.itemlist[pos - 1]

item, pos = c.get(itemnum)
item2 = c.prevItem(pos)

If, instead, you are doing some sort of operation on item to get a new itemnum, you should store them in a dict instead of a list. This way, get is just a dictionary lookup (much faster than list search):

def get(self, itemnum):
    return self.premade_dict[itemnum]

So one way or the other you should be able to replace some searches with cheaper operations.