Load data from Python pickle file in a loop?

Question

In a small data-acquisition project we use the Python's pickle to store recorded data, i.e. for each "event" we add it to the output file f with

pkl.dump(event, f, pkl.HIGHEST_PROTOCOL)

where import cPickle as pkl.

In the analysis of the data we read each event, but in contrast to a normal file where processing can be one in a rather elegant way:

with open(filename) as f:
    for line in f:
        do_something()

looping over all the data in a pickle file this becomes a bit more awkward:

with open(filename) as f:
    try:
        while True:
            event = pkl.load(f)
            do_something()
    except (EOFError, UnpicklingError):
        pass

Is it possible to make pickle reading more like the example for regular files above?

Answer 1

Yes, indeed. Use this generator below to make the events readable in a loop:

def pickleLoader(pklFile):
    try:
        while True:
            yield pkl.load(pklFile)
    except EOFError:
        pass

Now you can simply write:

with open(filename) as f:
    for event in pickleLoader(f):
        do_something()