pickle - putting more than 1 object in a file? [duplicate]

Question

I have got a method which dumps a number of pickled objects (tuples, actually) into a file.

I do not want to put them into one list, I really want to dump several times into the same file. My problem is, how do I load the objects again? The first and second object are just one line long, so this works with readlines. But all the others are longer. naturally, if I try

myob = cpickle.load(g1.readlines()[2])

where g1 is the file, I get an EOF error because my pickled object is longer than one line. Is there a way to get just my pickled object?

Answer 1

If you pass the filehandle directly into pickle you can get the result you want.

import pickle  # write a file f = open("example", "w") pickle.dump(["hello", "world"], f) pickle.dump([2, 3], f) f.close()  f = open("example", "r") value1 = pickle.load(f) value2 = pickle.load(f) f.close() 

pickle.dump will append to the end of the file, so you can call it multiple times to write multiple values.

pickle.load will read only enough from the file to get the first value, leaving the filehandle open and pointed at the start of the next object in the file. The second call will then read the second object, and leave the file pointer at the end of the file. A third call will fail with an EOFError as you'd expect.

Although I used plain old pickle in my example, this technique works just the same with cPickle.

Answer 2

I think the best way is to pack your data into a single object before you store it, and unpack it after loading it. Here's an example using a tuple as the container(you can use dict also):

a = [1,2]
b = [3,4]

with open("tmp.pickle", "wb") as f:
    pickle.dump((a,b), f)

with open("tmp.pickle", "rb") as f:
    a,b = pickle.load(f) 

Answer 3

Don't try reading them back as lines of the file, justpickle.load()the number of objects you want. See my answer to the question How to save an object in Python for an example of doing that.