Unpicking data pickled in Python 2.5, in Python 3.1 then uncompressing with zlib

Question

In Python 2.5 I stored data using this code:

def GLWriter(file_name, string):
   import cPickle
   import zlib
   data = zlib.compress(str(string))
   file = open(file_name, 'w')
   cPickle.dump(data, file)

It worked fine, I was able to read that data by doing that process in reverse. It didn't need to be secure, just something that wasn't readable to the human eye. If I put "test" into it and then opened the file it created, it looked like this:

S'x\x9c+I-.\x01\x00\x04]\x01\xc1'
p1
.

For various reasons we're forced to use Python 3.1 now and we need to code something that can read these data files.

Pickle no longer accepts a string input so I've had to open the file with "rb". When I do that and try opening it with pickle.load(file), I get this error:

File "<stdin>", line 1, in <module>
File "C:\Python31\lib\pickle.py", line 1365, in load
   encoding=encoding, errors=errors).load()
UnicodeDecodingError: 'ascii' codec can't decode byte 0x9c in position 1: ordinal not in range(128)

Figuring that I might not be able to open the file in pickle, I started doing some research and found that pickle is just wrapping a few characters each side of the main block of data that zlib is producing. I then tried to trim it down to zlibs output and put that through zlib.decompress. My issue there is that it reads the file and interprets the likes of "\x04" as four characters rather than one. A lot of testing and searching later and I can't find a way to make pickle load the file, or make python recognise these codes so I can put it through zlib.

So my question is this: How can I recover the original data using Python3.1?

I would love to ask my clients to install Python2.5 and do it manually but that's not possible.

Many thanks for your assistance!

Answer 1

The problem is that Python 3 is attempting to convert the pickled Python 2 string into a str object, when you really need it to be bytes. It does this using the ascii codec, which doesn't support all 256 8-bit characters, so you are getting an exception.

You can work around this by using the latin-1 encoding (which supports all 256 characters), and then encoding the string back into bytes:

s = pickle.load(f, encoding='latin1')
b = s.encode('latin1')
print(zlib.decompress(b))