Read python pickle data stream in Android

Question

I have this file which contains python pickle data stream. I've to read contents of this file in Android.

For example, if I wanted to read this data stream in python, I'd just use the following code

queue = pickle.load(open('filename', 'rb'))

I want to achieve same thing in Android such that I can read this pickle stream data and store it in some kind of collection.

How can I achieve this?

Answer 1

UPDATE: This only works for pickle protocols 2 and 3.

I think the Unpickler class from Pyrolite (MIT license) may be of particular interest to you. It is technically Java, but Android is basically Java. To unpickle you would do something similar to the following:

InputStream stream = new FileInputStream("filename");
Unpickler unpickler = new Unpickler();
Object data = unpickler.load(stream);
// And cast *data* to the appropriate type.

With the imports:

import java.io.FileInputStream;
import java.io.InputStream;
import net.razorvine.pickle.Unpickler;

These are the objects supported by default:

PYTHON    ---->     JAVA
------              ----
None                null
bool                boolean
int                 int
long                long or BigInteger  (depending on size)
string              String
unicode             String
complex             net.razorvine.pickle.objects.ComplexNumber
datetime.date       java.util.Calendar
datetime.datetime   java.util.Calendar
datetime.time       java.util.Calendar
datetime.timedelta  net.razorvine.pickle.objects.TimeDelta
float               double   (float isn't used) 
array.array         array of appropriate primitive type (char, int, short, long, float, double)
list                java.util.List<Object>
tuple               Object[]
set                 java.util.Set
dict                java.util.Map
bytes               byte[]
bytearray           byte[]
decimal             BigDecimal    
custom class        Map<String, Object>  (dict with class attributes including its name in "__class__")

Please also note:

The unpickler simply returns an Object. Because Java is a statically typed language you will have to cast that to the appropriate type. Refer to this table to see what you can expect to receive.

---

UPDATE: I ran tests using the various pickle protocols (0-3) and found that it fails for 0 and 1, but succeeds for 2 and 3.

Here's the python code used to generate the pickled data:

import pickle

class Data(object):
    def __init__(self):
        self.x = 12

data = Data()

for p in [0, 1, 2]:
    with open('data.{}'.format(p), 'wb') as fh:
        pickle.dump(data, fh, protocol=p)

# Python 3 only.
with open('data.3', 'wb') as fh:
    pickle.dump(data, fh, protocol=3)

And the java code to unpickle it:

import java.io.FileInputStream;
import java.io.InputStream;
import java.io.IOException;
import java.util.Map;
import net.razorvine.pickle.Unpickler;

public class Test {
    public static void main(String[] args) throws IOException {
        String filename = args[0];
        InputStream inputStream = new FileInputStream(filename);
        Unpickler unpickler = new Unpickler();
        Map<String, Object> data = (Map<String, Object>)unpickler.load(inputStream);
    }
}

When run with data.0 and data.1, it fails with:

Exception in thread "main" net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for copy_reg._reconstructor)
  at net.razorvine.pickle.objects.ClassDictConstructor.construct(ClassDictConstructor.java:23)
  at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:617)
  at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:170)
  at net.razorvine.pickle.Unpickler.load(Unpickler.java:84)
  at Test.main(Test.java:13)

When run with data.2 and data.3, it succeeds.