Are zlib.compress on Python and Deflater.deflate on Java (Android) compatible?

Question

I am porting a Python application to Android and, at some point, this application has to communicate with a Web Service, sending it compressed data.

In order to do that it uses the next method:

def stuff(self, data):
    "Convert into UTF-8 and compress."
    return zlib.compress(simplejson.dumps(data))

I am using the next method to try to emulate this behavior in Android:

private String compressString(String stringToCompress)
{
    Log.i(TAG, "Compressing String " + stringToCompress);
    byte[] input = stringToCompress.getBytes(); 
    // Create the compressor with highest level of compression 
    Deflater compressor = new Deflater(); 
    //compressor.setLevel(Deflater.BEST_COMPRESSION); 
    // Give the compressor the data to compress 
    compressor.setInput(input); 
    compressor.finish(); 
    // Create an expandable byte array to hold the compressed data. 
    // You cannot use an array that's the same size as the orginal because 
    // there is no guarantee that the compressed data will be smaller than 
    // the uncompressed data. 
    ByteArrayOutputStream bos = new ByteArrayOutputStream(input.length); 
    // Compress the data 
    byte[] buf = new byte[1024]; 
    while (!compressor.finished()) 
    { 
        int count = compressor.deflate(buf); 
        bos.write(buf, 0, count); 
    } 

    try { 
        bos.close(); 
    } catch (IOException e) 
    { 

    } 
    // Get the compressed data 
    byte[] compressedData = bos.toByteArray(); 

    Log.i(TAG, "Finished to compress string " + stringToCompress);

    return new String(compressedData);
}

But the HTTP response from the server is not correct and I guess it is because the result of the compression in Java is not the same as the one in Python.

I ran a little test compressing "a" both with zlib.compress and deflate.

Python, zlib.compress() -> x%9CSJT%02%00%01M%00%A6

Android, Deflater.deflate -> H%EF%BF%BDK%04%00%00b%00b

How should I compress the data in Android to obtain the same value of zlib.compress() in Python?

Any help, guidance or pointer is greatly appreciated!

Answer 1

compress and deflate are different compression algorithms so the answer is they will not be compatible. As an example of the difference here is 'a' compressed using the two algorithms via Tcl:

% binary encode hex [zlib compress a]
789c4b040000620062
% binary encode hex [zlib deflate a]
4b0400

Your python code is indeed doing compress. And the android code is doing deflate, however you are also getting the UTF-8 byte order mark prepended to the android version (\xef\xbf\xbf)

You can emit deflate data using python:

def deflate(data):
    zobj = zlib.compressobj(6,zlib.DEFLATED,-zlib.MAX_WBITS,zlib.DEF_MEM_LEVEL,0)
    zdata = zobj.compress(data)
    zdata += zobj.flush()
    return zdata

>>> deflate("a")
'K\x04\x00'

Answer 2

Although they are not exactly the same algorithms, it seems that they are totally compatible (meaning that if you compress, for example, an String using Deflater.deflate you can correctly uncompress it using zlib).

What caused my problem was that all form variables in a POST need to be percent escaped, and the Android application was not doing that. Encoding the data to Base64 before sending it, and modifying the server to decode it using Base64 before uncompressing it using zlib solved the problem.