Cannot parse Google Play app ratings data

Question

Google Play ecosystem allows ratings data to be accessible from bucket, i.e. cloud storage. While I can successfully download CSV from Play developer's console and process it, the file is in utf-16 encoding. Here are the first 180 bytes:

'255,254,80,0,97,0,99,0,107,0,97,0,103,0,101,0,32,0,78,0,97,0,109,0,101,0,44,0,65,0,112,0,112,0,32,0,86,0,101,0,114,0,115,0,105,0,111,0,110,0,32,0,67,0,111,0,100,0,101,0,44,0,82,0,101,0,118,0,105,0,101,0,119,0,101,0,114,0,32,0,76,0,97,0,110,0,103,0,117,0,97,0,103,0,101,0,44,0,68,0,101,0,118,0,105,0,99,0,101,0,44,0,82,0,101,0,118,0,105,0,101,0,119,0,32,0,83,0,117,0,98,0,109,0,105,0,116,0,32,0,68,0,97,0,116,0,101,0,32,0,97,0,110,0,100,0,32,0,84,0,105,0,109,0,101,0,44,0,82,0,101,0,118,0,105,0,101,0,119,0'

or decoded:

u'Package Name,App Version Code,Reviewer Language,Device,Review Submit Date and Time,Review Submit Millis Since Epoch,Review Last Update Date and Time,Review Last Update Millis Since'

However, when I try to access ratings via cloud storage:

try:
    stats = cloudstorage.listbucket(folder_path)
except Exception as e:
    raise
files = [stat for stat in stats]
newest = max(files, key=lambda f: f.st_ctime)
newest_ratings_file = cloudstorage.open(newest.filename)
s = newest_ratings_file.read()
newest_ratings_file.close()

, I receive next sequence of bytes (first 180 here):

31,139,8,0,0,0,0,0,0,0,228,189,201,147,44,219,150,222,181,75,74,53,85,42,149,132,74,133,138,82,53,231,189,42,189,71,137,155,85,209,55,247,221,115,239,203,38,154,236,251,150,65,146,125,223,247,73,47,76,48,96,0,162,4,234,104,198,48,131,17,19,16,102,96,134,193,95,112,7,175,94,25,67,13,48,195,140,25,3,224,251,126,219,35,79,158,200,136,188,55,162,210,12,51,220,210,118,100,132,135,251,246,237,238,43,214,222,107,173,111,125,235,255,249,191,103,195,102,216,14,199,122,221,15,187,225,67,152,214,187,83,189,251,34,12,133,11,253,125,8,203,250,116,21,174,195,97,56,15,103,250,60,162,255,59,236,49,175,215,59,109,223,13,247,236,243,33,76,234,232

Which doesn't look like any known encoding to me. Trying to decode using 'utf-8', 'utf-16', 'utf-16-be', 'utf-16-le' leads to decoding errors like:

UnicodeDecodeError: 'utf16' codec can't decode bytes in position 170-171: illegal encoding

CharDetect (https://pypi.python.org/pypi/chardet) didn't help as well :(

I have not much idea where to move next. Guys/Girls, any suggestions how to resolve this ? Muchas gracias!

Answer 1

You can check the file Content-Type and Content-Encoding using Cloud Storage browser or gsutil ls -L <file> to get some clue on how to decode it.

In this case(Content-Encoding: gzip Content-Type:text/csv; charset=utf-16le), the file needs to be unzipped then decoded.