How to read JSON from socket in python? (Incremental parsing of JSON)

Question

I have a socket opened and I'd like to read some json data from it. The problem is that the json module from standard library can only parse from strings (load only reads the whole file and calls loads inside) It even looks that all the way inside the module it all depends on the parameter being string.

This is a real problem with sockets since you can never read it all to string and you don't know how many bytes to read before you actually parse it.

So my questions are: Is there a (simple and elegant) workaround? Is there another json library that can parse data incrementally? Is it worth writing it myself?

Edit: It is XBMC jsonrpc api. There are no message envelopes, and I have no control over the format. Each message may be on a single line or on several lines. I could write some simple parser that needs only getc function in some form and feed it using s.recv(1), but this doesn't as a very pythonic solution and I'm a little lazy to do that :-)

Answer 1

If you're getting the JSON from an HTTP stream, use the Content-Length header to get the length of the JSON data. For example:

import httplib
import json

h = httplib.HTTPConnection('graph.facebook.com')
h.request('GET', '/19292868552')
response = h.getresponse()
content_length = int(response.getheader('Content-Length','0'))

# Read data until we've read Content-Length bytes or the socket is closed
data = ''
while len(data) < content_length or content_length == 0:
    s = response.read(content_length - len(data))
    if not s:
        break
    data += s

# We now have the full data -- decode it
j = json.loads(data)
print j