How to handle a huge stream of JSON dictionaries?

Question

I have a file that contains a stream of JSON dictionaries like this:

{"menu": "a"}{"c": []}{"d": [3, 2]}{"e": "}"}

It also includes nested dictionaries and it looks like I cannot rely on a newline being a separator. I need a parser that could be used like this:

for d in getobjects(f):
  handle_dict(d)

The point is that it would be perfect if the iteration only happened at the root level. Is there a Python parser that would handle all JSON's quirks? I am interested in a solution that would work on files that wouldn't fit into RAM.

Answer 1

I think JSONDecoder.raw_decode may be what you're looking for. You may have to do some string formatting to get it in the perfect format depending on newlines and such, but with a bit of work, you'll probably be able to get something working. See this example.

import json
jstring = '{"menu": "a"}{"c": []}{"d": [3, 2]}{"e": "}"}'
substr = jstring
decoder = json.JSONDecoder()

while len(substr) > 0:
    data,index = decoder.raw_decode(substr)
    print data
    substr = substr[index:]

Gives output:

{u'menu': u'a'}
{u'c': []}
{u'd': [3, 2]}
{u'e': u'}'}