Parsing incomplete json array

Question

I have downloaded 5MB of a very large json file. From this, I need to be able to load that 5MB to generate a preview of the json file. However, the file will probably be incomplete. Here's an example of what it may look like:

[{
    "first": "bob",
    "address": {
        "street": 13301,
        "zip": 1920
    }
}, {
    "first": "sarah",
    "address": {
        "street": 13301,
        "zip": 1920
    }
}, {"first" : "tom"

From here, I'd like to "rebuild it" so that it can parse the first two objects (and ignore the third).

Is there a json parser that can infer or cut off the end of the string to make it parsable? Or perhaps to 'stream' the parsing of the json array, so that when it fails on the last object, I can exit the loop? If not, how could the above be accomplished?

Answer 1

If your data will always look somewhat similar, you could do something like this:

import json

json_string = """[{
    "first": "bob",
    "address": {
        "street": 13301,
        "zip": 1920
    }
}, {
    "first": "sarah",
    "address": {
        "street": 13301,
        "zip": 1920
    }
}, {"first" : "tom"
"""

while True:
    if not json_string:
        raise ValueError("Couldn't fix JSON")
    try:
        data = json.loads(json_string + "]")
    except json.decoder.JSONDecodeError:
        json_string = json_string[:-1]
        continue
    break

print(data)

This assumes that the data is a list of dicts. Step by step, the last character is removed and a missing ] appended. If the new string can be interpreted as JSON, the infinite loop breaks. Otherwise the next character is removed and so on. If there are no characters left ValueError("Couldn't fix JSON") is raised.

For the above example, it prints:

[{'first': 'bob', 'address': {'zip': 1920, 'street': 13301}}, {'first': 'sarah', 'address': {'zip': 1920, 'street': 13301}}]