Python, write json / dictionary objects to a file iteratively (one at a time)

Question

I have a large for loop in which I create json objects and I would like to be able to stream write the object in each iteration to a file. I would like to be able to use the file later in a similar fashion later (read objects one at a time). My json objects contain newlines and I can't just dump each object as a line in a file. How can I achieve this?

To make it more concrete, consider the following:

for _id in collection:
    dict_obj = build_dict(_id)  # build a dictionary object 
    with open('file.json', 'a') as f:
        stream_dump(dict_obj, f) 

stream_dump is the function that I want.

Note that I don't want to create a large list and dump the whole list using something like json.dump(obj, file). I want to be able to append the object to the file in each iteration.

Thanks.

Answer 1

Since you are generating the files yourself, you can simply write out one JSON object per line:

for _id in collection:
    dict_obj = build_dict(_id)  # build a dictionary object 
    with open('file.json', 'a') as f:
        f.write(json.dumps(dict_obj))
        f.write('\n')

And then read them in by iterating over lines:

with open('file.json', 'r') as f:
    for line in f:
        dict_obj = json.loads(line)

This isn't a great general solution, but it's a simple one if you are both the generator and consumer.