Update json file

Question

I have json file with some data, and would like to occasionally update this file.

I read the file:

with open('index.json', 'rb') as f:
    idx = json.load(f)

then check for presence of a key from potentially new data, and if key is not present update the file:

with open('index.json', mode='a+') as f:
    json.dump(new_data, f, indent=4)

However this procedure just creates new json object (python dict) and appends it as new object in output json file, making the file not valid json file.

Is there any simple way to append new data to json file without overwriting whole file, by updating the initial dict?

Answer 1

One way to do what you're after is to write one JSON object per line in the file. I'm using that approach and it works quite well.

A nice benefit is that you can read the file more efficiently (memory-wise) because you can read one line at a time. If you need all of them, there's no problem with assembling a list in Python, but if you don't you're operating much faster and you can also append.

So to initially write all your objects, you'd do something like this:

with open(json_file_path, "w") as json_file:
    for data in data_iterable:
        json_file.write("{}\n".format(json.dumps(data)))

Then to read efficiently (will consume little memory, no matter the file size):

with open(json_file_path, "r") as json_file:
    for line in json_file:
        data = json.loads(line)
        process_data(data)

To update/append:

with open(json_file_path, "a") as json_file:
    json_file.write("{}\n".format(json.dumps(new_data)))

Hope this helps :)