Python json.dump produces file of size zero bytes

Question

At some stage in my code, I need to dump a python dictionary into a json file. Here is the code that does that:

def output_batch(self, batch):
    with open(os.path.join(self.get_current_job_directory(), 'batch_' + str(self.index) + '.json'), 'w') as json_file:
        json.dump(batch, json_file, sort_keys = True, indent = 4, separators = (',', ': '))
    exit()

Here self.index is a unique integer index that identifies this specific instance. self.get_current_job_directory() returns the path to were the file should be created.

Logging the length of the dictionary just before dumping it returns 42691. However the output file has a size of zero bytes. This doesn't really make any sense. Even in case of an empty dict the file would be 2 bytes long (to compensate for the {}).

The code is part of a highly parallelized map-reduce style pipeline and is almost impossible to reproduce here. Now there are a million things that can go wrong in such a setup, but no other thread has access to the dict being dumped and no other thread would access the same output file.

*** Update: The process exits (by calling exit()) right after this function call, maybe has got something to do with this?

Any suggestion or ideas on why this might happen?

Answer 1

It could be that the with context isn't being exited correctly being at the end of the function. Try to close the file explicitly.

def output_batch(self, batch):
    json_file = open(...):
    json.dump(batch, json_file, sort_keys = True, indent = 4, separators = (',', ': '))
    json_file.close()

Moreover, make sure you call open('w') with writing permissions, and that you have permissions to write to that place.