reading large JSON file in Python (raw_decode)

Question

I am trying to read in large JSON file (data.json) in Python. Because the JSON file has multiple JSON objects, and multiple dictionaries will be created in Python(the number of dictionaries are unknown), I used decoder.raw_decode() and generator. The following is the code:

import json
import pprint
import io
import pprint

def parse():

    with open('data.json',encoding='utf-8') as jfile:
        try:
            while True:
                decoder = json.JSONDecoder()
                obj, idx = decoder.raw_decode(jfile)
                yield obj
        except ValueError as e:
            print(e)
            pass
        else:
            print("aha")


def main():
    imputd=parse()
    if imputd: 
        while True:
            try:
                print(str(next(imputd)).readlines())
            except StopIteration as e:
                print(e)
                break

main()

I get the error:

Traceback (most recent call last):
  File "H:\Document\Python\j10.py", line 57, in <module>
    main()
  File "H:\Document\Python\j10.py", line 36, in main
    print(str(next(imputd)).readlines())
  File "H:\Document\Python\j10.py", line 21, in parse
    obj, idx = decoder.raw_decode(jfile)
  File "C:\Python34\lib\json\decoder.py", line 360, in raw_decode
    obj, end = self.scan_once(s, idx)
TypeError: first argument must be a string, not _io.TextIOWrapper

I edited code based on Martijn's answer:

import json
import io



file=open('data.json.txt')
def readin():
    return file.read(2000)


def parse():
    decoder = json.JSONDecoder()
    buffer = ''    
    for chunk in iter(readin, ''):
        buffer += chunk
        while buffer:
            try:
                result, index = decoder.raw_decode(buffer)
                yield result
                buffer = buffer[index:]
            except ValueError:
                 # Not enough data to decode, read more
                break

def main():
    imputd=parse()
    if imputd: 
        while True:
            try:
                print(str(next(imputd)).readlines())
            except StopIteration as e:
                print(e)
                break

main()

and I get an UnicodeError:

Traceback (most recent call last):
  File "H:\Document\Python\j11.py", line 35, in <module>
    main()
  File "H:\Document\Python\j11.py", line 30, in main
    print(str(next(imputd)).readlines())
  File "H:\Document\Python\j11.py", line 14, in parse
    for chunk in iter(readin, ''):
  File "H:\Document\Python\j11.py", line 8, in readin
    return file.read(2000)
  File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 4217: character maps to <undefined>

Answer 1

You are passing in the file object, but decoder.raw_decode() only takes text data. You need to do the reading yourself:

obj, idx = decoder.raw_decode(jfile.read())

You are then yielding Python objects created from the JSON data, so your .readlines() call in your main() function loop will also fail.

You are not using raw_decode() correctly, however. You are yourself responsible for feeding it chunks of text, it'll not read that text from the file for you. If you wanted to handle the file in chunks, and there are no clear delimiters between the JSON entries, you'll be forced to read the file in blocks:

decoder = json.JSONDecoder()
buffer = ''
for chunk in iter(partial(jfile.read, buffersize), ''):
     buffer += chunk
     while buffer:
         try:
             result, index = decoder.raw_decode(buffer)
             yield result
             buffer = buffer[index:]
         except ValueError:
             # Not enough data to decode, read more
             break

This will still yield completely decoded objects; if your file is one long JSON object (like one top-level list or dictionary) then this'll not yield the contents of that object one by one; it'll still read the whole object before yielding.