line contains NULL byte error in python csv reader

Question

I am trying to read each line of a csv file and get a "line contains NULL byte" error.

reader = csv.reader(open(mycsv, 'rU'))
for line in reader:
     print(line)


Traceback (most recent call last):
  File "<stdin>", line 1, in <module
_csv.Error: line contains NULL byte

Using the below I found that I have null bytes.

if '\0' in open(mycsv).read():
     print("have null byte")

What's the best way to work around this? Do a replace '\0' on all lines? I need to process this kind of file daily and have about 400,000 lines (1Gb) of data. I assume a replace would substantially slow this down even more.

Answer 1

Try this!

import csv 

def mycsv_reader(csv_reader): 
  while True: 
    try: 
      yield next(csv_reader) 
    except csv.Error: 
      # error handling what you want.
      pass
    continue 
  return

if __name__ == '__main__': 
    reader = mycsv_reader(csv.reader(open(mycsv, 'rU')))
    for line in reader:
        print(line)