python limit must be integer

Question

I'm trying to run the following code but for some reason I get the following error: "TypeError: limit must be an integer".

Reading csv data file

import sys
import csv

maxInt = sys.maxsize
decrement = True

while decrement:
    decrement = False
    try:
        **csv.field_size_limit(maxInt)**
    except OverflowError:
        maxInt = int(maxInt/10)
        decrement = True

with open("Data.csv", 'rb') as textfile:
    text = csv.reader(textfile, delimiter=" ", quotechar='|')
    for line in text:
        print ' '.join(line)

The error occurs in the starred line. I have only added the extra bit above the csv read statement as the file was too large to read normally. Alternatively, I could change the file to a text file from csv but I'm not sure whether this will corrupt the data further I can't actually see any of the data as the file is >2GB and hence costly to open.

Any ideas? I'm fairly new to Python but I'd really like to learn a lot more.

Answer 1

I'm not sure whether this qualifies as an answer or not, but here are a few things:

First, the csv reader automatically buffers per line of the CSV, so the file size shouldn't matter too much, 2KB or 2GB, whatever.

What might matter is the number of columns or amount of data inside the fields themselves. If this CSV contains War and Peace in each column, then yeah, you're going to have an issue reading it.

Some ways to potentially debug are to run print sys.maxsize, and to just open up a python interpreter, import sys, csv and then run csv.field_size_limit(sys.maxsize). If you are getting some terribly small number or an exception, you may have a bad install of Python. Otherwise, try to take a simpler version of your file. Maybe the first line, or the first several lines and just 1 column. See if you can reproduce the smallest possible case and remove the variability of your system and the file size.