Why the object, which I read a csv file using pandas from, is TextFileReader object

Question

I read a csv file using pandas:

data_raw = pd.read_csv(filename, chunksize=chunksize)
print(data_raw['id'])

Then, it reports TypeError:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'TextFileReader' object has no attribute '__getitem__'

What can I do to resolve the problem? And how can I change the data_raw into a dataFrame object? I use the python2.7 and pandas v0.19.1

Answer 1

One way around this problem is to set nrows parameter in pd.read_csv() function and that way you select subset of data you want to load into the dataframe. Of course, drawback is that you wont be able to see and work with full dataset. Code example:

data = pd.read_csv(filename, nrows=100000)

Answer 2

When you pass chunksize option to read_csv(), it creates a TextFileReader reader - an open-file-like object that can be used to read the original file in chunks. See usage example here: How to read a 6 GB csv file with pandas When this option is not provided, the function indeed reads the file content.