Pandas: how to get status of lines read when using read_csv?

Question

I am loading a very huge csv file, like 10 millions records, using pandas and read_csv method and I wanted to know if there is a way to show the progress of that loading, something like:

100,000 lines read
150,000 lines read

Thanks.

Answer 1

To show progress like this:

Completed 1 %
Completed 2 % 
... 
Completed 99 % 
Completed 100 %

you can try this:

import os, pandas
filename = "VeryLong.csv"
lines_number = sum(1 for line in open(filename))
lines_in_chunk = 500 # I don't know what size is better
counter = 0
completed = 0
reader = pandas.read_csv(filename, chunksize=lines_in_chunk)
for chunk in reader:
    # < ... reading the chunk somehow... >
    # showing progress:
    counter += lines_in_chunk
    new_completed = int(round(float(counter)/lines_number * 100))
    if (new_completed > completed): 
        completed = new_completed
        print "Completed", completed, "%"