Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how to NOT read_csv if csv is empty

Tags:

python

pandas

Using Python 2.7 and Pandas

I have to parse through my directory and plot a bunch of CSVs. If the CSV is empty, the script breaks and produces the error message:

pandas.io.common.EmptyDataError: No columns to parse from file

If I have my file paths stored in

file_paths=[]

how do I read through each one and only plot the non empty CSVs? If I have an empty dataframe defined as df=[] I attempt the following code

for i in range(0,len(file_paths)):
   if pd.read_csv(file_paths[i] == ""):
      print "empty"
   else df.append(pd.read_csv(file_paths[i],header=None))
like image 887
Devin Liner Avatar asked Feb 09 '17 17:02

Devin Liner


People also ask

How do I save a CSV file without indexing?

pandas DataFrame to CSV with no index can be done by using index=False param of to_csv() method. With this, you can specify ignore index while writing/exporting DataFrame to CSV file.

What does CSV in read_csv () stand for?

A comma-separated values (csv) file is returned as two-dimensional data structure with labeled axes. See also DataFrame.to_csv. Write DataFrame to a comma-separated values (csv) file. read_csv.


2 Answers

I would just catch the appropriate exception, as a catch all is not recommended in python:

import pandas.io.common

for i in range(0,len(file_paths)):
   try:
      pd.read_csv(file_paths[i])
   except pandas.io.common.EmptyDataError:
      print file_paths[i], " is empty"
like image 88
Zeugma Avatar answered Sep 18 '22 19:09

Zeugma


Note, as of pandas 0.22.0 (that I can be sure of) , the exception raised for empty csv is pandas.errors.EmptyDataError. And if you're importing pandas like import pandas as pd, then use pd instead of pandas.

If your csv filenames are in an array manyfiles, then

import pandas as pd
for filename in manyfiles:
    try:
        df = pd.read_csv(filename)

    except pd.errors.EmptyDataError:
        print('Note: filename.csv was empty. Skipping.')
        continue # will skip the rest of the block and move to next file

    # operations on df

I'm not sure if pandas.io.common.EmptyDataError is still valid or not. Can't find it in reference docs. And I also would advise against the catch-all except: as you won't be able to know if it's something else causing the issue.

like image 34
Nikhil VJ Avatar answered Sep 18 '22 19:09

Nikhil VJ