Menu
I'm reading a basic csv file where the columns are separated by commas with these column names:
userid, username, body
However, the body column is a string which may contain commas. Obviously this causes a problem and pandas throws out an error:
CParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 8
Is there a way to tell pandas to ignore commas in a specific column or a way to go around this problem?
785
You need to specify text qualifiers. Generally a double quote (") is used as text qualifiers. All the text is always put inside it and all the commas inside a text qualifier is ignored. This is a standard method for all CSV, languages and all platforms for properly handling the text.
A CSV file is a simple text file where each line contains a list of values (or fields) delimited by commas. Although the term "Comma" appears in the format name itself, but you will encounter CSV files where data is delimited using tab ( \t ) or pipe ( | ) or any other character that can be used as a delimiter.
The pandas DataFrame class supports serializing and de-serializing of CSV in an extenstive way through the read_csv() method. The read_csv() method of pandas DataFrame class reads a CSV file and loads each record as a row in the DataFrame.
If True and parse_dates is enabled, pandas will attempt to infer the format of the datetime strings in the columns, and if it can be inferred, switch to a faster method of parsing them. In some cases this can increase the parsing speed by 5-10x.
Imagine we're reading your dataframe called comma.csv:
userid, username, body 01, n1, 'string1, string2' One thing you can do is to specify the delimiter of the strings in the column with:
df = pd.read_csv('comma.csv', quotechar="'") In this case strings delimited by ' are considered as total, no matter commas inside them.
128
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
