I am trying to read a CSV file of 1.2G, which contains 25K records, each consists of a id and a large string.
However, around 10K rows, I get this error:
pandas.io.common.CParserError: Error tokenizing data. C error: out of memory
Which seems weird, since the VM has 140GB RAM and at 10K rows the memory usage is only at ~1%.
This is the command I use:
pd.read_csv('file.csv', header=None, names=['id', 'text', 'code'])
I also ran the following dummy program, which could successfully fill up my memory to close to 100%.
list = []
list.append("hello")
while True:
list.append("hello" + list[len(list) - 1])
This is weird.
Actually I ran into the same situation.
df_train = pd.read_csv('./train_set.csv')
But after I tried a lot of stuff to solve this error. And it works. Like this:
dtypes = {'id': pd.np.int8,
'article':pd.np.str,
'word_seg':pd.np.str,
'class':pd.np.int8}
df_train = pd.read_csv('./train_set.csv', dtype=dtypes)
df_test = pd.read_csv('./test_set.csv', dtype=dtypes)
Or this:
ChunkSize = 10000
i = 1
for chunk in pd.read_csv('./train_set.csv', chunksize=ChunkSize): #分块合并
df_train = chunk if i == 1 else pd.concat([df_train, chunk])
print('-->Read Chunk...', i)
i += 1
BUT!!!!!Suddenlly the original version works fine as well!
Like I did some useless work and I still have no idea where really went wrong.
I don't know what to say.
This sounds like a job for chunksize
. It splits the input process into multiple chunks, reducing the required reading memory.
df = pd.DataFrame()
for chunk in pd.read_csv('Check1_900.csv', header=None, names=['id', 'text', 'code'], chunksize=1000):
df = pd.concat([df, chunk], ignore_index=True)
This error can occur with an invalid csv file, rather than the stated memory error.
I got this error with a file that was much smaller than my available RAM and it turned out that there was an opening double quote on one line without a closing double quote.
In this case, you can check the data, or you can change the quoting behavior of the parser, for example by passing quoting=3
to pd.read_csv
.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With