Error in `python': free(): invalid pointer: 0x00007fc3c90dc98e

Question

Got this error on an ubuntu server (100GB RAM) in anacondas py3.6 installation after it opened multiple files this way in a for loop (7 files * 4GB):

temp_df = pd.read_csv(datafolder + str(file), encoding="ISO-8859-1", delimiter=';',low_memory=False)

this is the error, it appeared after i set

low_memory=False

Doesnt happen when low_memory = True

* Error in `python': free(): invalid pointer: 0x00007fc3c90dc98e *

Anyone ideas? Thanks

Answer 1

This issue is caused due to multiple threads performed parallelly when opening a file , this causes sometimes segmentation fault as well. As you said your file consist of 4 gb each so multiple threads which are executed parallelly to read up your file is causing the issue.