I am processing a huge dataset (50 million rows) in CSV. I am trying to slice it and save it as Feather Format in order to save some memory while loading the feather format later.
As a workaround, I loaded the data in chunks as CSV file and later merged it into one data frame.
This is what I have tried so far:
df[2000000:4000000].to_feather('name')
I have got the following error:
ValueError: feather does not support serializing a non-default index for the index; you can .reset_index() to make the index into column(s)
Then I tried to reset the index but still, I get the same error.
Try with .loc
:
df.loc[2000000:4000000].reset_index().to_feather("./myfeather.ftr")
You'll have to reset the indexes to save the datataframe to feather format. Works for me.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With