Save Pandas df containing long list as csv file

Question

I am trying to save a pandas dataframe as .csv file. Currently my code looks like this:

with open('File.csv', 'a') as f:
        df.to_csv(f, header=False)

The saving works but the problem is that the lists in my dataframe are just compressed to [first,second,...,last] and all the entries in the middle are discarded. If I just look at the original dataframe all entries are there. Is there any way how I can convert the list to a string which contains all the elements (str(df) also discards the middle elements) or how I can save a full numpy array in a cell of a csv table?

Thank you for your help, Viviane

Answer 1

I had issues while saving dataframes too. I had a dataframe in which some columns consisted of lists as its elements. When I saved the datfarme using df.to_csv and then read it from disk using df.read_csv, the list and arrays were turned into a string of characters. Hence [1,2,3] was transformed to '[1,2,3]'. When I used HDF5 format the problem was solved.

If you dataframe is called df_temp, then you can use:

store = pd.HDFStore('store.h5')
store['df'] = df_temp

to save the dataframe in HDF5 format and you can read it using the following command:

store = pd.HDFStore('store.h5')
df_temp_read = store['df']

You can look at this answer. I should also mention that pickle did not work for me, since I lost the column names when reading from the file. Maybe I did something wrong, but apart from that, pickle can cause compatibility issues if you plan to read the file in different python versions.

Answer 2

Your code should work properly. I couldn't reproduce described behavior.

Here is a bit more "pandaic" version:

df.to_csv('File.csv', header=False, mode='a')

PS pay attention at the mode='a' (append) parameter...

UPDATE:

How to get rid of ellipsis when displaying / printing a DF:

with pd.option_context("display.max_columns", 0):
    print(df)