HDFStore error appending - "Cannot serialize the column"

Question

I have a dataframe, df:

    datetime                      bid      ask     bidvolume  askvolume
0   2007-03-30 21:00:00.332000   1.9682   1.9678       4         0.8

Trying to append this to a new datastore. The datastore does not exist so I use the following to create and append the data;

store = pd.HDFStore(storePath,mode='w')
store.append('data',df)
store.close()

I get this error: on the store.append line.

TypeError: Cannot serialize the column [bid] because
its data contents are [floating] object dtype

How do I get the data to store properly?

Answer 1

Please note: the following method convert_objects() is now deprecated and may not work Call DataFrame.convert_objects():

df = DataFrame(randn(10, 1), dtype=object).convert_objects()
df.to_hdf('/tmp/blah.h5', 'df', append=True)

It might be worth checking to see if you can get your data in the correct format before you start saving to HDF5. For example, wherever df is created, convert the objects there, instead of converting them when you save. In general, operations in pandas will be very cumbersome with a Series of floats with a dtype of object. Your life will be much easier if you convert your object arrays (where possible) as soon as you need to do anything with them.