Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

return a list of all datasets in a hdf file with pandas

Tags:

python

pandas

hdf

This may be a stupid question, but i have yet to find an answer in the pandas docs or elsewhere. The same question has been asked before here. But the only answer was to look at the pandas docs, which as I stated don't provide an answer to this problem.

I want to be able to build an hdf file with several datasets. Once this hdf has been closed I would like to be able to list each of the datasets contained within. For example:

import pandas as pd
import numpy as np

store = pd.HDFStore('test.h5')
df1 = pd.DataFrame(np.random.randn(10,2), columns=list('AB')
df2 = pd.DataFrame(np.random.randn(10,2), columns=list('AB')
store['df1'] = df1
store['df2'] = df2
print(store)

Returns:

<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df1           frame          (shape->[10,2])
/df2           frame          (shape->[10,2])

However if you close the hdf with store.close() and then attempt to read it using pd.read_hdf() the following error returns:

ValueError: key must be provided when HDF contains multiple datasets.

Is there a way to return a list of all these datasets?

Thanks in advance for any help!

like image 308
Grr Avatar asked Feb 25 '16 19:02

Grr


People also ask

How do I read a HDF file?

The simplest way to read a binary . hdf/. nc file is to use the program "ncdump" that is distributed as part of the HDF library. The program will return a simple ASCII dump of the HDF file content.

What is HDF in pandas?

Hierarchical Data Format (HDF) is self-describing, allowing an application to interpret the structure and contents of a file with no outside information. One HDF file can hold a mix of related objects which can be accessed as a group or as individual objects.

Can DataFrame hold list?

Data frame columns can contain lists You can also create a data frame having a list as a column using the data.


1 Answers

Yes, there is.

store = pd.HDFStore('test.h5')
print(store)

<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df1           frame          (shape->[10,2])
/df2           frame          (shape->[10,2])
like image 183
Stop harming Monica Avatar answered Sep 19 '22 13:09

Stop harming Monica