python h5py file read "OSError: Unable to open file (bad superblock version number)"

Question

Short Description

I have an issue where an hdf5 file has been written on a Windows machine and is unable to be opened on a Linux machine. The error message is "OSError: Unable to open file (bad superblock version number)". (As such, this issue may not be related to h5py at all, but rather a general linux/windows compatibility issue in python file open).

Long Description

A python virtual environment with the following packages was used on both Windows and Linux:

• Flask-0.12.2
• Flask-RESTful-0.3.6
• Jinja2-2.10
• MarkupSafe-1.0
• Werkzeug-0.14.1
• aniso8601-3.0.0
• click-6.7
• h5py-2.7.1
• h5py-cache-1.0
• itsdangerous-0.24
• lockfile-0.12.2
• numpy-1.14.0
• pytz-2018.3
• six-1.11.0

On Windows, the file could be opened and read without issues, but on Linux it couldn't, throwing an OSError. Simply starting a new python session and typing the following is enough:

import h5py

f1 = h5py.File("myfile.hdf5", "r")

Full error:

Traceback (most recent call last):

File "stdin", line 1, in module

File "/usr/local/lib/python3.6/site-packages/h5py/_hl/files.py", line 312, in _ _ init_ _

fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)

File "/usr/local/lib/python3.6/site-packages/h5py/_hl/files.py", line 142, in make_fid

fid = h5f.open(name, flags, fapl=fapl)

File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

File "h5py/h5f.pyx", line 78, in h5py.h5f.open

OSError: Unable to open file (bad superblock version number)

The HDF5 file is written on a Windows machine with Java HDF5 Library from a program that I cannot modify, with hdf5 java 1.10.0 in SWMR (single-write, multiple-reader) mode.

It is possible that the program doesn't close the file properly before sending it further to my program, a light-weight Linux application.

In http://web.mit.edu/fwtools_v3.1.0/www/H5.format.html, the "Version Number of the Super Block" is described as follows...

This value is used to determine the format of the information in the super block. When the format of the information in the super block is changed, the version number is incremented to the next integer and can be used to determine how the information in the super block is formatted.

Values of 0 and 1 are defined for this field.

This field is present in version 0+ of the superblock.

...which doesn't help me understand what the bad superblock version number error could be about.

Here's a sample file I'm trying to open: https://drive.google.com/open?id=10hpbWj4HBwIMq0X6Rq7yVzJATOiYHJcc

Why make a stackoverflow question out of this?

This issue might affect everyone who, on a Linux machine, want to read hdf5 files generated on a Windows machine and not correctly closed/formatted/etc. I'd like to know the reason(s) why this happens and how to work around this on my end, on Linux. If the only solution is "It needs to be fixed by the Windows program generating the HDF5 file, as this cannot be fixed afterwards", then that is an acceptable answer as well. Is that the case here?

Actions Taken

• Upgrading to h5py to 2.8.0rc1 doesn't solve the issue
• Other HDF5 files can be opened as expected

Related Topics

I've looked at the following topics and sites for possible reasons, but have come up empty handed:

1) https://support.hdfgroup.org/HDF5/faq/bkfwd-compat.html

2) h5py OSError: Unable to open file (File signature not found)

3) HDF5 file created with h5py can't be opened by h5py

4) https://github.com/h5py/h5py/issues/757

5) http://web.mit.edu/fwtools_v3.1.0/www/H5.format.html

Edit 1:

Thanks to @Tom de Geus, I tried HDF View on Linux and Windows and discovered that the sample file cannot be opened on Linux HDF View, but it can be opened with Windows HDF View. This suggests that the issue is in the file and HDF, not h5py.

Answer 1

Thanks to Pierre de Buyl for confirming that the file could indeed be opened on Linux with the correct HDF version, 1.10.0. On Windows, it turns out I was running HDF 1.10.1 (installed through h5py), but on Linux the default installation of h5py gave me 1.8.18. After building HDF from sources with cmake, I was able to open the file on Linux as well, with version 1.10.0 or above.

Guide for installing HDF 1.10.0 with cmake: https://support.hdfgroup.org/HDF5/release/cmakebuild.html

The error message "OSError: Unable to open file (bad superblock version number)" was confusing but seems to indicate that the old version of HDF (1.8.18) wasn't configured to recognize the new superblock version number written by HDF version 1.10.0.

Thank you for your help Tom de Geus and Pierre de Buyl :)