HDF5 Warnings When Accessing Xarray DataSet

Question

I'd like to understand what is causing the warning messages that I'm getting in the following scenario:

In an earlier operation I've created some NetCDF files and saved them to disk using xarray.to_netcdf().

Lazy evaluation of these datasets is perfectly fine in a jupyter notebook and I receive no warnings/errors when:

• opening these .nc files via ds = xarray.open_mfdataset('/path/to/files/*.nc')
• loading dimension data into memory via ds.time.values
• lazy selection via ds.sel(time=starttime)

I seem to be able to do everything that I want to do in making calculations on memory loaded data. However I often receive the same set of errors when:

• loading data to plot via ds.sel(time=starttime).SCALAR_DATA.plot()
• extracting/loading data via ts = pd.Series(ds.SCALAR_DATA.loc[:,y,x], index=other_data.index)

Note that despite these warnings the operations I perform do result in the desired outcomes (plots, timeseries structures, etc.).

The common denominator in generating the following error seems to be loading data from the opened dataset. EDIT: It seems after some further experimentation that the package versions in my working environment may be causing some conflicts among those dependent on HDF5.

The following errors repeat some number of times.

HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found

...

HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 2:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 476 in H5O__attr_open_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #006: H5Adense.c line 394 in H5A__dense_open(): can't locate attribute in name index
    major: Attribute
    minor: Object not found

Any suggestions on what might be causing these would be greatly appreciated.

Answer 1

These warnings could be caused by netcdf4 version 1.6.X.

Downgrading to netcdf4=1.5.8 fixed the issue in my case.

See also https://github.com/SciTools/iris/issues/5187

Answer 2

I was struggling with very similar errors the past few days and eventually discovered that restricting my dask client to use 1 thread per worker solved the problem, i.e.,:

import xrarray as xr
from dask.distributed import Client
c = Client(n_workers=os.cpu_count()-2, threads_per_worker=1)

ds = xr.open_mfdataset('/path/to/files/*.nc')
ds.sel(.... )

worth a shot if jpolly's solution doesn't work for you (in my case, I'm not using conda...)