Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Specify encoding/compression for many variables in xarray dataset when write to_netcdf

I have been writing out some xarray.Datasets that have multiple variables. Currently, in order to keep the size manageable I specify the encoding, e.g. zlib, but needs to be applied on a variable (dataArray) by variable basis.

What is the good way to apply the same encoding argument to all variables? e.g.

<xarray.Dataset>
Dimensions:  (lat: 1440, lon: 2880)
Coordinates:
  * lat      (lat) float64 -90.0 -89.88 -89.75 -89.62 -89.5 -89.38 -89.25 ...
  * lon      (lon) float64 -180.0 -179.9 -179.8 -179.6 -179.5 -179.4 -179.2 ...
Data variables:
a1     (lat, lon) float64 nan nan nan nan nan nan nan nan 0.0 ...
b     (lat, lon) float64 nan nan nan nan nan nan 0.0 0.0 0.0 ...
c     (lat, lon) float64 nan nan nan nan nan nan nan nan 0.0 ...
d      (lat, lon) float64 nan nan nan nan nan nan nan nan 0.0 ...
e      (lat, lon) float64 nan nan nan nan nan nan nan nan 0.0 ...

when writing out this dataset, we would use:

ds.to_netcdf(filename, encoding={'a1':{'zlib': True,'complevel': 5},'b':{'zlib': True,'complevel': 5},'c':{'zlib': True,'complevel': 5},'d':{'zlib': True,'complevel': 5},'e':{'zlib': True,'complevel': 5}})

For more variables this gets very long winded. One option would be to write each dataarray to the netcdf in a sequential append basis, but this also seems convoluted when the dataset is ready to go.

like image 589
dreab Avatar asked Nov 23 '16 13:11

dreab


2 Answers

Or you can store the encoding info for each variable within the Dataset:

comp = dict(zlib=True, complevel=5)
for var in ds.data_vars: 
    var.encoding.update(comp)
ds.to_netcdf(filename)
like image 97
adr Avatar answered Nov 11 '22 14:11

adr


I'd just create the encoding parameter dynamically:

comp = dict(zlib=True, complevel=5)
encoding = {var: comp for var in ds.data_vars}
ds.to_netcdf(filename, encoding=encoding)
like image 24
TomTom101 Avatar answered Nov 11 '22 13:11

TomTom101