How to read NetCDF variable float data into a Numpy array with the same precision and scale as the original NetCDF float values?

Question

I have a NetCDF file which contains a variable with float values with precision/scale == 7/2, i.e. there are possible values from -99999.99 to 99999.99.

When I take a slice of the values from the NetCDF variable and look at them in in my debugger I see that the values I now have in my array have more precision/scale than what I see in the original NetCDF. For example when I look at the values in the ToosUI/ncdump viewer they display as '-99999.99' or '12.45' but when I look at the values in the slice array they look like '-99999.9921875' (a greater scale length). So if I'm using '-99999.99' as the expected value to indicate a missing data point then I won't get a match with what gets pulled into the slice array since those values have a greater scale length and the additional digits in the scale are not just zeros for padding.

For example I see this if I do a ncdump on a point within the NetCDF dataset:

Variable: precipitation(0:0:1, 40:40:1, 150:150:1)

float precipitation(time=1348, lat=180, lon=360);
  :units = "mm/month";
  :long_name = "precipitation totals";

 data:

  {
    {
      {-99999.99}
    }
  }

However if I get a slice of the data from the variable like so:

value = precipitationVariable[0:1:1, 40:41:1, 150:151:1]

then I see it like this in my debugger (Eclipse/PyDev):

value == ndarray: [[[-99999.9921875]]]

So it seems as if the NetCDF dataset values that I read into a Numpy array are not being read with the same precision/scale of the original values in the NetCDF file. Or perhaps the values within the NetCDF are actually the same as what I'm seeing when I read them, but what's shown to me via ncdump is being truncated due to some format settings in the ncdump program itself.

Can anyone advise as to what's happening here? Thanks in advance for your help.

BTW I'm developing this code using Python 2.7.3 on a Windows XP machine and using the Python module for the NetCDF4 API provided here: https://code.google.com/p/netcdf4-python/

Answer 1

There is no simple way of doing what you want because numpy stores the values as single precision, so they will always have the trailing numbers after 0.99.

However, netCDF already provides a mechanism for missing data (see the best practices guide). How was the netCDF file written in the first place? The missing_value is a special variable attribute that should be used to indicate those values that are missing. In the C and Fortran interfaces, when the file is created all variable values are set to be missing. If you wrote a variable all in one go, you can then set the missing_value attribute to an array of indices where the values are missing. See more about the fill values in the C and Fortran interfaces. This is the recommended approach. The python netCDF4 module plays well with these missing values, and such arrays are read as masked arrays in numpy.

If you must work with the file you currently have, then I'd suggest creating a mask to cover values around your missing value:

import numpy as np
value = precipitationVariable[:]
mask = (value < -99999.98) & (value > -100000.00) 
value = np.ma.MaskedArray(value, mask=mask)