Possible bug in GDAL?

Question

Given this file:

http://mtarchive.geol.iastate.edu/2019/02/18/mrms/ncep/GaugeCorr_QPE_01H/GaugeCorr_QPE_01H_00.00_20190218-150000.grib2.gz

… I get different output depending on the software I use.

wgrib2 2019021815.grib2 -csv wgrib2.csv

and

cdo outputtab,date,time,lat,lon,value 2019021815.grib2 > cdo.txt

both output:

-3 (undefined) :  8,869,250 records
0              : 14,848,865 records
Other values   :    781,885 records
___________________________________
Total          : 24,500,000 records

However:

gdal_translate.exe -of xyz 2019021815.grib2 gdal.csv

outputs -3 for all 24,500,000 records.

I get the same result if I parse the file using Python + GDAL.

Is this a bug in GDAL? I'd like to be able to parse these files using Python.

---

I'm using Windows for wgrib2, which can be downloaded from ftp://ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/Windows_7/

I'm also using Windows for GDAL, which can be downloaded from https://gdal.org/download.html

I'm using Linux for cdo, which can be downloaded from https://code.mpimet.mpg.de/projects/cdo/

Answer 1

My answer to your issue would be to switch to using the Weather-Climate Toolkit from NOAA.

Download Here for Windows/Mac/Linux

It can export gridded and radial NEXRAD/MRMS data to CSV. A sample batch command for anyone needing this assistance would be :

wct-export DataSetName.grib2.gz csv Output.csv wctbatchconfig.xml

If it throws errors, try a different/earlier/beta version of WCT from : https://www1.ncdc.noaa.gov/pub/data/wct/dist/ and be sure to check the sample commands on the website.

Answer 2

As you suggest, this seems to be a bug in GDAL. I tested the same command using GDAL 2.4.2, and it works. The change log for 2.4.0 points to several fixes to the GRIB driver.

GRIB driver:

• replace DataSource? and derived classes with VSILFILE directly for > 4GB file support on Windows
• turn printf() warning as CPLDebug() messages
• read and write missing data values correctly for complex packing when original data is integer (#1063)
• g2clib: avoid int overflow. Fixes ​https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=8194
• g2clib: avoid potential out of bound access (CID 1393528)
• degrib: fix floating point division by zero. Fixes ​https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=9407.
• degrib: avoid potential floating point division by zero. Fixes ​https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=10291
• add UNIT[] node to SRS on reading, so that is valid