1

I have a Python script that attempts to open a grib file as such:

from osgeo import gdal

url = "/vsicurl/https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2"

gdal.Open(url)

This method has worked for me previously using the same data source. However, when I run it on my mac today I get the following error. If I run it on a windows machine, the script works.

ERROR 4: /vsicurl/https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2 
is a grib file, but no raster dataset was successfully identified.

I've tried using combinations of python 3.7, python 3.8, python 2.7, gdal 2.3.3, and gdal 3.1.4. Does anybody know how to fix this error?

2
  • You have tested all the combinations on Windows and on Mac, and with Windows they all work but with Mac they all fail, right? – user30184 May 5 at 18:52
  • i have only tested the combinations on mac, on windows i'm using python 3.7 and gdal 3.1.4 and it works – JWB May 5 at 18:59
2

Running gdalinfo with debugging options reveals the reason for the failure:

gdalinfo /vsicurl/https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t16z.wrfsfcf01.grib2 --debug on --config CPL_CURL_VERBOSE YES

The end of the log:

> GET /data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t16z.wrfsfcf01.grib2 HTTP/1.1
Host: ftpprd.ncep.noaa.gov
Accept: */*
Range: bytes=84213760-84230143

* Mark bundle as not supporting multiuse
< HTTP/1.1  302 Your allowed limit has been reached. Please go to https://www.weather.gov/abusive-user-block  for more info
* no chunk, no close, no size. Assume close to signal end
<
* Closing connection 1
VSICURL: Got response_code=302
GRIB: ERROR: Ran out of file in Section 6
ERROR: Problems Jumping past section 6

ERROR 4: /vsicurl/https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t16z.wrfsfcf01.grib2 is a grib file, but no raster dataset was successfully identified.
gdalinfo failed - unable to open '/vsicurl/https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t16z.wrfsfcf01.grib2'.

GDAL /vsicurl/ system is sending too many requests too fast and the site sets a block. I do not know why your Windows machine is not blocked. Could it be slower or in a slower network?

As a workaround you must download the file and use the local copy.

curl "https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t16z.wrfsfcf01.grib2" -o output.grib2

...
gdalinfo output.grib2
Driver: GRIB/GRIdded Binary (.grb, .grb2)
Files: output.grib2
Size is 1799, 1059
...
5
  • This is really strange because it still works on my windows machine. However I did give you some wrong info, the python version i'm using is 3.9.4 and the gdal version is 3.2.2 on windows. I forgot I was using a conda environment – JWB May 5 at 19:38
  • Is your Windows machine perhaps much slower? It seems that the web site is blocking GDAL because of too many/ too frequent requests. – user30184 May 5 at 20:26
  • they're about the same speed. Also, I don't think I mentioned it, but this was working on my mac yesterday. Do you think it's too much traffic just in general for the server or from me specifically? I've only ran this script a couple of times. – JWB May 5 at 20:49
  • It is not just you. I got the same error yesterday and other GDAL users have confirmed that github.com/OSGeo/gdal/issues/3799. NWS informs that they are constantly monitoring and adjusting the thresholds weather.gov/abusive-user-block. – user30184 May 6 at 6:17
  • Thanks for your help! I'll go with your workaround for now and accept your answer :) – JWB May 6 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.