0

I am attempting to download global SoilGrids data, following the tutorial for R. I want to download a full global map at 250 m resolution. Eventually I want to reproject it to EPSG:4326 but first I am just working on converting to a locally-saved tiff.

I can download the data at 1000 m resolution, but when I set it to 500 m it fails. This occurs whether I use the gdalUtils or the gdalUtilities packages. With gdalUtils I am able to specify the gdal installation (2.4.1), with gdalUtilities it defaults to a more recent version (I think it's using 3.0.4 but it may also be finding a more recent version).

library(rgdal)
library(gdalUtils)
library(gdalUtilities)


#path to GDAL 2.4.2 installation
gdalUtils::gdal_setInstallation("C:\\OSGeo4W64\\bin", rescan = TRUE)
getOption("gdalUtils_gdalPath")

igh='+proj=igh +lat_0=0 +lon_0=0 +datum=WGS84 +units=m +no_defs' # proj string for Homolosine projection

################################
# This works:
gdalUtils::gdal_translate(
  '/vsicurl?max_retry=3&retry_delay=1&list_dir=no&url=https://files.isric.org/soilgrids/latest/data/ocs/ocs_0-30cm_mean.vrt',
  "./ocs_0-30cm_mean_igh_2.tif",
  tr=c(1000,1000)
)

# This doesn't:
gdalUtils::gdal_translate(
  '/vsicurl?max_retry=3&retry_delay=1&list_dir=no&url=https://files.isric.org/soilgrids/latest/data/ocs/ocs_0-30cm_mean.vrt',
  "./ocs_0-30cm_mean_igh_3.tif",
  tr=c(500,500)
)

# gdalUtilities calls GDAL 3.0.1
# This works:
gdalUtilities::gdal_translate(
  '/vsicurl?max_retry=3&retry_delay=1&list_dir=no&url=https://files.isric.org/soilgrids/latest/data/ocs/ocs_0-30cm_mean.vrt',
  "./ocs_0-30cm_mean_igh_2b.tif",
  tr=c(1000,1000)
)

# This doesn't:
gdalUtilities::gdal_translate(
  '/vsicurl?max_retry=3&retry_delay=1&list_dir=no&url=https://files.isric.org/soilgrids/latest/data/ocs/ocs_0-30cm_mean.vrt',
  "./ocs_0-30cm_mean_igh_3b.tif",
  tr=c(500,500)
)

Using gdalUtils I get an unhelpful error:

Warning message: In system(cmd, intern = TRUE) : running command '"C:\OSGeo4W64\bin\gdal_translate.exe" -tr 500 500 -of "GTiff" "/vsicurl?max_retry=3&retry_delay=1&list_dir=no&url=https://files.isric.org/soilgrids/latest/data/ocs/ocs_0-30cm_mean.vrt" "./ocs_0-30cm_mean_igh_3.tif"' had status 1

Using gdalUtilities I get (maybe?) more helpful error message. Note that it seems to be searching for a file path that contains "\./".

Error in gdal_utils("translate", src_dataset, dst_dataset, opts) :
gdal_utils translate: an error occured In addition: Warning message: In CPL_gdaltranslate(source, destination, options, oo, quiet) : GDAL Error 4: `/vsicurl?max_retry=3&retry_delay=1&list_dir=no&url=https://files.isric.org/soilgrids/latest/data/ocs\./ocs_0-30cm_mean/tileSG-000-023/tileSG-000-023_3-1.tif' does not exist in the file system, and is not recognized as a supported dataset name.

A last note that running the command gdal_translate -tr 1000 1000 https://files.isric.org/soilgrids/latest/data/ocs/ocs_0-30cm_mean.vrt ./ocs_0-30cm_mean_igh_3.tif --config GDAL_HTTP_UNSAFESSL YES within an OSGeo4W shell also produces an error:

Input file size is 159246, 58034 0ERROR 4: No such file or directory

2 Answers 2

1

I was able to read in the data using gdalwarp after downloading the entire file structure for a .vrt file to my computer. Trying to read the files directly from the WebDAV server must have been too large of a request to complete before timing out/dropping the connection.

I used wget for windows to download the files, and it took about 4 hours for each map (~15000 files, ~5 GB)

wget https://files.isric.org/soilgrids/latest/data/soc/soc_0-5cm_mean/ -r -np -nH --cut-dirs=3 -nc
wget https://files.isric.org/soilgrids/latest/data/soc/soc_0-5cm_mean.vrt -O ./soc/soc_0-5cm_mean.vrt

Then I was able to run the following command without a problem, converting the entire map to a tiff in EPSG 4326 at 7.5" resolution:

gdalUtilities::gdalwarp(t_srs="EPSG:4326", multi=TRUE, wm=200, 
         co=c("BIGTIFF=YES", "COMPRESS=DEFLATE", "TILED=TRUE"),
         tr=c(1/480,1/480), # Desired output resolution
         "soc_0-5cm_mean.vrt", # Input VRT
         "soc_0-5cm_mean.tif") # Output file

Unsurprisingly, this command also took quite a long time to run (~45 minutes).

0

I have to admit I'm not familiar with this format and gdalUtils::gdal_translate() at all, but let me share my trial & error attempts nevertheless. Maybe it will be of help, maybe not, and maybe this might encourage others to answer something more sophisticated.

  1. In order to address the present error messages, the file to be queried does exist and can of course be acquired manually using a browser of your choice.

  2. It seems like the file can be read directly using {terra}, but as soon as you try to access raster values, the error messages says "no such file or directory" - just like your error message coming from {gdalUtilities}:

library(terra)
#> terra 1.6.49

# this seems to work
rast("https://files.isric.org/soilgrids/latest/data/ocs/ocs_0-30cm_mean.vrt")
#> class       : SpatRaster 
#> dimensions  : 58034, 159246, 1  (nrow, ncol, nlyr)
#> resolution  : 250, 250  (x, y)
#> extent      : -19949750, 19861750, -6147500, 8361000  (xmin, xmax, ymin, ymax)
#> coord. ref. : Interrupted_Goode_Homolosine 
#> source      : ocs_0-30cm_mean.vrt 
#> name        : ocs_0-30cm_mean 
#> min value   :               6 
#> max value   :             208

# this does not
rast("https://files.isric.org/soilgrids/latest/data/ocs/ocs_0-30cm_mean.vrt") |> plot()
#> Warning: No such file or directory (GDAL error 4)
#> Warning: [plot] SpatRaster has no cell values

I assume this to be expected behaviour connected to the file format, consisting of a meta data shell, just linking to individual tiles holding real values.

  1. If you check the server directory in a browser, you'll notice a *.vrt.ovr file for every *.vrt file listed, as well as a folder containing lots of individual tiles, e.g. ocs_0-30cm_mean.

  2. Downloading a local copy of the vrt file in order to execute rast("ocs_0-30cm_mean.vrt") shows the same behaviour as described in 2) since the linked tiles are not available locally.

  3. If you downloaded a local copy of the folder ocs_0-30cm_mean to acquire the tiles (caution: 4.3 GB to download; I used wget here and it took some time...), you seem to be able to load the data properly:

library(terra)
#> terra 1.6.49

# this works
rast("ocs_0-30cm_mean.vrt")
#> class       : SpatRaster 
#> dimensions  : 58034, 159246, 1  (nrow, ncol, nlyr)
#> resolution  : 250, 250  (x, y)
#> extent      : -19949750, 19861750, -6147500, 8361000  (xmin, xmax, ymin, ymax)
#> coord. ref. : Interrupted_Goode_Homolosine 
#> source      : ocs_0-30cm_mean.vrt 
#> name        : ocs_0-30cm_mean 
#> min value   :               6 
#> max value   :             208

# this also works now
rast("ocs_0-30cm_mean.vrt") |> plot()

As I said, absolutely without any experience with said file format, so please be my guest to rectify this.

7
  • Thanks @dimfalk. I get the same behaviour. My understanding is that the .vrt.ovr is an "overview" file that contains enough data to make high-level plots but not enough to access the data in its native resolution. Dec 20, 2022 at 15:43
  • According to the docs, vrt is a virtual GDAL dataset to be composed from other GDAL datasets. Opened up in notepad, you'll see a xml structure referencing to all the individual tiles like ./ocs_0-30cm_mean/tileSG-018-027/tileSG-018-027_2-4.tif. ISRIC FAQ also says that you need to download the whole ocs_0-30cm_mean folder for this to work since the .ovr.vrt files only exist to provide an overview.
    – dimfalk
    Dec 20, 2022 at 19:13
  • Maybe try gdalwarp instead of gdal_translate, c.f. this answer?
    – dimfalk
    Dec 20, 2022 at 19:42
  • 1
    Thanks for those suggestions. I was trying to follow the R tutorial for downloading the files linked on that same FAQ page. The gdalwarp command used in that answer did work for me, but only at a 0.25 degree resolution, whereas I'm trying to preserve the native resolution of the dataset (250 m, or ~7.5"). I'm now downloading an entire folder and I'll see if the either gdal_translate or gdalwarp work better with local files Dec 20, 2022 at 20:14
  • Happy to report that I was able to read the data correctly after downloading the entire file structure. Dec 21, 2022 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.