Asking for the precise detail of gdal_translate's output .xyz multiband file format (when converting from GRIB)

Question

gdal_translate is able to convert a multidimensional GRIB data file (Copernicus ERA5 reanalysis) with the following command-line command (gdal_translate.exe from 11/2016):

gdal_translate source.grib target1.xyz

gdal_translate will in this case create two files:

1. target1.xyz.aux.xml
2. target1.xyz

target1.xyz.aux.xml is rather clear, it contains readable data like this:

<PAMDataset>
  <PAMRasterBand band="1">
    <Metadata>
      <MDI key="GRIB_COMMENT">10 metre u wind component [m/s]</MDI>
      <MDI key="GRIB_ELEMENT">10U</MDI>
      <MDI key="GRIB_FORECAST_SECONDS">0 sec</MDI>
      <MDI key="GRIB_REF_TIME">310262400 sec UTC</MDI>
      <MDI key="GRIB_SHORT_NAME">0-SFC</MDI>
      <MDI key="GRIB_UNIT">[m/s]</MDI>
      <MDI key="GRIB_VALID_TIME">310262400 sec UTC</MDI>
    </Metadata>
  </PAMRasterBand>
  <PAMRasterBand band="2">
    etc.

target1.xyz is a mystery for me. It's definitely not readable ASCII, but binary. It's not possible to paste the first 100 bytes into this SE edit window. Reading from file start as 64-bit doubles produces crazy values (huge exponents), so it definitely has a structure and most likely a header.

No solution:

Once I have the xml file, I can parse it's headers and get the corresponding band numbers. Then I can call gdal_translate again, this time with a band number (note "target2" file name). Example:

gdal_translate source.grib target2.xyz -b 12345

In this case, I do get readable ASCII, example:

22    60     4.953857421875
22.25 60     5.292724609375
22.5  60     5.439208984375
22.75 60     5.296630859375
23    60     5.023193359375
23.25 60     4.396240234375
23.5  60     3.891357421875
23.75 60     4.203857421875
24    60     4.513427734375
24.25 60     4.732177734375
24.5  60     4.951904296875
etc.

However, each conversion takes one second and there can well be 30000 bands to extract => 8 hours! I would very much like to read the target1.xyz file using own code, but it seems impossible to find any information about it's format.

Tried in the first conversion ("target1", with no band argument):

• Using other target1 file name extension, in order to get another output format - gdal_translate ignores.
• Specifying the output format with the -of [format] parameter, gdal_translate ignores.
• Reanaming target1.xyz => target1.tiff and loading into a program with capabilities - no go (says invalid file format). Seemed like a waste of time to try more renaming + loading, blind shooting.

Tried in the second conversion ("target2", with band argument):

• Specifying multiple bands simultaneously (as the specification says it is possible, see https://gdal.org/programs/gdal_translate.html) - gdal_translate ignores all but the first band (this is also verified elsewhere in SE); in addition the DOS command line has a length limit. (I could have imagined that gdal_translate had created multiple output file name variations in this case). Example:

  gdal_translate source.grib target2.xyz -b 12345 12346 12347 etc.   
  

Question: What is the file format and structure of the target1.xyz file that

gdal_translate source.grib target1.xyz

creates?

Answer 1

If your GDAL is from 2016 it's time to update.

Your output file format is a GeoTIFF.

The default output format (in GDAL < 2.3) is a GeoTIFF. With your GDAL, if you want an XYZ ascii file, you need to tell GDAL that by specifying the format with the -of flag.

gdal_translate -of XYZ source.grib target1.xyz

Starting with GDAL 2.3 released in 2018, if not specified, the format is guessed from the file extension.

Note that while gdal_translate can output multiple bands in general, it can only output multiple bands to formats that support this. However, the XYZ format only supports a single band. So if you try to output multiple bands (with a recent GDAL) you will get the message:

Warning 6: XYZ driver only uses the first band of the dataset.

You can of course specify a different band using a -b flag, but you still can only output a single band at a time to the XYZ format

gdal_translate -b 1 -of XYZ source.grib target1.xyz
gdal_translate -b 2 -of XYZ source.grib target2.xyz
gdal_translate -b 3 -of XYZ source.grib target3.xyz
gdal_translate -b 4 -of XYZ source.grib target4.xyz
etc...
gdal_translate -b 999 -of XYZ source.grib target999.xyz