Feeding gdalwarp with binary data instead of ascii, how to specify header information

Question

I have a working pipeline where I download a tar.gz file which contains multiple binary files in the german RADOLAN format (Translate PNG obtained through RADOLAN from DWD's projection to EPSG:3857).

I parse these binary files and save them as ASCII files. For each binary file I generate two ASCII files, one with the precipitation data to generate the colored clouds and one with the "valid" radar data, which I then use with an edge detect algorithm to generate the radar boundaries (grey line).

I generate the image via gdalwarp, because I need to reproject this RADOLAN projection into Web Mercator (EPSG:3857)

For this I use the following command:

gdalwarp
  -overwrite
  -s_srs "+proj=stere +lat_0=90.0 +lon_0=10.0 +lat_ts=60.0 +a=6370040 +b=6370040 +units=m"
  -t_srs "EPSG:3857"
  source-pixels.asc
  target-pixels.tif

gdalwarp
  -overwrite
  -s_srs "+proj=stere +lat_0=90.0 +lon_0=10.0 +lat_ts=60.0 +a=6370040 +b=6370040 +units=m"
  -t_srs "EPSG:3857"
  source-edges.asc
  target-edges.tif

I use that command twice for each binary file, since I have two ASCII files.

Finally I read those tif files wit PIL (Python Image Library) where I colorize the clouds, detect the edges, and merge the images.

My problem is with the ASCII files.

Writing these ASCII files is very slow. I use numpy's savetxt method to write the arrays into the file. This takes about 0.33 seconds per file, and I'm saving 48 ASCII files (from 24 bin files). I'm waiting 16 seconds just to get the files written.

In contrast, when I use numpy.save, each file takes a bit less than 0.03 seconds.

So I'm thinking about not using the ASCII files, but to feed binary files into gradwarp.

The issue is that these ASCII files contain additional information in a header.

This is the content of one of the edge files:

ncols         900
nrows         900
xllcorner     -523462
yllcorner     -4658645
cellsize      1000
NODATA_value  -1
668 668 668 668 668 668 668 668 668 668 668 668 668 668 ...

668 indicates that this pixel is out of range, that no radar was able to collect information about that position. Those pixels (positions) which do contain valid radar information, even if this means that nothing is there, are indicated by a 0.

The pixel data ASCII files have the same header, but the values vary according to the precipitation intensity.

The important thing here is the header information:

ncols         900
nrows         900
xllcorner     -523462
yllcorner     -4658645
cellsize      1000
NODATA_value  -1

These ASCII files apparently follow some gis-related specification which I don't know about. I found those values in a PDF file.

Now, if I want to feed binary data into gdalwarp, this header won't be in the file. How can I pass that header information to gdalwarp? Also, how do the pixels in the binary file need to be specified? How do I specify gdalwarp how it should interpret that binary data?

I'm a hobbyist at this, so don't expect any background knowledge from my part.

The most I came to think of was to create GeoTIFF files instead of ASCII files, as it apparently is possible to add some geoinformation to the TIF file, so that gdalwarp would be doing a TIF to TIF reprojection, which I guess sounds sensible.

How would I structure such a GeoTIFF file? Would that be what comes the closest to a binary format as I'm intending to use it? Are there any libraries for generating those files with NumPy? Or should I take another approach?

Answer 1

Assuming the bin/save files are a generic flat binary format, you don't have to do much. Just save the header to a separate *.hdr file, you may have to specify pixel type to match the binary data type.

See the GDAL EHdr format:

EHdr -- ESRI .hdr Labelled

GDAL supports reading and writing the ESRI .hdr labeling format, often referred to as ESRI BIL format. Eight, sixteen and thirty-two bit integer raster data types are supported as well as 32 bit floating point.

Other options are to write out a raw raster VRT (*.vrt) header or an ENVI .hdr header.