I have a working pipeline where I download a tar.gz file which contains multiple binary files in the german RADOLAN format (Translate PNG obtained through RADOLAN from DWD's projection to EPSG:3857).
I parse these binary files and save them as ASCII files. For each binary file I generate two ASCII files, one with the precipitation data to generate the colored clouds and one with the "valid" radar data, which I then use with an edge detect algorithm to generate the radar boundaries (grey line).
I generate the image via
gdalwarp, because I need to reproject this RADOLAN projection into Web Mercator (EPSG:3857)
For this I use the following command:
gdalwarp -overwrite -s_srs "+proj=stere +lat_0=90.0 +lon_0=10.0 +lat_ts=60.0 +a=6370040 +b=6370040 +units=m" -t_srs "EPSG:3857" source-pixels.asc target-pixels.tif gdalwarp -overwrite -s_srs "+proj=stere +lat_0=90.0 +lon_0=10.0 +lat_ts=60.0 +a=6370040 +b=6370040 +units=m" -t_srs "EPSG:3857" source-edges.asc target-edges.tif
I use that command twice for each binary file, since I have two ASCII files.
Finally I read those
tif files wit PIL (Python Image Library) where I colorize the clouds, detect the edges, and merge the images.
My problem is with the ASCII files.
Writing these ASCII files is very slow. I use
savetxt method to write the arrays into the file. This takes about 0.33 seconds per file, and I'm saving 48 ASCII files (from 24 bin files). I'm waiting 16 seconds just to get the files written.
In contrast, when I use
numpy.save, each file takes a bit less than 0.03 seconds.
So I'm thinking about not using the ASCII files, but to feed binary files into
The issue is that these ASCII files contain additional information in a header.
This is the content of one of the edge files:
ncols 900 nrows 900 xllcorner -523462 yllcorner -4658645 cellsize 1000 NODATA_value -1 668 668 668 668 668 668 668 668 668 668 668 668 668 668 ...
668 indicates that this pixel is out of range, that no radar was able to collect information about that position. Those pixels (positions) which do contain valid radar information, even if this means that nothing is there, are indicated by a 0.
The pixel data ASCII files have the same header, but the values vary according to the precipitation intensity.
The important thing here is the header information:
ncols 900 nrows 900 xllcorner -523462 yllcorner -4658645 cellsize 1000 NODATA_value -1
These ASCII files apparently follow some gis-related specification which I don't know about. I found those values in a PDF file.
Now, if I want to feed binary data into
gdalwarp, this header won't be in the file. How can I pass that header information to
gdalwarp? Also, how do the pixels in the binary file need to be specified? How do I specify
gdalwarp how it should interpret that binary data?
I'm a hobbyist at this, so don't expect any background knowledge from my part.
The most I came to think of was to create GeoTIFF files instead of ASCII files, as it apparently is possible to add some geoinformation to the TIF file, so that
gdalwarp would be doing a TIF to TIF reprojection, which I guess sounds sensible.
How would I structure such a GeoTIFF file? Would that be what comes the closest to a binary format as I'm intending to use it? Are there any libraries for generating those files with NumPy? Or should I take another approach?