2

So I have a 20 GB in size Raster Image in .img format. Though I will be examining the code, to find out why, I am AMAZED at how quickly QGIS renders the entire file in a scaled format, and prior to writing code I was wondering if anyone knows what other format that offers actual compression (this .img file I am using has an .ige component the docs say immediately turns off compression) but also has that pick and choose functionality I believe QGIS is using.

I believe based on what I'm seeing that the file is going to be heavily indexed inside, which allows the file pointer to skip around ALOT, whereas a PNG file is likely to be compressed row by row and this would probably require a person to decompress and load pretty much all the data to then scale the image and render it.

I don't want to give up the performance I am seeing in QGIS, but I'm uncertain how to proceed.

Does anyone know a better more compressed format that offers the same kind of fast scaling functionality?

Edit https://pastebin.com/bvbEcVtd

Here are the results of my tests on the Tree Canopy raster. This actually does create some precedent for finishing my next test, the access test. Assuming the good good people who are slowly ending our world don't cut time to finish the smallest thing short.

The somewhat scary thing is the difference between memory copy from GDAL in squares of 512 (which is native in this raster) to a Bitmap vs the time it takes to compress and save the PNG.

I will add access speed statistics as well.

Edit: So any evidence based answers that compliment the above experimentation I listed ? I didn't get around to the access tests yet, too busy being bugged by the repeat end of the world.

Improve this question
8
  • 3
    The docs for the GDAL PNG driver seem to support your theory in the 2nd paragraph. But if you're just looking for fast rendering then just having overviews/pyramids is likely to be one of the larger factors unless you're zoomed all the way in. Internally tiled TIFFs with either internal or external overviews are common and perform well, and both the base raster and overviews support a few different kinds of compression
    – mikewatt
    Commented Apr 8, 2021 at 18:29
  • i was thinking about dpi vs max screen resolution if thats what you mean by pyramids,.. got i'm sick of solving the same problems ! anyway, i always thought tiffs compressed poorly... but i'll look into it.
    – John Sohn
    Commented Apr 8, 2021 at 19:45
  • 2
    Pyramids (or overviews) are a set of increasingly downsampled images that come along with the base image. When rendering in QGIS or whatever other GIS software, the program will be able to select the resolution that most closely matches the screen resolution instead of resampling the entire 20GB raster every time it needs to redraw. You can create them with gdal.org/programs/gdaladdo.html
    – mikewatt
    Commented Apr 8, 2021 at 19:51
  • ERDAS IMG is not compressed, a read begin offset can be calculated with ( row * columns + column ) * bytes per pixel - no need for indexing; the IMG format does support run length encoding, a compression type I've not seen practically used since the early 2000's. However a GeoTIFF can perform just as well or better with LZW or deflate compression if it's tiled: rather than reading and decompressing an entire row only the blocks (default 128x128 pixels) that are in the read area are decompressed - this makes a real performance difference in 2TB+ uncompressed size GeoTIFF (~600GB compressed).
    – Michael Stimson
    Commented Apr 8, 2021 at 23:26
  • 2
    There is a common belief that TIFFs are uncompressed and huge. But TIFF is mainly an envelope around image data that can be compressed. See en.wikipedia.org/wiki/TIFF#TIFF_Compression_Tag.
    – user30184
    Commented Apr 9, 2021 at 7:15

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.