Read and then write rasterio GeoTIFF file without loading all data into memory

Question

I work with large drone image raster files, on the order of 40,000 x 40,000 and above. I have a large uncompressed GeoTIFF file and I want to use rasterio to rewrite the file in compressed format. I can do this by loading all data into memory, but is there a way to execute this writing without loading everything to memory?

For my original file I can open it with:

 dat = rasterio.open("grid_001.tif")

Then to rewrite the file with compression I tried:

profile = dat.profile.copy()
profile.update(
        compress='lzw')

with rasterio.open("grid_001_compressed.tif", 'w', **profile) as dst:
        dst.write(dat)

This will give me an error:

 ValueError: Source shape (44134, 44162) is inconsistent with given indexes 3

This is an expected error, because when I open the dataset it creates an iterator or lazy object without actually accessing the data. Now, if I did a command like:

 dat = dat.read()

This will load all of the data from the file into memory, and I can an array of dimensions [3, 44134, 44162] that I can write. BUT, this takes a lot of memory to implement.

Hence, is there a way to perform the same operation, but without loading everything into memory? I am not sure if windowed reads would help in this case or anything.

Answer 1

You can use rasterio's window or block reading & writing

dat = rasterio.open("grid_001.tif")
profile = dat.profile.copy()
profile.update(compress='lzw')

with rasterio.open("grid_001_compressed.tif", 'w', **profile) as dst:
    for ji, window in dat.block_windows(1)
        dst.write(dat.read(window=window), window=window)