I am trying to step by step save a lot of data in a single GeoTiff file but rasterio seems to fail right away when I try to open the file I want to write my data to due to my available RAM not being large enough to store the image all at once.
I am encountering this error:
rasterio._err.CPLE_OutOfMemoryError: /io/gdal-3.6.4/frmts/mem/memdataset.cpp, 1289: cannot allocate 1x40000000000 bytes
From this Code:
import numpy as np
import rasterio
from affine import Affine
from rasterio.profiles import Profile
target_transform = Affine(0.4999989733319868, 0.0, 371169.31953843555,
0.0, -0.49999725418315205, 6707833.49293414) # some transform
target_shape = (100000, 100000) # some large shape
target_profile = Profile(data={
'driver': "COG",
'count': '1',
'nodata': np.NaN,
'transform': target_transform,
'crs': "EPSG:2154",
'height': target_shape[0],
'width': target_shape[1],
'dtype': 'float32'
})
with rasterio.open("test.tif", 'w', **target_profile) as dst: # This is where i get my error.
for ij, window in dst.block_windows(1):
data = np.ones((1, window.height, window.width))
dst.write(data, window=window)
I would like to be able to write to the file step by step without the need to have enough RAM to have it in memory all at once. I had hoped just windowed writing would fix this issue but rasterio seems to complain when I try to open a file that is potentially too large.
Are there possibly some parameters to the rasterio.open function that would solve this problem?