1

I am trying to step by step save a lot of data in a single GeoTiff file but rasterio seems to fail right away when I try to open the file I want to write my data to due to my available RAM not being large enough to store the image all at once.

I am encountering this error:

rasterio._err.CPLE_OutOfMemoryError: /io/gdal-3.6.4/frmts/mem/memdataset.cpp, 1289: cannot allocate 1x40000000000 bytes

From this Code:

import numpy as np
import rasterio
from affine import Affine
from rasterio.profiles import Profile

target_transform = Affine(0.4999989733319868, 0.0, 371169.31953843555,
                          0.0, -0.49999725418315205, 6707833.49293414) # some transform
target_shape = (100000, 100000) # some large shape
target_profile = Profile(data={
        'driver': "COG",
        'count': '1',
        'nodata': np.NaN,
        'transform': target_transform,
        'crs': "EPSG:2154",
        'height': target_shape[0],
        'width': target_shape[1],
        'dtype': 'float32'
    })

with rasterio.open("test.tif", 'w', **target_profile) as dst: # This is where i get my error. 
    for ij, window in dst.block_windows(1):
        data = np.ones((1, window.height, window.width))
        dst.write(data, window=window)

I would like to be able to write to the file step by step without the need to have enough RAM to have it in memory all at once. I had hoped just windowed writing would fix this issue but rasterio seems to complain when I try to open a file that is potentially too large.

Are there possibly some parameters to the rasterio.open function that would solve this problem?

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.