I have been using the Python GDAL API to read tif raster files as NumPy arrays. Previously, I simply read the raster into an array directly with GDAL:
ds = gdal.Open('example.tif') fullarray = np.array(ds.ReadAsArray())
However, with larger rasters, I receive a MemoryError. As a workaround, I have been looping over the raster and reading windows into a NumPy array:
#arbitrarily choosing rows, cols value here rows, cols = 20000, 20000 arr = np.zeros((4, rows, cols)) band = ds.GetRasterBand(1) xsize, ysize = band.XSize, band.YSize x_edge, y_edge = int(xsize - cols + 1), int(ysize - rows + 1) x_extra, y_extra = int(x_edge%cols), int(y_edge%rows) for i in tqdm(range(0, x_edge, cols)): for j in range(0, y_edge, rows): #read dataset into pre-allocated array ds.ReadAsArray(i, j, cols, rows, arr)
I experienced some significant speed-ups (at least 2x faster) after using a pre-allocated array. However, this is still quite a bottleneck in my code. Is there a better way to do this?