I am processing large MODIS time series (12 years) dataset with an array dimension of 4800 x 4800 with a data type of Int16-Sixteen bit signed integer. I want to build a 3 dimensional array (row,col,time(10 years)) using np.dstack. This is because I need to access the elements of each index in the 3D array and perform interpolation to those values flag as clouds. I know I cannot build my 3D array with the a dimension of 4800 x 4800 x 644 array my computer simply crashes after processing 10 rasters/array, hence I plan to process it by chunks/blocks. How to set the optimal block size (1200 X 1200 x 644 or 2400 x 2400 x 644) so I can process it faster without my computer crashing? I am running a 32-bit python with 12GB ram and 64-bit operating system.
The "optimal" size should be determined by experimenting on your machine, as different machines have different computing power. I've encountered a similar situation before. I recommend you make your chunks even smaller (eg 400*400*644). If it works, then you can consider increasing the size in the subsequent tests.