2

I am processing large MODIS time series (12 years) dataset with an array dimension of 4800 x 4800 with a data type of Int16-Sixteen bit signed integer. I want to build a 3 dimensional array (row,col,time(10 years)) using np.dstack. This is because I need to access the elements of each index in the 3D array and perform interpolation to those values flag as clouds. I know I cannot build my 3D array with the a dimension of 4800 x 4800 x 644 array my computer simply crashes after processing 10 rasters/array, hence I plan to process it by chunks/blocks. How to set the optimal block size (1200 X 1200 x 644 or 2400 x 2400 x 644) so I can process it faster without my computer crashing? I am running a 32-bit python with 12GB ram and 64-bit operating system.

  • Please edit your question and provide details of the amount of RAM you have and whether you are running 32 or 64 bit python. – user2856 Jan 29 '16 at 3:23
0

The "optimal" size should be determined by experimenting on your machine, as different machines have different computing power. I've encountered a similar situation before. I recommend you make your chunks even smaller (eg 400*400*644). If it works, then you can consider increasing the size in the subsequent tests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.