I created a workflow where I try to apply an algorithm over a selection of transects (line geometry) and images in a ImageCollection. I noticed that processing time goes up very quickly when the amount of transects is increasing and I'd like to explore the possibility to imrpove the script (and processing time). I started looking into the profiler and I am thinking where to start digging and how to approach this issue.
The workflow roughly follows this procedure:
- Create an imagecollection (SpectralUnmixing + Billateral filtering on images): for now 1 year of landsat observations (+/- 20 images)
- iterate over all transects (+/- 250 with a length of 30km): break up the transects in point, reduceRegion for each point and extract pixel values at transect.
- Build Arrays with for each pixel the extract values of Spectal unmixing output
- Do some array calculations such as first/2nd order derivatives for peaks & valleys on the transect
- return the required output.
So I was expecting that most of the computation time would be in the reduceRegion (step2) and billateral filtering (step 1) and not so much in the Array computations (step 4). So when I look at the profiler I see that most computation time is indeed plumbing, loading assets and applying image algorithms to it (related to billateral filtering function I assume). Yet, most of the memory goes to the algorithm array group, which I assume are the steps that fall under number 4.
Does the peak memory below in the image indeed indicate that my array computations can be optimized or is it mainly the image computations that could be improved? And where in the alogrithms is there a possibility to improve computation time spent on overhead/plumbing?
The screen shots are from tests applied to 1 image and 1 transect.