0

I created a workflow where I try to apply an algorithm over a selection of transects (line geometry) and images in a ImageCollection. I noticed that processing time goes up very quickly when the amount of transects is increasing and I'd like to explore the possibility to imrpove the script (and processing time). I started looking into the profiler and I am thinking where to start digging and how to approach this issue.

The workflow roughly follows this procedure:

  1. Create an imagecollection (SpectralUnmixing + Billateral filtering on images): for now 1 year of landsat observations (+/- 20 images)
  2. iterate over all transects (+/- 250 with a length of 30km): break up the transects in point, reduceRegion for each point and extract pixel values at transect.
  3. Build Arrays with for each pixel the extract values of Spectal unmixing output
  4. Do some array calculations such as first/2nd order derivatives for peaks & valleys on the transect
  5. return the required output.

https://code.earthengine.google.com/8edf8f719c2f8ae01b1d8629bcdf9dde

So I was expecting that most of the computation time would be in the reduceRegion (step2) and billateral filtering (step 1) and not so much in the Array computations (step 4). So when I look at the profiler I see that most computation time is indeed plumbing, loading assets and applying image algorithms to it (related to billateral filtering function I assume). Yet, most of the memory goes to the algorithm array group, which I assume are the steps that fall under number 4.

Does the peak memory below in the image indeed indicate that my array computations can be optimized or is it mainly the image computations that could be improved? And where in the alogrithms is there a possibility to improve computation time spent on overhead/plumbing?

The screen shots are from tests applied to 1 image and 1 transect.

peak memory

computationn time

1
  • Are you using .map() on the line segments? Overall it's difficult to give suggestion about performance improvement if we do not know anything specific about the code you are using
    – JonasV
    Nov 19, 2020 at 11:52

1 Answer 1

1

Does the peak memory below in the image indeed indicate that my array computations can be optimized or is it mainly the image computations that could be improved?

Algorithm Array doesn't mean array algorithms in general, but constructing arrays. So it might be large just because it's measuring the inputs to your arrays whereas the other array algorithms are operating on that existing array, not allocating large amounts of memory on their own.

Or does the plumbing time suggest that the overhead is just to large somehow?

Yes, “plumbing” is intended to measure overhead: work that isn't specifically the algorithms you asked for but is necessary to set them up and connect them.

However, keep in mind that computation steps can be reorganized such that the profiler won't necessarily attribute them correctly (the most notable case of this is computing image pixels from a chain of image algorithms), so it may be useful to not just assume the profiler is “blaming” the right thing, but actually try removing/replacing parts of your script and seeing how the numbers change.

Not the answer you're looking for? Browse other questions tagged or ask your own question.