I'm running an ODC instance indexing and loading Sentinel-2 L2A COGs from the Element84 bucket on AWS.
I use a Dask.distributed cluster deployed in "fixed" mode, for the definition of "fixed" provided here.
Data access is very very slow due to the bucket being in the
us-west-2 region and my infrastructure being on a private cloud in Europe.
This wouldn't be much of a problem, except that on some random cases computations hang on some
dc_load_Bxx Dask task, where
Bxx is one of the Sentinel-2 L2A bands (it is not always the same one).
So basically, the cluster gets "idle", while only one task is in execution, but indefinitely.
Re-running the computation makes it successful, which indicates a temporary problem probably related to data access, but I'm not sure, as I cannot see any error or warning in the Dask scheduler/workers logs.
Any thoughts about things that might go wrong in my current setting, or any suggestion where I can start investigating?