Error handling on missing dataset file

Question

I'm working with the sentinel-2-l2a-cogs dataset hosted on AWS, trying to index some areas taking advantage of the STAC API provided by Element84.

Unfortunately, I ran across the item S2B_36LUM_20180701_0_L2A that comes with broken URLs (404) to the related COG files.

Having indexed this item into ODC, when I do a dc.load() on that area and time frame I get an error:

Error opening source dataset: \
https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/L/UM/2018/7/S2B_36LUM_20180701_0_L2A/SCL.tif

CPLE_OpenFailedError: \
'/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/L/UM/2018/7/S2B_36LUM_20180701_0_L2A/SCL.tif' \
does not exist in the file system, and is not recognized as a supported dataset name.

Is there something I can do to let datacube just ignore that dataset and keep loading whatever other dataset is available?

Answer 1

The way to handle this is to add the argument skip_broken_datasets=True to your dc.load().

As it says, this will ignore the broken datasets.

The next thing to do is to report the broken datasets to Element 84 in their cirrus repository.