How to avoid (occasional) 404 errors when trying to access Sentinel-2 images using pystac-client

Question

I use pystac-client to access some sentinel images. In Google Colab, I install pystac-client and rasterio by:

%%shell
pip install pystac-client
pip install rasterio

Import both modules:

from pystac_client import Client
import rasterio

Define a bounding box, for which I'd like to retrieve Sentinel-2 images:

bbox = \
{'latHigher': 57.4209358,
 'latLower': 57.4198741,
 'lonHigher': -5.7950406,
 'lonLower': -5.795841}

Based on the README.md on GitHub, I do:

catalog = Client.open("https://earth-search.aws.element84.com/v0")
mysearch = catalog.search(collections=['sentinel-s2-l2a-cogs'],
                          bbox=[bbox['lonLower'],bbox['latLower'],bbox['lonHigher'],bbox['latHigher']],
                          datetime='2020-05-01/2020-08-30', 
                          max_items=20)   
print(f"{mysearch.matched()} items found")
resdict = mysearch.get_all_items_as_dict()

96 items found

I get many URLs pointing to Sentinel tiffs, I can read most of them, for example:

src = rasterio.open(resdict['features'][12]['assets']['B08']['href'])

But for some, ie:

src = rasterio.open(resdict['features'][13]['assets']['B08']['href'])

I get an error:

---------------------------------------------------------------------------
CPLE_HttpResponseError                    Traceback (most recent call last)
rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()

rasterio/_shim.pyx in rasterio._shim.open_dataset()

rasterio/_err.pyx in rasterio._err.exc_wrap_pointer()

CPLE_HttpResponseError: HTTP response code: 404

During handling of the above exception, another exception occurred:

RasterioIOError                           Traceback (most recent call last)
<ipython-input-6-dcc11b541f68> in <module>()
----> 1 src = rasterio.open(resdict['features'][13]['assets']['B08']['href'])

1 frames
/usr/local/lib/python3.7/dist-packages/rasterio/env.py in wrapper(*args, **kwds)
    435 
    436         with env_ctor(session=session):
--> 437             return f(*args, **kwds)
    438 
    439     return wrapper

/usr/local/lib/python3.7/dist-packages/rasterio/__init__.py in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    218         # None.
    219         if mode == 'r':
--> 220             s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
    221         elif mode == "r+":
    222             s = get_writer_for_path(path, driver=driver)(

rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()

RasterioIOError: HTTP response code: 404

How could I change my catalog.search query to avoid this error?

Answer 1

I'm guessing the problem is that the catalog is returning images that no longer exist, or that don't have the same bands (B08 looks like) that you're querying for. Could also be some networking issues. Since it's just an occasional error (and I'm guessing you're more irritated at the errors in long running code) I'd just handle the exception.

One way to handle this would be to just stick it in a try/except structure. I've added two empty lists in the front to be populated as the loop runs in order to show success/failure links:

working_links = []
not_working_links = []
for x in my_list:
   try:
      src = rasterio.open(resdict['features'][x]['assets']['B08']['href'])
      working_links.append(x)
   except CPLE_HttpResponseError as error_404:
      print(error_404)
      not_working_links.append(x)
      <do something to handle the 404 error. Probably just log it and move on>

I'd run the above and then eyeball why the query is failing for certain images. If the reason is that certain images don't have the band you're querying for then you could alter the query to only fetch images that meet your tighter requirement. Take a look at the filter doc on their github page on how to do a filter on the json parameters. And take a look at the stac link to see what parameters are available. You might want to stick that into jsonlint.com to be more human readable.