OSError: Failed to list error in Open Data Cube

Question

I am trying to index sentinel-l2a-cogs following the guide here. The product definition is added to the database.

(cubeenv) λ datacube product list
s2_l2a  Sentinel-2a and Sentinel-2b imagery, processed to Level 2A (Surface Reflectance) and converted to Cloud Optimized GeoTIFFs

but I get the error on running s3-to-dc --stac --no-sign-request s3://deafrica-sentinel-2/sentinel-s2-l2a-cogs/37/M/CS/2017/10/**/S2A_37MCS_20171016_0_L2A.json s2_l2a

Traceback (most recent call last):
  File "D:\anaconda\envs\cubeenv\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\anaconda\envs\cubeenv\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\anaconda\envs\cubeenv\Scripts\s3-to-dc.exe\__main__.py", line 7, in <module>
  File "D:\anaconda\envs\cubeenv\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "D:\anaconda\envs\cubeenv\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "D:\anaconda\envs\cubeenv\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\anaconda\envs\cubeenv\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "D:\anaconda\envs\cubeenv\lib\site-packages\odc\apps\dc_tools\s3_to_dc.py", line 129, in cli
    added, failed = dump_to_odc(
  File "D:\anaconda\envs\cubeenv\lib\site-packages\odc\apps\dc_tools\s3_to_dc.py", line 56, in dump_to_odc
    for uri, metadata in uris_docs:
  File "D:\anaconda\envs\cubeenv\lib\site-packages\odc\apps\dc_tools\_docs.py", line 89, in parse_doc_stream
    for uri, doc in doc_stream:
  File "D:\anaconda\envs\cubeenv\lib\site-packages\odc\apps\dc_tools\s3_to_dc.py", line 31, in stream_docs
    for document in documents:
  File "D:\anaconda\envs\cubeenv\lib\site-packages\odc\aio\__init__.py", line 494, in __call__
    for rr, ee in future_results(
  File "D:\anaconda\envs\cubeenv\lib\site-packages\odc\ppt\__init__.py", line 85, in future_results
    have_more = fill(it, need_n, active)
  File "D:\anaconda\envs\cubeenv\lib\site-packages\odc\ppt\__init__.py", line 70, in fill
    x = next(src)
  File "D:\anaconda\envs\cubeenv\lib\site-packages\odc\aio\__init__.py", line 486, in generate_requests
    for url in urls:
  File "D:\anaconda\envs\cubeenv\lib\site-packages\odc\apps\dc_tools\s3_to_dc.py", line 25, in stream_urls
    for url in urls:
  File "D:\anaconda\envs\cubeenv\lib\site-packages\odc\aio\__init__.py", line 433, in find
    raise IOError(f"Failed to list: {url}")
OSError: Failed to list: s3://deafrica-sentinel-2/sentinel-s2-l2a-cogs/37/M/CS/2017/10/

On running datacube system check I get the following output.

Version:       1.8.6
Config files:  C:\Users\91911\.datacube_integration.conf
Host:          localhost:5432
Database:      opendatacube
User:          postgres
Environment:   None
Index Driver:  default

Valid connection:       YES

I am running in cubeenv conda environment and I am able to connect to the database as well. Additionally my DATACUBE_CONFIG_PATH is pointing to datacube_integration.conf. What is causing the problem and how do I resolve the issue? I was able to downloaded the STAC document manually using the following command

aws s3 cp --no-sign-request --region af-south-1 s3://deafrica-sentinel-2/sentinel-s2-l2a-cogs/37/M/CS/2017/10/S2A_37MCS_20171003_0_L2A/S2A_37MCS_20171003_0_L2A.json ./

Answer 1

I think there was a minor issue with the docs, which I have fixed.

Try indexing from the Element 84-managed bucket in us-west-2, like this:

s3-to-dc --stac --no-sign-request \
"s3://sentinel-cogs/sentinel-s2-l2a-cogs/37/M/CS/**/*S2A_37MCS_20171016_0_L2A*.json" s2_l2a