Download USGS datasets on either Australia or Africa sandbox

Question

Is it possible to use my own AWS credentials to download a landsat8 dataset from USGS on either of the Sandbox (Africa/Australia)? I tried the following which didn't work:

import boto3
from matplotlib.pyplot import imshow
import rasterio as rio
from rasterio.session import AWSSession

!aws configure set aws_access_key_id {secret}
!aws configure set aws_secret_access_key {secret}
!aws configure set default.region ap-southeast-2

aws_session = AWSSession(boto3.Session(), requester_pays=True)

with rio.Env(aws_session):
    with rio.open('s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_SR_B2.TIF') as src:
        profile = src.profile
        arr = src.read(1)

imshow(arr)

I am getting the following error:

---------------------------------------------------------------------------
CPLE_OpenFailedError                      Traceback (most recent call last)
rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()

rasterio/_shim.pyx in rasterio._shim.open_dataset()

rasterio/_err.pyx in rasterio._err.exc_wrap_pointer()

CPLE_OpenFailedError: '/vsis3/usgs-landsat/collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_SR_B2.TIF' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

RasterioIOError                           Traceback (most recent call last)
/tmp/ipykernel_299/797907408.py in <module>
      1 with rio.Env(aws_session):
----> 2     with rio.open(cog) as src:
      3         profile = src.profile
      4         arr = src.read(1)
      5 

/env/lib/python3.8/site-packages/rasterio/env.py in wrapper(*args, **kwds)
    435 
    436         with env_ctor(session=session):
--> 437             return f(*args, **kwds)
    438 
    439     return wrapper

/env/lib/python3.8/site-packages/rasterio/__init__.py in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    218         # None.
    219         if mode == 'r':
--> 220             s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
    221         elif mode == "r+":
    222             s = get_writer_for_path(path, driver=driver)(

rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()

RasterioIOError: '/vsis3/usgs-landsat/collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_SR_B2.TIF' not recognized as a supported file format.

Answer 1

As suggested by Alex Leith:

Yes, but you need to unset the other variables that tell GDAL to do unsigned requests, I think.

I think this might be the environment variable: AWS_REQUEST_PAYER='requester'

I have unset AWS_NO_SIGN_REQUEST, updated the "AWS_END_POINT" to "s3.amazonaws.com" and added AWS_REQUEST_PAYER='requester' to make it work.