1

I'm trying to access data from NASA's Earthdata S3 buckets, but I get a "<filename> does not exist in the file system, and is not recognized as a supported dataset name." error after waiting a long time (± 50 minutes, the process is downloading some data the whole time) doing the following:

from osgeo import gdal

gdal_config_options = {
    "AWS_ACCESS_KEY_ID": creds["accessKeyId"],
    "AWS_SESSION_TOKEN":  creds["sessionToken"],
    "AWS_SECRET_ACCESS_KEY": creds["secretAccessKey"],
}

url = "/vsis3/prod-lads/VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc"
    
for k, v in gdal_config_options.items():
    gdal.SetConfigOption(k, v)

out = gdal.Info(url)

The creds variable is a dictionary with temporary credential information that I get from here, you need a free account to get them.

When I introduce an error in one of the keys/tokens (e.g. "AWS_ACCESS_KEY_ID": creds["accessKeyId"] + "x", I do get a message immediately saying my credentials are unknown. So I do think they are being ingested correctly by GDAL.

I also managed to download the entire file using boto3, by doing the following:

import boto3

client = boto3.client(
    's3',
    aws_access_key_id=creds["accessKeyId"],
    aws_secret_access_key=creds["secretAccessKey"],
    aws_session_token=creds["sessionToken"]
)

client.download_file('prod-lads', 'VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc', 'test.nc')

Any ideas what I'm doing wrong or how to make this work? In the end I'm interested in accessing the files metadata without downloading the entire file.

2
  • 1
    Could you give an example of what the s3_url looks like? It should look similar to this /vsis3/eodata/Sentinel-2/.../file_name.extention (if using AWS_VIRTUAL_HOSTING=FALSE) where `/vsis3/ represents the endpoint, eodata is the bucket and the rest is the item.
    – PyMapr
    Commented Jul 20, 2023 at 6:26
  • Hi, @PyMapr! The URL I pass to gdal.Info is for example /vsis3/prod-lads/VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc, so the bucket-name is prod-lads. I just tried setting AWS_VIRTUAL_HOSTING=FALSE, but that doens't seem to help. Commented Jul 20, 2023 at 7:00

1 Answer 1

2

I got an answer through the gdal-dev mailing list from @Even.

It turns out that when I called gdal.Info, GDAL was starting to establish a list of all the files in the directory of the file passed. There are quite a lot of files there, so this slows down the process significantly.

Setting GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR in addition to the other configuration options prevents this and gdal.info returns the metadata in a couple of seconds.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.