gdalinfo authenticate for remote file

Question

I am trying to access the metadata for the following COG (Cloud Optimized GeoTIFF) using

gdalinfo /vsicurl/https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/082/070/LC09_L2SR_082070_20220427_20220429_02_T1/LC09_L2SR_082070_20220427_20220429_02_T1_SR_B3.TIF

But this resulted in error

ERROR 4: `/vsicurl/https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/082/070/LC09_L2SR_082070_20220427_20220429_02_T1/LC09_L2SR_082070_20220427_20220429_02_T1_SR_B3.TIF' not recognized as a supported file format.
gdalinfo failed - unable to open '/vsicurl/https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/082/070/LC09_L2SR_082070_20220427_20220429_02_T1/LC09_L2SR_082070_20220427_20220429_02_T1_SR_B3.TIF'.

I tried importing in QGIS (with credentials) as well but it resulted in same error.

I am able to download the file when I paste the url in browser (it asks for credentials using server ers.cr.usgs.gov and on successful login, it downloads the file)

The url is taken from stacindex.org

How to pass in the credentials so GDAL can authenticate?

Update: I tried passing config options as mentioned in mailing list here but it still results in the same error

 gdalinfo  /vsicurl/https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/082/070/LC09_L2SR_082070_20220427_20220429_02_T1/LC09_L2SR_082070_20220427_20220429_02_T1_SR_B3.TIF --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config CPL_CURL_VERBOSE ON --config CPL_VSIL_CURL_USE_HEAD FALSE --config GDAL_HTTP_COOKIEFILE /tmp/cookies.txt --config GDAL_HTTP_COOKIEJAR /tmp/cookies.txt --config CPL_VSIL_CURL_ALLOWED_EXTENSIONS TIF

GDAL 3.3.3

Answer 1

For those who stumble upon this query, here is the workaround.

GDAL throws this error because the response from the URL is not a TIFF but an html document asking the user to login. We need to send a POST request to login first and subsequently do a GET request to get the actual TIFF. I don't know how you'd do that in shell script but here is the python version.

import requests
from bs4 import BeautifulSoup
url = 'https://ers.cr.usgs.gov/login' # this is end point for login
with requests.Session() as s:
    
  r = s.get(url)
  soup = BeautifulSoup(r.content, 'html5lib')
  sval = soup.find('input', attrs={'name':'csrf'})['value']
  
  data = {"username": "MYUSERNAME", # change your username and password
      "password": "MYPASSWORD",
      "csrf": sval}

  bf= s.post(url, data = data)
  print(bf.status_code)
  respb = s.get('https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/089/089/LC09_L2SP_089089_20220428_20220430_02_T1/LC09_L2SP_089089_20220428_20220430_02_T1_SR_B1.TIF',
              allow_redirects=True,
              headers = {'content-type': 'image/tiff'})
  with open('output.tif', 'wb') as src:
    src.write(respb.content)