4

I'm using /vsicurl/ file handler to read Landsat 8 data out of S3. Following GDAL documentation "to minimize the total number of HTTP requests outside of the target GeoTIFF file" I set GDAL_DISABLE_READDIR_ON_OPEN=YES and CPL_VSIL_CURL_ALLOWED_EXTENSIONS=tif,ovr environment variables/configuration options to avoid any side-car files (such as .aux.xml, .aux, etc.) to be probed. But it seems that setting GDAL_DISABLE_READDIR_ON_OPEN=YES doesn't make any difference to gdal_translate. Why does GDAL request directory anyway > GET /L8/044/034/LC80440342015224LGN00/? Can I force it not to request the directory as it produces 404?

Here's the trace of gdal_translate I get setting config option CPL_CURL_VERBOSE=YES:

    $ time gdal_translate --config CPL_VSIL_CURL_ALLOWED_EXTENSIONS ".tif",".ovr" --config CPL_CURL_VERBOSE YES --config GDAL_DISABLE_READDIR_ON_OPEN TRUE \
--config GDAL_PAM_ENABLED NO -of GTiff -oo OVERVIEW_LEVEL=3 "/vsicurl/http://landsat-pds.s3.amazonaws.com/L8/044/034/LC80440342015224LGN00/LC80440342015224LGN00_B3.TIF" output.tif

    * Couldn't find host landsat-pds.s3.amazonaws.com in the .netrc file; using defaults
    * Hostname was NOT found in DNS cache
    *   Trying 54.231.176.186...
    * Connected to landsat-pds.s3.amazonaws.com (54.231.176.186) port 80 (#0)
    > HEAD /L8/044/034/LC80440342015224LGN00/LC80440342015224LGN00_B3.TIF HTTP/1.1
    Host: landsat-pds.s3.amazonaws.com
    Accept: */*

    < HTTP/1.1 200 OK
    < x-amz-id-2: 6jYJ3/tw1Woc2JSlRuFKvGxIu3s/ErZdVwYIbkXwelAZX+0SoOwU3gg9aiwoAiwv8KV1KE+LR4A=
    < x-amz-request-id: CE140F074802D82D
    < Date: Thu, 02 Feb 2017 10:59:15 GMT
    < Last-Modified: Thu, 13 Aug 2015 15:19:12 GMT
    < ETag: "ee74623d48978a4035b1a85e8040233e"
    < Accept-Ranges: bytes
    < Content-Type: image/tiff
    < Content-Length: 61547907
    * Server AmazonS3 is not blacklisted
    < Server: AmazonS3
    * Failed writing header
    * Closing connection 0
    * Couldn't find host landsat-pds.s3.amazonaws.com in the .netrc file; using defaults
    * Hostname was found in DNS cache
    *   Trying 54.231.176.186...
    * Connected to landsat-pds.s3.amazonaws.com (54.231.176.186) port 80 (#1)
    > GET /L8/044/034/LC80440342015224LGN00/LC80440342015224LGN00_B3.TIF HTTP/1.1
    Range: bytes=0-16383
    Host: landsat-pds.s3.amazonaws.com
    Accept: */*

    < HTTP/1.1 206 Partial Content
    < x-amz-id-2: 7bIrEUOfquUVUxRpAFsEXOTT2CEGUWm9g1tk9XzvgzEXOyxYliT4xSw5WlJsMXQMpWfAcfVHCck=
    < x-amz-request-id: BA2652D472196DEE
    < Date: Thu, 02 Feb 2017 10:59:15 GMT
    < Last-Modified: Thu, 13 Aug 2015 15:19:12 GMT
    < ETag: "ee74623d48978a4035b1a85e8040233e"
    < Accept-Ranges: bytes
    < Content-Range: bytes 0-16383/61547907
    < Content-Type: image/tiff
    < Content-Length: 16384
    * Server AmazonS3 is not blacklisted
    < Server: AmazonS3
    < 
    * Connection #1 to host landsat-pds.s3.amazonaws.com left intact
    * Couldn't find host landsat-pds.s3.amazonaws.com in the .netrc file; using defaults
    * Found bundle for host landsat-pds.s3.amazonaws.com: 0x1a4d6b0
    * Re-using existing connection! (#1) with host landsat-pds.s3.amazonaws.com
    * Connected to landsat-pds.s3.amazonaws.com (54.231.176.186) port 80 (#1)
    > HEAD /L8/044/034/LC80440342015224LGN00/LC80440342015224LGN00_B3.TIF.ovr HTTP/1.1
    Host: landsat-pds.s3.amazonaws.com
    Accept: */*

    < HTTP/1.1 200 OK
    < x-amz-id-2: W6kxsFbpflE2o0K+I+vDo9fWnC8qjJ8ruh7jqhgLVmPahf/KwE0QyWq3RAxXEgbMHVRfkO0cUV8=
    < x-amz-request-id: F39702D73422472D
    < Date: Thu, 02 Feb 2017 10:59:16 GMT
    < Last-Modified: Thu, 13 Aug 2015 15:19:06 GMT
    < ETag: "646dd2c687417965a8a98c3fac207d5c"
    < Accept-Ranges: bytes
    < Content-Type: application/octet-stream
    < Content-Length: 7911114
    * Server AmazonS3 is not blacklisted
    < Server: AmazonS3
    * Failed writing header
    * Closing connection 1
    * Couldn't find host landsat-pds.s3.amazonaws.com in the .netrc file; using defaults
    * Hostname was found in DNS cache
    *   Trying 54.231.176.186...
    * Connected to landsat-pds.s3.amazonaws.com (54.231.176.186) port 80 (#2)
    > GET /L8/044/034/LC80440342015224LGN00/LC80440342015224LGN00_B3.TIF.ovr HTTP/1.1
    Range: bytes=0-16383
    Host: landsat-pds.s3.amazonaws.com
    Accept: */*

    < HTTP/1.1 206 Partial Content
    < x-amz-id-2: fmVF0NQqgsoaG12WxNJXz6ldsi+/EmHZ89Ect8PQW0fg3tEg/0AiDjFJc7bEPEAEb0oMvlKhNtE=
    < x-amz-request-id: EF8E91510427F292
    < Date: Thu, 02 Feb 2017 10:59:16 GMT
    < Last-Modified: Thu, 13 Aug 2015 15:19:06 GMT
    < ETag: "646dd2c687417965a8a98c3fac207d5c"
    < Accept-Ranges: bytes
    < Content-Range: bytes 0-16383/7911114
    < Content-Type: application/octet-stream
    < Content-Length: 16384
    * Server AmazonS3 is not blacklisted
    < Server: AmazonS3
    < 
    * Connection #2 to host landsat-pds.s3.amazonaws.com left intact
    * Couldn't find host landsat-pds.s3.amazonaws.com in the .netrc file; using defaults
    * Found bundle for host landsat-pds.s3.amazonaws.com: 0x1a4f6c0
    * Re-using existing connection! (#2) with host landsat-pds.s3.amazonaws.com
    * Connected to landsat-pds.s3.amazonaws.com (54.231.176.186) port 80 (#2)
    > GET /L8/044/034/LC80440342015224LGN00/ HTTP/1.1
    Host: landsat-pds.s3.amazonaws.com
    Accept: */*

    < HTTP/1.1 404 Not Found
    < x-amz-request-id: 2036666677ADF9BC
    < x-amz-id-2: Qnht5YNBibSCFD8OjhMjqVH+7dyZ2hy1nll0fnngyzWPbSuYBcnbnrK1rYtTV2nOTgbxFZ7G4P8=
    < Content-Type: application/xml
    < Transfer-Encoding: chunked
    < Date: Thu, 02 Feb 2017 10:59:15 GMT
    * Server AmazonS3 is not blacklisted
    < Server: AmazonS3
    < 
    * Connection #2 to host landsat-pds.s3.amazonaws.com left intact
    Input file size is 95, 97
    0* Couldn't find host landsat-pds.s3.amazonaws.com in the .netrc file; using defaults
    * Found bundle for host landsat-pds.s3.amazonaws.com: 0x1a4f6c0
    * Re-using existing connection! (#2) with host landsat-pds.s3.amazonaws.com
    * Connected to landsat-pds.s3.amazonaws.com (54.231.176.186) port 80 (#2)
    > GET /L8/044/034/LC80440342015224LGN00/LC80440342015224LGN00_B3.TIF.ovr HTTP/1.1
    Range: bytes=7897088-7911113
    Host: landsat-pds.s3.amazonaws.com
    Accept: */*

    < HTTP/1.1 206 Partial Content
    < x-amz-id-2: n9CmjbkwetFZAf2UVVxw8qsIfqQPSYK6TJvGRCSAN8fRoG6SXE2d4nz3aA9yh6Bn6oPZIstEaP4=
    < x-amz-request-id: 98E79AD332C5C171
    < Date: Thu, 02 Feb 2017 10:59:17 GMT
    < Last-Modified: Thu, 13 Aug 2015 15:19:06 GMT
    < ETag: "646dd2c687417965a8a98c3fac207d5c"
    < Accept-Ranges: bytes
    < Content-Range: bytes 7897088-7911113/7911114
    < Content-Type: application/octet-stream
    < Content-Length: 14026
    * Server AmazonS3 is not blacklisted
    < Server: AmazonS3
    < 
    * Connection #2 to host landsat-pds.s3.amazonaws.com left intact
    ...10...20...30...40...50...60...70...80...90...100 - done.

    real    0m3.169s
    user    0m0.036s
    sys 0m0.206s

I'm using GDAL 2.1.0, released 2016/04/25

  • Works for me with GDAL 2.2-dev. Slight difference: my version is trying to find also file ...tif.ovr.ovr (overview for the overview file). – user30184 Feb 2 '17 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.