I'm trying to extract a small part of a largish COG GeoTIFF using gdal_translate. The source file is stored on S3 and I have a presigned URL to it, and use /vsicurl/ to access it. For whatever reason, GDAL does not use range requests to extract the part, but instead just downloads the complete file.
If I switch to using /vsis3/ to perform the same operation on the same file, it works as expected and GDAL makes a number of range requests only extracting the required data.
Example using /vsicurl/ with GDAL 3.6.0 (potentially sensitive info obfuscated):
gdal_translate -projwin 173.10156468104645 -41.33975369049866 173.1016761949935 -41.339789888978814 -outsize 3155 1024 -CO COMPRESS=DEFLATE --config CPL_CURL_VERBOSE YES /vsicurl/https://****.s3.eu-west-1.amazonaws.com/uploads/*****/results/ortho.tiff\?X-Amz-Algorithm=AWS4-HMAC-SHA256\&X-Amz-Credential=****\&X-Amz-Date=20240212T083110Z\&X-Amz-Expires=900\&X-Amz-SignedHeaders=host\&X-Amz-Signature=***** extract.tiff
* Couldn't find host ****.s3.eu-west-1.amazonaws.com in the (nil) file; using defaults
* Trying 52.218.89.72:443...
* Connected to ****.s3.eu-west-1.amazonaws.com (52.218.89.72) port 443 (#0)
* ALPN: offers h2,http/1.1
* CAfile: /home/per/miniconda3/ssl/cacert.pem
* CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted http/1.1
* Server certificate:
* subject: CN=*.s3-eu-west-1.amazonaws.com
* start date: Jan 31 00:00:00 2024 GMT
* expire date: Jan 15 23:59:59 2025 GMT
* subjectAltName: host "****.s3.eu-west-1.amazonaws.com" matched cert's "*.s3.eu-west-1.amazonaws.com"
* issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
* SSL certificate verify ok.
* using HTTP/1.1
> GET /uploads/****/results/ortho.tiff?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=****%2F20240212%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20240212T083110Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=**** HTTP/1.1
Host: ****.s3.eu-west-1.amazonaws.com
Accept: */*
< HTTP/1.1 200 OK
< x-amz-id-2: IPy6DiOO9lJL9a/MYTfbGyZGpBARXitggXT8Fq8dZhJ5QmGJdLqwcDFRySdvm2O9ASl7k6w6CHo=
< x-amz-request-id: JHMY8H735AS7ZEMB
< Date: Mon, 12 Feb 2024 08:33:06 GMT
< Last-Modified: Thu, 25 Jan 2024 22:25:58 GMT
< ETag: "8197c82eb0afec6914ce8b78a2838edd-768"
< x-amz-storage-class: STANDARD_IA
< x-amz-server-side-encryption: AES256
< Accept-Ranges: bytes
< Content-Type: image/tiff
< Server: AmazonS3
< Content-Length: 6442213402
As can be seen from the request and resulting content length, the complete file is being requested.
In comparison, this is the equivalent operation using /vsis3/:
gdal_translate -projwin 173.10156468104645 -41.33975369049866 173.1016761949935 -41.339789888978814 -outsize 3155 1024 -CO COMPRESS=DEFLATE --config CPL_CURL_VERBOSE YES /vsis3/****/uploads/****/results/ortho.tiff extract.tiff
* Couldn't find host ****.s3.amazonaws.com in the (nil) file; using defaults
* Trying 52.218.112.58:443...
* Connected to ****.s3.amazonaws.com (52.218.112.58) port 443 (#0)
* ALPN: offers h2,http/1.1
* CAfile: /home/per/miniconda3/ssl/cacert.pem
* CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted http/1.1
* Server certificate:
* subject: CN=*.s3.amazonaws.com
* start date: Oct 10 00:00:00 2023 GMT
* expire date: Jul 3 23:59:59 2024 GMT
* subjectAltName: host "****.s3.amazonaws.com" matched cert's "*.s3.amazonaws.com"
* issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
* SSL certificate verify ok.
* using HTTP/1.1
> GET /?delimiter=%2F&prefix=uploads%2F****%2Fresults%2F HTTP/1.1
Host: ****.s3.amazonaws.com
Accept: */*
x-amz-date: 20240212T084629Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=AKIAUVMHKGTQFWLNY2MH/20240212/eu-west-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=****
< HTTP/1.1 200 OK
< x-amz-id-2: XTCtTFeKIke2pX2ZKkV5qVwj6CyeVgHdwKfN/3qR0Zg43Gxvo1ea+YZDhJWvqulxSdoEgeMtIhk=
< x-amz-request-id: KHDH9X84N0Z1PB4Q
< Date: Mon, 12 Feb 2024 08:46:30 GMT
< x-amz-bucket-region: eu-west-1
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Server: AmazonS3
<
* Connection #0 to host ****.s3.amazonaws.com left intact
* Couldn't find host ****.s3.amazonaws.com in the (nil) file; using defaults
* Found bundle for host: 0x563c0ff48980 [serially]
* Can not multiplex, even if we wanted to
* Re-using existing connection #0 with host ****.s3.amazonaws.com
> GET /uploads/****/results/ortho.tiff HTTP/1.1
Host: ****.s3.amazonaws.com
Accept: */*
Range: bytes=0-16383
x-amz-date: 20240212T084629Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=AKIAUVMHKGTQFWLNY2MH/20240212/eu-west-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=****
< HTTP/1.1 206 Partial Content
< x-amz-id-2: A4AO5bBqpED9AgjpPWqFvb6Q+qU5RQNcpMwm+snewfAveeuZrq8Cn/Pj0J0hGUfaoob4YOyWK+0=
< x-amz-request-id: KHDP25FSKT4Q0T3B
< Date: Mon, 12 Feb 2024 08:46:30 GMT
< Last-Modified: Thu, 25 Jan 2024 22:25:58 GMT
< ETag: "8197c82eb0afec6914ce8b78a2838edd-768"
< x-amz-storage-class: STANDARD_IA
< x-amz-server-side-encryption: AES256
< Accept-Ranges: bytes
< Content-Range: bytes 0-16383/6442213402
< Content-Type: image/tiff
< Server: AmazonS3
< Content-Length: 16384
(A lot of similar 16k requests follow.)
Using /vsis3/ is not an option in my real use case since it would require to share AWS authentication with external users, sharing a presigned URL is ok though.
So, the question is: why doesn't GDAL use range requests for /vsicurl/ in this case, and what can I do to fix it?
