2
ogrinfo --debug on --config CPL_CURL_VERBOSE YES -ro -al -so "/vsicurl/http://ascendxyzdatastagingweu.blob.core.windows.net/915295c046e0428e8f4fc94ce9188133-upload9/JORDSTYKKE.shp"

Given the above command I am trying to use vsicurl to read a shapefile from a HTTP storage. I have tested this on smaller shapefiles where it works fine, but with this large shapefile it runs into errors.

The file JORDSTYKKE.shx has on local disk a size of length 19818524 bytes and the command works on local disk just fine. Inspecting the curl requests seen below, this is also the length returned Content-Length: 19818524. I don't know why, but it seems that it later tries to download content outside this range.

VSICURL: Downloading 16384-19824639 where 19824639 is larger than 19818524.

Full trace:

> HEAD /915295c046e0428e8f4fc94ce9188133-upload9/JORDSTYKKE.shx HTTP/1.1
Host: ascendxyzdatastagingweu.blob.core.windows.net
Accept: */*

< HTTP/1.1 200 OK
< Content-Length: 19818524
< Content-Type: application/octet-stream
< Content-MD5: GyMVOaqiZqoPXVmHAUXIIA==
< Last-Modified: Sun, 21 Dec 2014 19:56:30 GMT
< ETag: 0x8D1EB6F58E378FB
< Vary: Origin
* Server Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 is not blacklisted
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: 2047f20d-0001-0035-243b-710e22000000
< x-ms-version: 2009-09-19
< x-ms-meta-CbModifiedTime: Thu, 17 Jul 2014 22:23:33 GMT
< x-ms-lease-status: unlocked
< x-ms-blob-type: BlockBlob
< Date: Sun, 21 Dec 2014 20:25:28 GMT
<
* Connection #0 to host ascendxyzdatastagingweu.blob.core.windows.net left intact
VSICURL: GetFileSize(http://ascendxyzdatastagingweu.blob.core.windows.net/915295c046e0428e8f4fc94ce9188133-upload9/JORDSTYKKE.shx)=19818524  response_cod
e=200
VSICURL: Downloading 0-16383 (http://ascendxyzdatastagingweu.blob.core.windows.net/915295c046e0428e8f4fc94ce9188133-upload9/JORDSTYKKE.shp)...
* Couldn't find host ascendxyzdatastagingweu.blob.core.windows.net in the _netrc file; using defaults
* Found bundle for host ascendxyzdatastagingweu.blob.core.windows.net: 0xfe89f0
* Re-using existing connection! (#0) with host ascendxyzdatastagingweu.blob.core.windows.net
* Connected to ascendxyzdatastagingweu.blob.core.windows.net (168.61.57.78) port 80 (#0)
> GET /915295c046e0428e8f4fc94ce9188133-upload9/JORDSTYKKE.shp HTTP/1.1
Range: bytes=0-16383
Host: ascendxyzdatastagingweu.blob.core.windows.net
Accept: */*

< HTTP/1.1 206 Partial Content
< Content-Length: 16384
< Content-Type: application/octet-stream
< Content-Range: bytes 0-16383/1341796588
< Last-Modified: Sun, 21 Dec 2014 20:00:31 GMT
< ETag: 0x8D1EB6FE8A5B02B
< Vary: Origin
* Server Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 is not blacklisted
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: 2047f21c-0001-0035-333b-710e22000000
< x-ms-version: 2009-09-19
< x-ms-meta-CbModifiedTime: Thu, 17 Jul 2014 22:23:33 GMT
< x-ms-lease-status: unlocked
< x-ms-blob-type: BlockBlob
< Date: Sun, 21 Dec 2014 20:25:28 GMT
<
* Connection #0 to host ascendxyzdatastagingweu.blob.core.windows.net left intact
VSICURL: Got reponse_code=206
VSICURL: Downloading 0-16383 (http://ascendxyzdatastagingweu.blob.core.windows.net/915295c046e0428e8f4fc94ce9188133-upload9/JORDSTYKKE.shx)...
* Couldn't find host ascendxyzdatastagingweu.blob.core.windows.net in the _netrc file; using defaults
* Found bundle for host ascendxyzdatastagingweu.blob.core.windows.net: 0xfe89f0
* Re-using existing connection! (#0) with host ascendxyzdatastagingweu.blob.core.windows.net
* Connected to ascendxyzdatastagingweu.blob.core.windows.net (168.61.57.78) port 80 (#0)
> GET /915295c046e0428e8f4fc94ce9188133-upload9/JORDSTYKKE.shx HTTP/1.1
Range: bytes=0-16383
Host: ascendxyzdatastagingweu.blob.core.windows.net
Accept: */*

< HTTP/1.1 206 Partial Content
< Content-Length: 16384
< Content-Type: application/octet-stream
< Content-Range: bytes 0-16383/19818524
< Last-Modified: Sun, 21 Dec 2014 19:56:30 GMT
< ETag: 0x8D1EB6F58E378FB
< Vary: Origin
* Server Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 is not blacklisted
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: 2047f23b-0001-0035-523b-710e22000000
< x-ms-version: 2009-09-19
< x-ms-meta-CbModifiedTime: Thu, 17 Jul 2014 22:23:33 GMT
< x-ms-lease-status: unlocked
< x-ms-blob-type: BlockBlob
< Date: Sun, 21 Dec 2014 20:25:28 GMT
<
* Connection #0 to host ascendxyzdatastagingweu.blob.core.windows.net left intact
VSICURL: Got reponse_code=206
VSICURL: Downloading 16384-19824639 (http://ascendxyzdatastagingweu.blob.core.windows.net/915295c046e0428e8f4fc94ce9188133-upload9/JORDSTYKKE.shx)...
* Couldn't find host ascendxyzdatastagingweu.blob.core.windows.net in the _netrc file; using defaults
* Found bundle for host ascendxyzdatastagingweu.blob.core.windows.net: 0xfe89f0
* Re-using existing connection! (#0) with host ascendxyzdatastagingweu.blob.core.windows.net
* Connected to ascendxyzdatastagingweu.blob.core.windows.net (168.61.57.78) port 80 (#0)
> GET /915295c046e0428e8f4fc94ce9188133-upload9/JORDSTYKKE.shx HTTP/1.1
Range: bytes=16384-19824639
Host: ascendxyzdatastagingweu.blob.core.windows.net
Accept: */*

< HTTP/1.1 206 Partial Content
< Content-Length: 19802140
< Content-Type: application/octet-stream
< Content-Range: bytes 16384-19818523/19818524
< Last-Modified: Sun, 21 Dec 2014 19:56:30 GMT
< ETag: 0x8D1EB6F58E378FB
< Vary: Origin
* Server Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 is not blacklisted
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: 2047f278-0001-0035-0f3b-710e22000000
< x-ms-version: 2009-09-19
< x-ms-meta-CbModifiedTime: Thu, 17 Jul 2014 22:23:33 GMT
< x-ms-lease-status: unlocked
< x-ms-blob-type: BlockBlob
< Date: Sun, 21 Dec 2014 20:25:28 GMT
<
* Connection #0 to host ascendxyzdatastagingweu.blob.core.windows.net left intact
VSICURL: Got reponse_code=206
ERROR 4: Failed to read all values for 2477303 records in .shx file.
FAILURE:
Unable to open datasource `/vsicurl/http://ascendxyzdatastagingweu.blob.core.windows.net/915295c046e0428e8f4fc94ce9188133-upload9/JORDSTYKKE.shp' with th
e following drivers.
3
  • There was some trouble with discussion between GDAL and the web server hosting the shapefile. Fixed now in GDAL trac.osgeo.org/gdal/ticket/5786.
    – user30184
    Dec 22, 2014 at 5:36
  • Cool Thanks. Proof of it being worth reporting issues :) Dec 22, 2014 at 6:28
  • Thank Even. I just happened to see your bug report and what was the result. It is worth making good reports like the one by you with detailed description and test case that could be reproduced.
    – user30184
    Dec 22, 2014 at 9:37

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.