gdal.Open only returns dataset from geotiff when aux.xml file available

Question

With some specific files, gdal.Open() in python only returns a valid dataset if there's an *.aux.xml file available. The aux file has been generated by QGIS when opening there manually. If no aux file, gdal.Open returns a NoneType.

It is all described in this github repo: https://github.com/awesomemaptools/gdalissue

Here's the python script:

import sys
from osgeo import gdal
filename = sys.argv[1]
ds = gdal.Open(filename)
print(type(ds))
band = ds.GetRasterBand(1)
data = band.ReadAsArray()
print('value at 1,1:', data[1][1])

When no aux file is available in specific cases:

$ ./gdalOpen.py tmpfile1.tif
ERROR 1: _TIFFVSetField:tmpfile1.tif: Null count for "GeoDoubleParams" (type 12, writecount -1, passcount 1)
<type 'NoneType'>
Traceback (most recent call last):
  File "./gdalOpen.py", line 16, in <module>
    band = ds.GetRasterBand(1)
AttributeError: 'NoneType' object has no attribute 'GetRasterBand'

Any suggestions?

Answer 1

This is a bug in older versions of GDAL when libtiff emitted a non-fatal error. GDAL could still open the file (eg with gdalinfo) but the python bindings failed.

This is the ticket #5616 and this is the relevant commit that fixes it:

GTiff: to make Python bindings happy, avoid emitting CE_Failure errors due to libtiff errors when we still manage to open the file (#5616)

The GDAL 2.2 python bindings can open the file.

GDAL 1.11

$ gdalinfo --version
GDAL 1.11.3, released 2015/09/16

$ gdalinfo tmpfile1.tif 
ERROR 1: _TIFFVSetField:tmpfile1.tif: Null count for "GeoDoubleParams" (type 12, writecount -1, passcount 1)
Driver: GTiff/GeoTIFF
Files: tmpfile1.tif
Size is 380, 120
Coordinate System is:
<snip>
Metadata:
<snip>
Corner Coordinates:
Upper Left  (-121.9515470,  36.6054000) (121d57' 5.57"W, 36d36'19.44"N)
Lower Left  (-121.9515470,  36.5757660) (121d57' 5.57"W, 36d34'32.76"N)
Upper Right (-121.9086310,  36.6054000) (121d54'31.07"W, 36d36'19.44"N)
Lower Right (-121.9086310,  36.5757660) (121d54'31.07"W, 36d34'32.76"N)
Center      (-121.9300890,  36.5905830) (121d55'48.32"W, 36d35'26.10"N)
Band 1 Block=380x8 Type=Float32, ColorInterp=Gray

$ python gdalOpen.py tmpfile1.tif
1.11.3
Traceback (most recent call last):
  File "gdalOpen.py", line 8, in <module>
    ds = gdal.Open(filename)
  File "/usr/lib/python2.7/dist-packages/osgeo/gdal.py", line 1800, in Open
    return _gdal.Open(*args)
RuntimeError: _TIFFVSetField:tmpfile1.tif: Null count for "GeoDoubleParams" (type 12, writecount -1, passcount 1)

GDAL 2.2

$ gdalinfo --version
GDAL 2.2.0, released 2017/04/28

$ gdalinfo tmpfile1.tif 
Warning 1: _TIFFVSetField:tmpfile1.tif: Null count for "GeoDoubleParams" (type 12, writecount -1, passcount 1)
Driver: GTiff/GeoTIFF
Files: tmpfile1.tif
Size is 380, 120
Coordinate System is:
<snip>
Metadata:
<snip>
Corner Coordinates:
Upper Left  (-121.9515470,  36.6054000) (121d57' 5.57"W, 36d36'19.44"N)
Lower Left  (-121.9515470,  36.5757660) (121d57' 5.57"W, 36d34'32.76"N)
Upper Right (-121.9086310,  36.6054000) (121d54'31.07"W, 36d36'19.44"N)
Lower Right (-121.9086310,  36.5757660) (121d54'31.07"W, 36d34'32.76"N)
Center      (-121.9300890,  36.5905830) (121d55'48.32"W, 36d35'26.10"N)
Band 1 Block=380x8 Type=Float32, ColorInterp=Gray

$ python gdalOpen.py tmpfile1.tif
2.2.0
Warning 1: _TIFFVSetField:tmpfile1.tif: Null count for "GeoDoubleParams" (type 12, writecount -1, passcount 1)
value at 1,1: 0.465083

The shortened gdalOpen.py script:

import sys
from osgeo import gdal
gdal.UseExceptions()
filename = sys.argv[1]
print(gdal.__version__)
ds = gdal.Open(filename)
data = ds.GetRasterBand(1).band.ReadAsArray(0,0,2,2)
print('value at 1,1:', data[1][1])

In GDAL 1.11, running gdalinfo -stats tmpfile1.tif recreates the .aux.xml and allows the python bindings to open the raster.