3

I'm doing quite heavy processing of Earth Observation data, and it appears that a lot of my workflow can be speeded up by using HDF5 for local storage. I mostly work with large stacks of raster files, and HDF5 as well as numpexpr allow me to very efficiently process large amounts of data. I would like to have as my external format HDF5, but my data can change geographic projection and geolocation. Ideally, I'd like GDAL to be able to use that information (not too worried about other tools), but I can't seem to find how to define that for these files.

5

I am currently in the same position as you. HDF5 is a great format, but you need to define and characterize it yourself. Python has a very handy module called h5py, which is quite straight-forward. It is dependent on numpy and the documentation shows you some examples how to query and store your arrays. HDF5 also supports heterogeneous datasets, thus it is all about how you define your container and dimension structure. At the last SciPy 2015 conference there was a very convincing talk about the future of HDF5 and Python.

You might also consider netCDF4, which uses hdf5 as backbone as well and has somewhat more support from the Earth-observation community. IMHO defining dimensions (e.g. x-axis for longitude, y-axis is latitude and so on) is actually easier with the netCDF syntax.

  • Thanks, but actually using them for latitude and longitude is fine (climate community has been using it for ever), the problem is when you want to have a local projection (e.g. UTM) that can change, and that needs to be interpreted by third party libraries (GDAL) – Jose Sep 5 '15 at 17:55
3

The GDAL HDF5 driver officially only supports one product:HDF5 OMI/Aura Ozone (O3) Total Column 1-Orbit L2 Swath 13x24km (Level-2 OMTO3) (see the Limitation section), and it's georeferencing. Also, this driver is only available for reading (not writing), which may make your efforts futile. That being said, this may not be a severe limitation for you: GDAL can still read HDF5 data created using other means:

>>> ds = h5py.File('new_file.he5', 'w')
>>> grp = ds.create_group('MyTestGroup')
>>> data = grp.create_dataset('MyTestData', data=numpy.random.randint(1,5,(5,5)), dtype='uint8')
>>> data[:]
array([[4, 3, 4, 3, 3],
       [1, 4, 3, 3, 3],
       [2, 1, 1, 3, 3],
       [3, 3, 4, 2, 4],
       [3, 4, 4, 2, 3]], dtype=uint8)
>>> del data, grp, ds
>>> ds = gdal.Open('new_file.he5')
>>> ds.ReadAsArray()
array([[4, 3, 4, 3, 3],
       [1, 4, 3, 3, 3],
       [2, 1, 1, 3, 3],
       [3, 3, 4, 2, 4],
       [3, 4, 4, 2, 3]], dtype=uint8)

If you'll need georeferencing info. (maybe for display or transformation purposes), perhaps you could store it in the metadata (which apparently GDAL ignores) and access it when you need it. Here is an arbitrary example:

>>> ds = h5py.File('new_file.he5', 'r+')
>>> data = ds['MyTestGroup/MyTestData']
>>> sr = osr.SpatialReference()
>>> sr.ImportFromEPSG(26911)
0
>>> data.attrs['Projection'] = sr.ExportToWkt()
>>> data.attrs['GeoTransform'] = (426850.21630573744, 18.907041540963913, 0.0, 5765300.131841155, 0.0, -18.907041540963938)
  • So the way you georeference is by adding attributes that are GDAL-specific... Mmmm, not great, maybe netcdf with CF-1.7 might be the best solution. – Jose Oct 19 '16 at 18:07
  • I would not throw away HDF5, it is a fantastic library. If georeferencing is your only caveat, I'm sure you can find a work around to use GDAL's functionality. – Devin Cairns Oct 19 '16 at 18:39
0

I completely understand the point where you're coming from but you need to understand what Geo referencing & Projection is and how it is implemented.

The standard supported Format type for such images across Industry is Geo-Tiff. Initially the Image data is taken and is resampled according to various different pre-processing steps that are to take place after which the Latitude , Longitude ,Scale,Skew of the image are calculated and an appropriate Projection system is choosen. All of this information is put into Meta data of the Geotiff so that the processing software can understand.

Similarly , after resampling, you can handle it by putting your Geo-info and Proj-info in metadata for handling in HDF5 format as well.

I hope I made sense.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.