1

Does anyone know how to extract subfiles from a HDF5 file? The GIS remote sensing data I am getting is aggregated into a single .h5 file, but I need it separated into the component files. My aim is NOT to produce TIFFs, but to separate out the component files so that they can be processed one by one.

CONTEXT:

I am running Ubuntu 18.04. I have installed the CSPP VIIRS Flood Detection Software, here:

https://cimss.ssec.wisc.edu/cspp/viirs_flood_v1.1.shtml

I am working with VIIRS SDR data, downloaded from the NOAA CLASS website, here:

https://www.avl.class.noaa.gov/saa/products/search?sub_id=0&datatype_family=VIIRS_SDR&submit.x=24&submit.y=7

The software requires 4 files in a format like this: SVI01_j01_d20181030_t0626485_e0629370_b04908_c20181101224627873308_cspp_dev.h5 SVI02_j01_d20181030_t0626485_e0629370_b04908_c20181101224627873308_cspp_dev.h5 SVI03_j01_d20181030_t0626485_e0629370_b04908_c20181101224627873308_cspp_dev.h5 SVI05_j01_d20181030_t0626485_e0629370_b04908_c20181101224627873308_cspp_dev.h5

However, the NOAA data is delivered in an aggregated file like this:

GIMGO-SVI01-SVI02-SVI03-SVI05_j01_d20181030_t0626485_e0629370_b04908_c20181101224627873308_noac_ops.h5

If I copy the GIMGO file four times and rename it with the four names above (SVI01, etc), the Flood Detection program works perfectly and the results can be read in QGIS 3.8. However that's a miserable lot of manual work if you have a large number of files to process.

Ideally I would like to install software allowing me to disaggregate the GIMGO files with terminal commands. Batch processing of a folder would be best. I am a completely self-taught GIS user with no other coding background, so a literal sequence of commands would be of the most help.

  • Are you open to GDAL, either by command line or from QGIS? I had a similar problem with a NTv2 file which needed to be viewed to be evaluated for coverage. The canvas only showed one state but the custodian of the transformation indicated that all the states are present, the answer to this is to use GDAL_Translate with -sds (copy all subdatasets) which produced a GeoTIFF of each state as a separate file, which could presumably be converted using HDF5 gdal.org/drivers/raster/hdf5.html to individual files as your software needs rather than a stack of subdatasets that you have now. – Michael Stimson Nov 25 '19 at 2:49
  • 1
    This an excellent idea and could be combined with B-C B's loop, which is already using gdal in python3. I will follow up later. – Brian Holmes Nov 25 '19 at 15:00
2

The comments offer useful strategies for dealing with .h5 files, but the best solution to this problem lies at the source of the data.

In the "shopping cart" window of the NOAA CLASS website (https://www.avl.class.noaa.gov) click "Advanced options" and select "Package Geolocation with JPSS Data Products: No". This will produce separate SVI files that can be used with the Flood Detection software.

The option "De-aggregate JPSS Data Products" can be left at "No." It will not affect the operation of the software.

|improve this answer|||||
  • A solution with negligible extra effort, I like it! – Michael Stimson Nov 25 '19 at 23:40
0

I've been using HDF files and when I translate the data from hdf to tiff files it breaks it down into the different layers. Try using this code from python:

import os
import glob
from osgeo import gdal


Folder = '/home/brian/CSPP_FLOOD-May\ 14-0607-0918/disaggregate'

#get all files in Folder that have extension .h5
names = [os.path.splitext(os.path.basename(x))[0] for x in glob.glob(Folder + '/' + '*.h5')]
for name in names:
    Input = Folder + '/' + name + '.h5'
    Output = Folder + '/' + name + '.tif'
    # translate HDF to TIf file
    os.system("gdal_translate -sds " + Input + " " + Output)

|improve this answer|||||
  • The erro comes from your file not being called correctly. you have a space in your file. you should deal with spaces in file path in ubuntu using \. I'll update my answer to help you use a for loop. Also Have you checked if you had gdal installed? – B-C B. Nov 24 '19 at 20:37
  • yes, they should be read the same way I believe. I updated the code to reflect that. – B-C B. Nov 24 '19 at 21:34
  • Well, the loop worked perfectly in the QGIS python panel and I appreciate the suggestion for sure, but all it produced was a very large number of TIFFs. What I need are separate SDR files. They are inside the aggregated file, which needs to be taken apart into its component .h5 files. – Brian Holmes Nov 25 '19 at 0:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.