1

Does anyone know how to extract subfiles from a HDF5 file? The GIS remote sensing data I am getting is aggregated into a single .h5 file, but I need it separated into the component files. My aim is NOT to produce TIFFs, but to separate out the component files so that they can be processed one by one.

CONTEXT:

I am running Ubuntu 18.04. I have installed the CSPP VIIRS Flood Detection Software, here:

https://cimss.ssec.wisc.edu/cspp/viirs_flood_v1.1.shtml

I am working with VIIRS SDR data, downloaded from the NOAA CLASS website, here:

https://www.avl.class.noaa.gov/saa/products/search?sub_id=0&datatype_family=VIIRS_SDR&submit.x=24&submit.y=7

The software requires 4 files in a format like this: SVI01_j01_d20181030_t0626485_e0629370_b04908_c20181101224627873308_cspp_dev.h5 SVI02_j01_d20181030_t0626485_e0629370_b04908_c20181101224627873308_cspp_dev.h5 SVI03_j01_d20181030_t0626485_e0629370_b04908_c20181101224627873308_cspp_dev.h5 SVI05_j01_d20181030_t0626485_e0629370_b04908_c20181101224627873308_cspp_dev.h5

However, the NOAA data is delivered in an aggregated file like this:

GIMGO-SVI01-SVI02-SVI03-SVI05_j01_d20181030_t0626485_e0629370_b04908_c20181101224627873308_noac_ops.h5

If I copy the GIMGO file four times and rename it with the four names above (SVI01, etc), the Flood Detection program works perfectly and the results can be read in QGIS 3.8. However that's a miserable lot of manual work if you have a large number of files to process.

Ideally I would like to install software allowing me to disaggregate the GIMGO files with terminal commands. Batch processing of a folder would be best. I am a completely self-taught GIS user with no other coding background, so a literal sequence of commands would be of the most help.

Improve this question
2
  • Are you open to GDAL, either by command line or from QGIS? I had a similar problem with a NTv2 file which needed to be viewed to be evaluated for coverage. The canvas only showed one state but the custodian of the transformation indicated that all the states are present, the answer to this is to use GDAL_Translate with -sds (copy all subdatasets) which produced a GeoTIFF of each state as a separate file, which could presumably be converted using HDF5 gdal.org/drivers/raster/hdf5.html to individual files as your software needs rather than a stack of subdatasets that you have now.
    – Michael Stimson
    Commented Nov 25, 2019 at 2:49
  • 1
    This an excellent idea and could be combined with B-C B's loop, which is already using gdal in python3. I will follow up later.
    – Brian Holmes
    Commented Nov 25, 2019 at 15:00

2 Answers 2

Reset to default
3

The comments offer useful strategies for dealing with .h5 files, but the best solution to this problem lies at the source of the data.

In the "shopping cart" window of the NOAA CLASS website (https://www.avl.class.noaa.gov) click "Advanced options" and select "Package Geolocation with JPSS Data Products: No". This will produce separate SVI files that can be used with the Flood Detection software.

The option "De-aggregate JPSS Data Products" can be left at "No." It will not affect the operation of the software.

Improve this answer
1
  • A solution with negligible extra effort, I like it!
    – Michael Stimson
    Commented Nov 25, 2019 at 23:40
2

I've been using HDF files and when I translate the data from hdf to tiff files it breaks it down into the different layers. Try using this code from python:

import os
import glob
from osgeo import gdal


Folder = '/home/brian/CSPP_FLOOD-May\ 14-0607-0918/disaggregate'

#get all files in Folder that have extension .h5
names = [os.path.splitext(os.path.basename(x))[0] for x in glob.glob(Folder + '/' + '*.h5')]
for name in names:
    Input = Folder + '/' + name + '.h5'
    Output = Folder + '/' + name + '.tif'
    # translate HDF to TIf file
    os.system("gdal_translate -sds " + Input + " " + Output)
Improve this answer
3
  • The erro comes from your file not being called correctly. you have a space in your file. you should deal with spaces in file path in ubuntu using \. I'll update my answer to help you use a for loop. Also Have you checked if you had gdal installed?
    – B-C B.
    Commented Nov 24, 2019 at 20:37
  • yes, they should be read the same way I believe. I updated the code to reflect that.
    – B-C B.
    Commented Nov 24, 2019 at 21:34
  • Well, the loop worked perfectly in the QGIS python panel and I appreciate the suggestion for sure, but all it produced was a very large number of TIFFs. What I need are separate SDR files. They are inside the aggregated file, which needs to be taken apart into its component .h5 files.
    – Brian Holmes
    Commented Nov 25, 2019 at 0:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.