2

I'm trying to work with the climate projection data that NASA released, and is hosted publicly on S3: http://aws.amazon.com/nasa/nex/. The data is available at s3://nasanex/NEX-DCP30.

The code I'm working with uses the rasterio python library to export band information.

I worked out the pipeline on my Macbook, which had the netcdf package installed by homebrew, GDAL built from source (trunk branch) and configured with python bindings, and rasterio installed with pip.

I am able to work with the netCDF files fine on my Macbook. For instance, if I pull down the file at s3://nasanex/NEX-DCP30/BCSD/historical/mon/atmos/tasmax/r1i1p1/v1.0/CONUS/tasmax_amon_BCSD_historical_r1i1p1_CONUS_inmcm4_200501-200512.nc, and run gdalinfo it, the first 3 lines are this:

Driver: netCDF/Network Common Data Format
Files: tasmax_amon_BCSD_historical_r1i1p1_CONUS_inmcm4_200501-200512.nc
Size is 7025, 3105

Great.

I can do the same setup on an Ubuntu machine, using GDAL trunk or the 1.10.1 tag, and have HDF5 and netCDF enabled (as seen by this output in GDAL's ./configure script:

HDF5 support:              yes
NetCDF support:            yes

However, when I do a gdalinfo on the same file, I get a different driver being used:

Driver: HDF5/Hierarchical Data Format Release 5
Files: tasmax_amon_BCSD_historical_r1i1p1_CONUS_inmcm4_200501-200512.nc
Size is 512, 512

I've tried this on a Ubuntu VM on my Macbook, an Ubuntu AMI on EC2, and a workstation that has Ubuntu as it's native OS. All the same.

Worse, when I try to open the file with rasterio, while on my Macbook everything runs great, on Ubuntu I get a a segmentation fault when using rasterio.open(...)

Segmentation fault (core dumped)

This only happens with the NEX dataset netCDF files. I can read another netCDF file I got from a different source just fine...gdalinfo produces the correct Driver for that netCDF file:

Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
Driver: netCDF/Network Common Data Format
Files: BCSD_0.5deg_tas_Amon_access1-0_rcp85_r1i1p1_200601-210012.nc
Size is 720, 278

Does anyone have any insight into why these netCDF files would be readable on my Macbook and unreadable (and considered to be corrupt HDF5 datasets) on Ubuntu?

1

2 Answers 2

2

Turns out in order to read the NEX climate data, I needed very specific versions of the HDF5 and netCDF libraries. I'm not sure which part of the combination was not working for me, since the hours I lost getting it to work was enough time debugging the problem. But I did codify the solution into ansible provisioning scripts. See the 'netcdf' role in the deployment folder for this repo: https://github.com/lossyrob/nex-chunker-worker

I build manually with these codebases:

szip v2.1

hdf5 v1.8.12

netcdf: v4.3.2

gdal: commit 950849f98ceba0dcdaa0677abe59411caf3329b8 at the github's osgeo/gdal repo.

1
1

I can't say why gdal picks the HDF5 driver with the NEX files on Ubuntu. It shouldn't as the NetCDF driver is tried before the HDF5 driver (the order that drivers are listed in gdalinfo --formats shows the order they are tried). You may not have HDF5 support on your Mac. If you did, you might come across the same issue.

To workaround, you could try disabling the HDF5 driver. Either with an environment variable, or by deregistering it in python.

Environment variable:

GDAL_SKIP=HDF5

Deregister:

gdal.GetDriverByName('HDF5').Deregister()

However... as you have NetCDF support on Ubuntu and the NetCDF driver should be tried before the HDF5 driver, there may be an issue with the NetCDF driver reading those files and my suggested workaround wouldn't help in that case. I'd suggest preparing a small working script that demonstrates the issue and submit as a ticket to the GDAL issue tracker.

5
  • I tried disabling HDF5 support in the build, using the flag --without-hdf5. I didn't know about the environment variable that also allowed me to skip drivers; but the result is the same. I get ERROR 4: 'tasmax_amon_BCSD_historical_r1i1p1_CONUS_inmcm4_200501-200512.nc' not recognised as a supported file format. Thanks for your suggestion (and cluing me into how to disable drivers in GDAL on the fly). I'll write up a ticket to GDAL unless someone here has any other ideas.
    – lossyrob
    Jan 19, 2015 at 4:40
  • Ahhh you built from source. Have you tried installing gdal from the ubuntugis-unstable ppa? How do you download NEX data from S3 to test?
    – user2856
    Jan 19, 2015 at 6:36
  • See also this ticket - trac.osgeo.org/ubuntugis/ticket/27
    – user2856
    Jan 19, 2015 at 8:22
  • You'd need s3 credentials to get to the NASA NEX dataset, and download it with something like s3cmd. To make it easier I uploaded to google drive: drive.google.com/file/d/0B7ubrF-X8So3cTNPdlBNZExGdjg/… . I haven't tried installing GDAL through package management, I thought that I would need to build it manually to tie in the netCDF support; I'll give that a shot.
    – lossyrob
    Jan 19, 2015 at 19:52
  • Added NETCDF: in front of it gives me the proper driver. That's great for now, this is a pretty pressing issue so I'm going to run with that. Thank you so, so much Luke!
    – lossyrob
    Jan 19, 2015 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.