NetCDF file size calculation

Question

I'm planning to use the NetCDF file format for storing my time-series observation data on a server. I would like to use the "NetCDF-CF" conventions. Before I start writing any scripts for importing data from my SQL Server database to NetCDF, I'd like to make an estimate of the disk space requirements for the NetCDF file.

My data looks like this

• 1000 stations (each station has ID, Latitude, Longitude, Elevation)

• 10 quantities (for example, air temperature, air pressure,
  precipitation..)

• 10 years of data with hourly time resolution, that is 10 years*365 days*24 hours = 87600 hours per station and quantity

• the observation data value fits in 4-byte integer (if it's a decimal number then I can scale the value to integer.) So I need 10 years*365 days*24 hours*4 bytes = 350400 bytes per station and quantity.

• I presume that I also need to store the "time" information somewhere in a suitable format. Usually 8 bytes is enough to store a time value, so I need 10 years*365 days*24 hours*8 bytes = 700800 bytes for the time column.

Now calculating the NetCDF file size:

( [1000 stations] * [10 quantities] * [87600 hours] * [4 bytes per value]) + [700800 bytes for time] = 3504700800 bytes = 3.264 Gigabytes total

Is this roughly correct?

Or does NetCDF have any significant "storage overhead" like most relational databases do (for example in Postgres there is a 23 byte overhead per data row or in MSSQL it's 7 extra bytes per data row)?

Answer 1

Yes, your calculated size is roughly correct.

What little storage overhead is used in the netCDF classic format comes from alignment requirements: each variable's data must begin aligned on a 4-byte boundary, and similarly each record must begin on a 4-byte boundary. The 64-bit offset variant of the classic format requires a little more overhead for storing 64-bit file offsets to the beginning of each variable, rather than the 32-bit offsets used in the classic format.

If you need to use the netCDF-4 format, perhaps because you need compression or chunking, there is more overhead in the B-tree structures used to index data chunks. However, for as much data as you have, that overhead would be small unless you specified very small chunk sizes. leading to a large number of B-tree nodes.