Which data formats are best for Open Data distribution?

Question

What are the pros and cons of different data formats (performance, file size, etc.) when considering open data distribution?

Our organisation wants to publish data as open data. However, there is no clear idea on which data formats to use. Ofcourse, the more 'open' a data format is, the easier it is to use.

Which data formats are the most 'open' and therefore most usable for the distribution of Open Data when taking the following types in consideration?:

• raster data (I'm thinking: GeoTIFF, Erdas Imagine IMG?)
• vector data (I'm thinking: GML, CSV, ESRI Shapefile, DXF?)
• tabular data (I'm thinking: CSV?)
• 3D data (I'm thinking: CityGML?)
• 3D point coulds / LIDAR (I'm thinking: LAS?)
• am I forgetting something here?

Also, if there is documentation about open data formats I'm very interested if you would like to share.

Answer 1

The city of Vienna's open data initiative (http://data.wien.gv.at) uses Geoserver to provide access to raster and vector geodata via Geoserver WMS and WFS services. This has many advantages: Users can download data in different formats for offline use (e.g. geojson, KML, or zipped Shapefiles) or use the services live by embedding them in online maps or GIS projects.

Answer 2

For tabular csv. Excel is at best overly complicated and at worst totally inaccessible. Access is not accessible and PDF is a slap in the face.

For geospatial use geojson, it's text it's well supported and doesn't have the technical restrictions that the only other viable format (shapefile) has. Also unless you have a very good reason it should be in WGS84, bearing in mind that most users will be in another state and will not want state plane.

Answer 3

I quite like NetCDF for continuous/array data (i.e. rasters). Pros for NetCDF are:

• NetCDF is self describing (i.e., data definitions are available through the file header) so you don't need to supply secondary metadata files
• NetCDF4 allows for storage of n-dimensional data (using the HDF5 data format on disk, which is a bonus as this allows files as big as your OS can handle). This comes with reasonable compression and fast access to the data. Note that NetCDF3 doesn't support n-dimensional data, and has a file size limit of roughly 2GB on a 32-bit system.
• NetCDF is an open format so accessing the data is generally not a problem as well through common libraries. For example, in python it's simple enough from scipy to read in a slice of data:

from scipy.io import netcdf
f = netcdf.netcdf_file('source.nc')
print(nc.dimensions) #take a look at the dimensions of the data
print(nc.variables)  #A dictionary containing all the variables
nc.variables["some_data"].dimensions #The dimensions this variable is in, e.g. lat, lon
out_array = nc.variables["some_data"].data
f.close() #and we're done

The only downside to NetCDF4 that I can see is the not-great support in standard GIS packages like ArcGIS and QGIS (though I dearly would love to be corrected on this!).

EDIT Some other packages that support NetCDF

Some standard programming languages that support NetCDF (though to be fair, anything that can read HDF can read NetCDF4):

• Java
• Perl
• C++

For maths and stats users you have:

• matlab
• IDL
• R

Specifically in GIS:

• GDAL will convert the data for you
• Likewise FME
• ArcGIS supports NetCDF (though it's not the best level of support in my experience)
• There is a QGIS Plugin in development

If you want to quickly look at a NetCDF file I'd use the cross-platform Panoply from NASA. And if you're interested in more, UCAR Unidata has a list of software.

Answer 4

I would say:

• Shapefiles or GML for vector data
• .obj-Files for 3D models
• .xyz (simple CSV) for point clouds
• CSV for tabular data
• GeoTIFF for raster data

These formats are easily readable by Open Source Software and are easily transformable to any other format needed for specific applications.

Also +1 for making data open!

Answer 5

Virtually this exact same question came up at opendata.SE: What are the most useful formats in which to release geospatial data?

So, hopefully I'm not violating any policies in quoting my own answer there:

My experience, making maps from quite a few government datasets:

For point data, CSV is the best, with "lat" and "lon" columns. Very easy to work with in a wide range of tools, including text editors, spreadsheets, etc. There are two downsides:

1. GDAL requires a .vrt companion file.
2. The naming of the lat and lon columns is not totally standard. Many tools are pretty liberal in what they accept.

For lines and polygons, in decreasing order of preference:

1. GeoJSON. Easy to work with, and the ability to edit in a text editor or with geojson.io is a real bonus, if you need to do search/replace, remove a couple of weird objects or copy and paste from one file to another. Another benefit is that non-GIS developers can make sense of it. Only issues I've run into is when someone provides data as say MultiPoint instead of Point.
2. Shapefile. Very widely supported, but with two inconvenient points. First, it's a collection of files, so you have to pass around a .zip and extract it. Second, field names are limited to 10 characters. They're hard to edit for your average non-GIS person.
3. KML/KMZ. These often have a lot of irrelevant cruft (styling, icons, etc), and attributes are sometimes encoded as mini HTML tables, which are really hard to work with. At least you can edit them easily with Google tools.

Honestly, though, the best answer is probably "all of them". Do everyone a favour and release the data in CSV (if point), GeoJSON, zipped Shapefile and KMZ.