15
votes

What are the pros and cons of different data formats (performance, file size, etc.) when considering open data distribution?

Our organisation wants to publish data as open data. However, there is no clear idea on which data formats to use. Ofcourse, the more 'open' a data format is, the easier it is to use.

Which data formats are the most 'open' and therefore most usable for the distribution of Open Data when taking the following types in consideration?:

  • raster data (I'm thinking: GeoTIFF, Erdas Imagine IMG?)
  • vector data (I'm thinking: GML, CSV, ESRI Shapefile, DXF?)
  • tabular data (I'm thinking: CSV?)
  • 3D data (I'm thinking: CityGML?)
  • 3D point coulds / LIDAR (I'm thinking: LAS?)
  • am I forgetting something here?

Also, if there is documentation about open data formats I'm very interested if you would like to share.

5
  • 2
    for vector, you might also consider geojson and kml
    – neuhausr
    Jul 1, 2013 at 13:41
  • 1
    did you see this link? gis.stackexchange.com/questions/61744/…
    – user681
    Jul 1, 2013 at 13:52
  • 4
    You need to differentiate between data exchange formats and data storage format. geojson for example is an excellent Data Exchange format, but sucks as a Data storage format. I am assuming you are only concerned with the format for distributing data (i.e data exchange). Is that correct? Jul 1, 2013 at 14:26
  • @DevdattaTengshe: Good point! For now, the intention is to distribute the data in the most convenient exchange format. Jul 1, 2013 at 14:52
  • Thanks everyone for the feedback. Also, some tips about file types to use vs file size would be very helpfull. Jul 2, 2013 at 10:01

5 Answers 5

5
votes

The city of Vienna's open data initiative (http://data.wien.gv.at) uses Geoserver to provide access to raster and vector geodata via Geoserver WMS and WFS services. This has many advantages: Users can download data in different formats for offline use (e.g. geojson, KML, or zipped Shapefiles) or use the services live by embedding them in online maps or GIS projects.

1
5
votes

For tabular csv. Excel is at best overly complicated and at worst totally inaccessible. Access is not accessible and PDF is a slap in the face.

For geospatial use geojson, it's text it's well supported and doesn't have the technical restrictions that the only other viable format (shapefile) has. Also unless you have a very good reason it should be in WGS84, bearing in mind that most users will be in another state and will not want state plane.

5
votes

I quite like NetCDF for continuous/array data (i.e. rasters). Pros for NetCDF are:

  • NetCDF is self describing (i.e., data definitions are available through the file header) so you don't need to supply secondary metadata files
  • NetCDF4 allows for storage of n-dimensional data (using the HDF5 data format on disk, which is a bonus as this allows files as big as your OS can handle). This comes with reasonable compression and fast access to the data. Note that NetCDF3 doesn't support n-dimensional data, and has a file size limit of roughly 2GB on a 32-bit system.
  • NetCDF is an open format so accessing the data is generally not a problem as well through common libraries. For example, in python it's simple enough from scipy to read in a slice of data:
from scipy.io import netcdf
f = netcdf.netcdf_file('source.nc')
print(nc.dimensions) #take a look at the dimensions of the data
print(nc.variables)  #A dictionary containing all the variables
nc.variables["some_data"].dimensions #The dimensions this variable is in, e.g. lat, lon
out_array = nc.variables["some_data"].data
f.close() #and we're done

The only downside to NetCDF4 that I can see is the not-great support in standard GIS packages like ArcGIS and QGIS (though I dearly would love to be corrected on this!).

EDIT Some other packages that support NetCDF

Some standard programming languages that support NetCDF (though to be fair, anything that can read HDF can read NetCDF4):

For maths and stats users you have:

Specifically in GIS:

  • GDAL will convert the data for you
  • Likewise FME
  • ArcGIS supports NetCDF (though it's not the best level of support in my experience)
  • There is a QGIS Plugin in development

If you want to quickly look at a NetCDF file I'd use the cross-platform Panoply from NASA. And if you're interested in more, UCAR Unidata has a list of software.

2
  • NetCDF is an awful choice it really has no support outside of python. It might have good support, but tiffs, png, and jpeg have support in literally every language.
    – Calvin
    Jul 4, 2013 at 2:37
  • 2
    I strongly disagree. I've edited my response above to show a quick list of packages that support NetCDF. In my experience it's a format of choice for any multidimensional scientific data (e.g. astronomy and meteorology). PNG and TIFF aren't bad for distributing raster data, and certainly viewing the data is easy, but they don't scale well to large amounts of multidimensional data. Don't ever use JPEG to distribute scientific data (though if you're sending someone a map it works perfectly well).
    – om_henners
    Jul 5, 2013 at 0:31
4
votes

I would say:

  • Shapefiles or GML for vector data
  • .obj-Files for 3D models
  • .xyz (simple CSV) for point clouds
  • CSV for tabular data
  • GeoTIFF for raster data

These formats are easily readable by Open Source Software and are easily transformable to any other format needed for specific applications.

Also +1 for making data open!

6
  • 2
    I'll be interested to know why you have suggested Shapefiles and GML for vector data. Both of them are terrible formats. The only saving grace of GML, is that it is an OGC format. Jul 1, 2013 at 15:37
  • 1
    Shapefiles are readable in many applications, and can be transformed to something different without problems. What would you suggest?
    – til_b
    Jul 2, 2013 at 6:59
  • 3
    avoid shapefiles. They work, but they have serious technical limitations.
    – nickves
    Jul 2, 2013 at 9:33
  • 1
    So what do you suggest that does not have the technical limitations of shapefiles?
    – til_b
    Jul 2, 2013 at 9:55
  • 2
    @til_b GeoTIFF is a nice format from a perspective of being 'open'. However, for storage (or offering it as a download) it is terrible, because the files can get huge. Do you know of an open raster format that offers lossless compression? Jul 3, 2013 at 9:12
1
vote

Virtually this exact same question came up at opendata.SE: What are the most useful formats in which to release geospatial data?

So, hopefully I'm not violating any policies in quoting my own answer there:

My experience, making maps from quite a few government datasets:

For point data, CSV is the best, with "lat" and "lon" columns. Very easy to work with in a wide range of tools, including text editors, spreadsheets, etc. There are two downsides:

  1. GDAL requires a .vrt companion file.
  2. The naming of the lat and lon columns is not totally standard. Many tools are pretty liberal in what they accept.

For lines and polygons, in decreasing order of preference:

  1. GeoJSON. Easy to work with, and the ability to edit in a text editor or with geojson.io is a real bonus, if you need to do search/replace, remove a couple of weird objects or copy and paste from one file to another. Another benefit is that non-GIS developers can make sense of it. Only issues I've run into is when someone provides data as say MultiPoint instead of Point.
  2. Shapefile. Very widely supported, but with two inconvenient points. First, it's a collection of files, so you have to pass around a .zip and extract it. Second, field names are limited to 10 characters. They're hard to edit for your average non-GIS person.
  3. KML/KMZ. These often have a lot of irrelevant cruft (styling, icons, etc), and attributes are sometimes encoded as mini HTML tables, which are really hard to work with. At least you can edit them easily with Google tools.

Honestly, though, the best answer is probably "all of them". Do everyone a favour and release the data in CSV (if point), GeoJSON, zipped Shapefile and KMZ.

Not the answer you're looking for? Browse other questions tagged or ask your own question.