Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

What are the pros and cons of different data formats (performance, file size, etc.) when considering open data distribution?

Our organisation wants to publish data as open data. However, there is no clear idea on which data formats to use. Ofcourse, the more 'open' a data format is, the easier it is to use.

Which data formats are the most 'open' and therefore most usable for the distribution of Open Data when taking the following types in consideration?:

  • raster data (I'm thinking: GeoTIFF, Erdas Imagine IMG?)
  • vector data (I'm thinking: GML, CSV, ESRI Shapefile, DXF?)
  • tabular data (I'm thinking: CSV?)
  • 3D data (I'm thinking: CityGML?)
  • 3D point coulds / LIDAR (I'm thinking: LAS?)
  • am I forgetting something here?

Also, if there is documentation about open data formats I'm very interested if you would like to share.

Cheers!

Mark

share|improve this question
2  
for vector, you might also consider geojson and kml –  neuhausr Jul 1 '13 at 13:41
1  
did you see this link? gis.stackexchange.com/questions/61744/… –  Dan Patterson Jul 1 '13 at 13:52
4  
You need to differentiate between data exchange formats and data storage format. geojson for example is an excellent Data Exchange format, but sucks as a Data storage format. I am assuming you are only concerned with the format for distributing data (i.e data exchange). Is that correct? –  Devdatta Tengshe Jul 1 '13 at 14:26
    
@DevdattaTengshe: Good point! For now, the intention is to distribute the data in the most convenient exchange format. –  Mark Verschuur Jul 1 '13 at 14:52
    
Thanks everyone for the feedback. Also, some tips about file types to use vs file size would be very helpfull. –  Mark Verschuur Jul 2 '13 at 10:01

4 Answers 4

The city of Vienna's open data initiative (http://data.wien.gv.at) uses Geoserver to provide access to raster and vector geodata via Geoserver WMS and WFS services. This has many advantages: Users can download data in different formats for offline use (e.g. geojson, KML, or zipped Shapefiles) or use the services live by embedding them in online maps or GIS projects.

share|improve this answer
    
This is what we do here: maps.gcc.tas.gov.au/data There is no reason why you can't use GeoServer for non-spatial tables too. –  alexgleith Jul 3 '13 at 2:30

For tabular csv. Excel is at best overly complicated and at worst totally inaccessible. Access is not accessible and PDF is a slap in the face.

For geospatial use geojson, it's text it's well supported and doesn't have the technical restrictions that the only other viable format (shapefile) has. Also unless you have a very good reason it should be in WGS84, bearing in mind that most users will be in another state and will not want state plane.

share|improve this answer

I would say:

  • Shapefiles or GML for vector data
  • .obj-Files for 3D models
  • .xyz (simple CSV) for point clouds
  • CSV for tabular data
  • GeoTIFF for raster data

These formats are easily readable by Open Source Software and are easily transformable to any other format needed for specific applications.

Also +1 for making data open!

share|improve this answer
2  
I'll be interested to know why you have suggested Shapefiles and GML for vector data. Both of them are terrible formats. The only saving grace of GML, is that it is an OGC format. –  Devdatta Tengshe Jul 1 '13 at 15:37
1  
Shapefiles are readable in many applications, and can be transformed to something different without problems. What would you suggest? –  til_b Jul 2 '13 at 6:59
2  
avoid shapefiles. They work, but they have serious technical limitations. –  nickves Jul 2 '13 at 9:33
1  
So what do you suggest that does not have the technical limitations of shapefiles? –  til_b Jul 2 '13 at 9:55
2  
@til_b GeoTIFF is a nice format from a perspective of being 'open'. However, for storage (or offering it as a download) it is terrible, because the files can get huge. Do you know of an open raster format that offers lossless compression? –  Mark Verschuur Jul 3 '13 at 9:12

I quite like NetCDF for continuous/array data (i.e. rasters). Pros for NetCDF are:

  • NetCDF is self describing (i.e., data definitions are available through the file header) so you don't need to supply secondary metadata files
  • NetCDF4 allows for storage of n-dimensional data (using the HDF5 data format on disk, which is a bonus as this allows files as big as your OS can handle). This comes with reasonable compression and fast access to the data. Note that NetCDF3 doesn't support n-dimensional data, and has a file size limit of roughly 2GB on a 32-bit system.
  • NetCDF is an open format so accessing the data is generally not a problem as well through common libraries. For example, in python it's simple enough from scipy to read in a slice of data:
from scipy.io import netcdf
f = netcdf.netcdf_file('source.nc')
print(nc.dimensions) #take a look at the dimensions of the data
print(nc.variables)  #A dictionary containing all the variables
nc.variables["some_data"].dimensions #The dimensions this variable is in, e.g. lat, lon
out_array = nc.variables["some_data"].data
f.close() #and we're done

The only downside to NetCDF4 that I can see is the not-great support in standard GIS packages like ArcGIS and QGIS (though I dearly would love to be corrected on this!).

EDIT Some other packages that support NetCDF

Some standard programming languages that support NetCDF (though to be fair, anything that can read HDF can read NetCDF4):

For maths and stats users you have:

Specifically in GIS:

  • GDAL will convert the data for you
  • Likewise FME
  • ArcGIS supports NetCDF (though it's not the best level of support in my experience)
  • There is a QGIS Plugin in development

If you want to quickly look at a NetCDF file I'd use the cross-platform Panoply from NASA. And if you're interested in more, UCAR Unidata has a list of software.

share|improve this answer
    
NetCDF is an awful choice it really has no support outside of python. It might have good support, but tiffs, png, and jpeg have support in literally every language. –  Calvin Jul 4 '13 at 2:37
2  
I strongly disagree. I've edited my response above to show a quick list of packages that support NetCDF. In my experience it's a format of choice for any multidimensional scientific data (e.g. astronomy and meteorology). PNG and TIFF aren't bad for distributing raster data, and certainly viewing the data is easy, but they don't scale well to large amounts of multidimensional data. Don't ever use JPEG to distribute scientific data (though if you're sending someone a map it works perfectly well). –  om_henners Jul 5 '13 at 0:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.