4

Many field sciences require data management plans for their research data that specify how the data will be submitted for archival storage. These data come from various devices and are often in proprietary formats that require processing. For example, Trimble GPS devices produce SSF format files but these are not suitable for longterm storage because they are proprietary and require a Trimble software license.

If you worked for a library or the US Forest Service what vector format would you consider the most sustainable for people to use when they contribute data to your archive?

While not an open format Shapefile is a defacto standard and widely readable (in practice this is what people are using). To be clear, these are not archives that are served online they're stored, perhaps burned onto archival DVD.

Relevant Links

  1. GDAL - Geospatial Data Abstraction Library
  2. Sustainability of Digital Formats: Planning for Library of Congress Collections
  3. OGC standards working group
  4. FOSS Open Standards/Comparison of File Formats
7
  • 2
    Didn't you just ask then delete this question? Even DVD isn't guaranteed to be suitable for long term storage, there was a time when tapes were considered the only long term storage solution and now I don't think you can even buy a tape drive and DVD media can degrade even stored in optimal conditions. – Michael Stimson Mar 31 '17 at 5:03
  • I notice that you did not edit your earlier question at gis.stackexchange.com/questions/234228/… and instead deleted and re-asked it. The latter is not prevented but the former is preferred. – PolyGeo Mar 31 '17 at 5:04
  • 1
    What type of vector data archive does this library or the US Forest Service maintain? If they use different spatial data storage formats for their archives then the answer may vary, which will make this too broad for focussed Q&A. – PolyGeo Mar 31 '17 at 5:05
  • 1
    The biggest problem I have seen with archives of data is not the format but with the bucket 'o data approach, after only a very short time it's hard to locate any data that you know you have or to understand the data you do find.. Index your data with a table (spreadsheet or database) and keep/populate metadata. Quite often multiple versions of the same data are archived but are slightly different, understanding which version is the correct one comes down to metadata - if it's not there you have a mess! – Michael Stimson Mar 31 '17 at 5:13
  • When improvement to a question is requested via comments the way to do that is by using the edit button to revise it. – PolyGeo Mar 31 '17 at 5:13
4

In my opinion, Shapefile is a good choice, but it has many limitations for historical reasons. For example, a single shapefile cannot be greater than 2GB, etc. You have to decide whether to commit to those restrictions if you use shapefile as the archival format. There is a good discussion at GIS.SE as to Are there any attempts to replace the shapefile?.

For archival purposes, a good choice may be to use text formats, such as GML, GeoJSON mentioned in the above post, or even Well Known Text (WKT). Text formats are most portable, readable forever, and (so far) without license issues. We can always reconstruct auxiliary files 50 years later from the basic data. To separate concerns, one can choose an efficient/reliable method/format to compress the text files before archiving them.

3

The shapefile could be considered an open format in that its technical description is published and routines for reading and writing the format can be created without requesting permission or paying license fees.

1

Library of Congress (LoC) has a comprehensive and detailed list of (not only GIS) file formats with detailed information, between elso also considerations about long term use. LoC is an authority that can be cited in academic context and it's a great source for advice on any kind of data format.

See what they say e.g. for Geopackage sustainability:

In June 2020, the New South Wales State Archives lists GeoPackage 1.0 in its list of sustainable formats. The compilers of this resource would welcome information about other recommendations for the GeoPackage format from archival entities that collect and preserve geospatial datasets. Comments welcome.

However, for technical reasons, Geopackage is not a format perfectly apt for archiving :

The SQLite DB Geopackage is based on is a complex binary format (=disadvantage for archiving)

Source: Own translation from the german original, available in a presenatation by Markus Jobst for Geopackage Hackathon 05/2019, see slide 6 on the second pdf linked, called Geopackage Hackathon - Einführung

Despite of limitations, Shapefiles still have advantages for long term preservation in an academic context. See what LoC says on Shapefiles:

Among the archival institutions that list the Shapefile format as a preferred or acceptable format are: the U.S. National Archives; the UK Data Service; and the Data Archiving and Networked Services for the Netherlands.


Adding to the previous answers, text formats are also adviced by another academic institution, the ETH (Swiss Federal Institute of Technology) for long term archiving: https://documentation.library.ethz.ch/display/RC/File+formats+for+archiving

Even though they have no special advice for GIS vector formats (TIFF as raster format luckily is included!), the general recommendations give some hints about what to be aware of - see especially the general section about Preparing your files, with tips (some obvious, others not) like:

Avoid password protection, encryption and compression, Avoid special characters, Proper use of file extensions

For advice on GIS formats, see also: https://www.library.ucdavis.edu/guide/geospatial-data-for-storage-exchange/

For completion, this link helps to get an overview over existing GIS file formats and geospatial file extensions: https://gisgeography.com/gis-formats/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.