Seeking open vector format for long term GIS data storage in off-line repositories?

Question

Many field sciences require data management plans for their research data that specify how the data will be submitted for archival storage. These data come from various devices and are often in proprietary formats that require processing. For example, Trimble GPS devices produce SSF format files but these are not suitable for longterm storage because they are proprietary and require a Trimble software license.

If you worked for a library or the US Forest Service what vector format would you consider the most sustainable for people to use when they contribute data to your archive?

While not an open format Shapefile is a defacto standard and widely readable (in practice this is what people are using). To be clear, these are not archives that are served online they're stored, perhaps burned onto archival DVD.

Relevant Links

1. GDAL - Geospatial Data Abstraction Library
2. Sustainability of Digital Formats: Planning for Library of Congress Collections
3. OGC standards working group
4. FOSS Open Standards/Comparison of File Formats

Answer 1

In my opinion, Shapefile is a good choice, but it has many limitations for historical reasons. For example, a single shapefile cannot be greater than 2GB, etc. You have to decide whether to commit to those restrictions if you use shapefile as the archival format. There is a good discussion at GIS.SE as to Are there any attempts to replace the shapefile?.

For archival purposes, a good choice may be to use text formats, such as GML, GeoJSON mentioned in the above post, or even Well Known Text (WKT). Text formats are most portable, readable forever, and (so far) without license issues. We can always reconstruct auxiliary files 50 years later from the basic data. To separate concerns, one can choose an efficient/reliable method/format to compress the text files before archiving them.

Answer 2

The shapefile could be considered an open format in that its technical description is published and routines for reading and writing the format can be created without requesting permission or paying license fees.

Answer 3

Library of Congress (LoC) has a comprehensive and detailed list of (not only GIS) file formats with detailed information, between elso also considerations about long term use. LoC is an authority that can be cited in academic context and it's a great source for advice on any kind of data format.

See what they say e.g. for Geopackage sustainability:

In June 2020, the New South Wales State Archives lists GeoPackage 1.0 in its list of sustainable formats. The compilers of this resource would welcome information about other recommendations for the GeoPackage format from archival entities that collect and preserve geospatial datasets. Comments welcome.

However, for technical reasons, Geopackage is not a format perfectly apt for archiving :

The SQLite DB Geopackage is based on is a complex binary format (=disadvantage for archiving)

Source: Own translation from the german original, available in a presenatation by Markus Jobst for Geopackage Hackathon 05/2019, see slide 6 on the second pdf linked, called Geopackage Hackathon - Einführung

Despite of limitations, Shapefiles still have advantages for long term preservation in an academic context. See what LoC says on Shapefiles:

Among the archival institutions that list the Shapefile format as a preferred or acceptable format are: the U.S. National Archives; the UK Data Service; and the Data Archiving and Networked Services for the Netherlands.

---

Adding to the previous answers, text formats are also adviced by another academic institution, the ETH (Swiss Federal Institute of Technology) for long term archiving: https://documentation.library.ethz.ch/display/RC/File+formats+for+archiving

Even though they have no special advice for GIS vector formats (TIFF as raster format luckily is included!), the general recommendations give some hints about what to be aware of - see especially the general section about Preparing your files, with tips (some obvious, others not) like:

Avoid password protection, encryption and compression, Avoid special characters, Proper use of file extensions

For advice on GIS formats, see also: https://www.library.ucdavis.edu/guide/geospatial-data-for-storage-exchange/

For completion, this link helps to get an overview over existing GIS file formats and geospatial file extensions: https://gisgeography.com/gis-formats/