1

I'm assembling a moderately large spatial dataset as simple features in R (4.0.5).

Is there a good way to select the most performance-efficient output format/OGR driver to save my assembled data?

I'm interested in a general approach, but here are some particulars of my current case:

  • The data consists of multiple vector types plus non-spatial columns.
  • The spatial inputs include ESRI shapefiles, ESRI file geodatabases, and GeoJSON.
  • My purpose in writing the data is to read back into R for analysis.
  • My constraints are processing power and working memory, not disk space.

I'm looking for a general algorithm for selecting among the many available OGR formats.

Improve this question

1 Answer 1

Reset to default
3

Performance might be very dependent on data structure and size, complexity of geometry vs attributes etc. A very multi-dimensional problem space--consequently, no straightforward, general approach.

In your use case: GeoPackage. Can store several sets of spatial features in one file, can also store non-spatial tables as well, few constraints on column name length, well-supported and an open specification. Can use SQL to select subsets at load time. All good stuff.

But there's one other option if all you care about is getting this data back into R, and that's to save each set of features as an .RDS file using saveRDS. It's not OGR so you can't load it into anything else, but it would probably be the most efficient R -> files -> R only round-trip solution.

Note that most of the OGR formats are either limited in functionality and/or derived from vendor-specific formats. I would only use GeoPackages for saving spatial data to disk now, unless I had an end-user with a desperate and unfixable need for something else, such as a shapefile, or a web-oriented application that needed a GeoJSON file (which is practically limited to WGS84 coordinates) or I needed to load it into a GPS receiver in which case I'd probably have to use GPX (but that only stores points and tracks, not polygons).

Improve this answer
4
  • This is helpful guidance, thanks. I assume it comes from your accumulated expertise. Is there high-level comparative documentation anywhere? The best I've found is the GDAL vector drivers documentation, but it doesn't reveal anything about performance.
    – max norton
    Commented Apr 1, 2021 at 20:08
  • Performance might be very dependent on data structure and size, complexity of geometry vs attributes etc. A very multi-dimensional problem space. Would take some thinking through.
    – Spacedman
    Commented Apr 1, 2021 at 20:13
  • Added some comments on other OGR format drivers.
    – Spacedman
    Commented Apr 1, 2021 at 20:18
  • Thanks again for this answer. I am finding it useful to use RDS for saving intermediate files, and GeoPackage for sharable processed data. I just submitted an edit adding your comment "Performance... a very multi-dimensional problem space" because I think it provides the most direct answer to my question. I'll accept this answer once we incorporate that point.
    – max norton
    Commented Apr 13, 2021 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.