7

I am trying to read in a large shapefile using readOGR. The shapefile is 68837 features and 14 fields. It takes a considerably long time to read at >30min. I understand this is a well documented issue with readOGR, but have not been able to find an adequate, updated solution.

How are folks dealing with reading large spatial datasets in R? Are there newer packages that are more efficient at reading large datasets?

  • 3
    Have you tried using the sf package? r-spatial.github.io/sf/articles/sf1.html#reading-and-writing – Jot eN Apr 13 '18 at 15:27
  • 1
    Generally speaking, 68k rows is not what most would deem "large", even if each feature had an average of 31k vertices (reaching the 2Gb .shp file limit). Please be sure to include the physical sizes of the files involved or the average vertex count and dBase file record width, so that apples-to-apples comparisons can be provided. – Vince Apr 14 '18 at 0:20
  • 1
    Even when you have read it, what are you going to do with it? Have you got the power and the RAM to work with it? Should you convert it to a spatial database (PostGIS, GeoPackage) so you can work on parts of it? – Spacedman Apr 14 '18 at 6:43
15

The sf package does read much, much more quickly. Using the Natural Earth roads layer, with ~ 60k features:

require(rgdal)
require(sf)

dsn <- "./Natural_Earth_v2.0.0/10m_cultural/10m_cultural"
lay <- "ne_10m_roads"

system.time(test <- readOGR(dsn = dsn, layer = lay))

#OGR data source with driver: ESRI Shapefile 
#Source: "./Natural_Earth_v2.0.0/10m_cultural/10m_cultural", layer: "ne_10m_roads"
#with 56902 features
#It has 29 fields
#Integer64 fields read as strings:  scalerank question 
#   user  system elapsed 
# 39.085   1.315  41.624

system.time(test <- read_sf(dsn = dsn, layer = lay))

#   user  system elapsed 
#  2.134   0.127   2.312 

18 x faster ...

  • Is it still faster if you then have to convert to Spatial in order to interact with the raster package? – see24 Aug 23 '18 at 18:41
  • 1
    You can use sf objects just like a Spatial* object to interact with the raster package! – Simbamangu Aug 24 '18 at 5:26
  • Amazing! This should be updated in the raster documentation. For example, I was looking at rasterize and it only shows methods for Spatial* objects not sf. But if you look at methods(rasterize) you see there is a method for sf as well – see24 Aug 24 '18 at 12:07
0

Did you try the package shapefiles ?

I'm not sure how fast will it be with large data, but it is doing good for me.

  • Tried it with test <- read.shapefile("./Natural_Earth_v2.0.0/10m_cultural/10m_cultural/ne_10m_roads") and got error: Error in seq.default(1, (num.points * 2), by = 2) : wrong sign in 'by' argument – Simbamangu Apr 17 '18 at 7:04
  • @Simbamangu can you share the data ? – anup Apr 17 '18 at 16:25
  • Natural Earth – Simbamangu Apr 17 '18 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.