4

I'm trying to read the belgian landuse ESRI shapefile from geofabrik.de:

http://download.geofabrik.de/europe/belgium.html

In QGIS it's just a matter of seconds, but in R using the readOGR function my RAM (4gbs) fills and the reading process does not actually end even after several minutes.

I on a lenovo i5-4300 intel quadcore laptop with windows 7 on it and I use R studio Version 0.99.447 and R x64 3.2.1.

The code is the following:

shp.landuse <- readOGR(layer="landuse")

Is there a better way?

6
  • 2
    4Gb of RAM isn't much -- Windows 7 uses 3Gb before I get to the desktop. Windows and Linux both cache file I/O when RAM isn't used for anything else, so the behavior you're describing isn't a major concern. Please edit the question to provide more details on your operating system, the actual file sizes of the shapefile triple, and the exact timing involved as R imports the file into something it can use (which involves reading the .dbf, which drawing in QGIS doesn't)
    – Vince
    Jul 17, 2015 at 11:00
  • QGIS is dedicated GIS software, I would expect it to handle loading shape files better than R. However, as Vince says: you need to give more information including the R script you're using to load your data. Jul 17, 2015 at 11:15
  • I was reading lately that R holds all objects in virtual memory which has limitations. So maybe a better way would be to use something that handles memory better - This is just an opinion as I am just learning R
    – TsvGis
    Jul 20, 2015 at 6:52
  • Is creating a database an option? if so how to do it? Jul 20, 2015 at 7:45
  • What are you trying to do with this data? What are the analysis or display steps you are looking to do?
    – Simbamangu
    Jul 20, 2015 at 9:09

1 Answer 1

3

There's a better library for reading really large shapefiles - fastshp. Doesn't seem to be available in repositories but the .tgz binaries are here. Here are the results for rgdal and fastshp with a 130MB shapefile with 32,545 features:

library(fastshp)
library(rgdal)

system.time(
  test.fastshp <- read.shp("tz-landcover-ge.shp")
)

# user  system elapsed 
# 1.570   0.135   1.732 

system.time(
  test.rgdal <- readOGR(layer="tz-landcover-ge")
)
# OGR data source with driver: ESRI Shapefile 
# Source: "/Users/simbamangu/Downloads/tz-landcover-ge/", layer: "tz-landcover-ge"
# with 32545 features
# It has 9 fields
# user  system elapsed 
# 11.216   0.799  12.063 

7 times faster - would have been 178 times faster (0.063 sec) if I hadn't asked it to use format="polygons".

However, it provides a different class of object - a 'shp' instead of 'SpatialPolygonsDataFrame', which is a list of other objects (lines / polygon definitions depending on 'format' - doesn't work with points).

3
  • Hello, is it possible to merge different shapefiles in that format? Also is it possible to plot and subset these shapefiles? Jul 20, 2015 at 8:30
  • I never really understood the point of the fastshp package. So, it is fast. It does not produce a standard spatial object class so, you still have the overhead of coercing to a class you can work with. This is very likely why it is depreciated. I would also note that it is not good practice to aim novice users to depreciated packages. Because of changes to both R and library dependencies, it often ends up creating more headaches for the user and could yield outright incorrect results. Aug 15, 2016 at 20:28
  • Agreed. As per OP's final comment, fastshp might provide a way to 'filter' a large set and then coerce or work with a subset (unclear what exactly they wanted to do). I personally now would usually convert to PostGIS / Spatialite and readOGR from the db instead of a shapefile. Any suggestions from your side, perhaps a better answer? ;)
    – Simbamangu
    Aug 16, 2016 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.