Efficient preallocation of simple features in R (sf r)

Question

I have a use case that requires looping over a large number of API calls to populate a simple features dataset. Read/write is very slow when I preallocate a large empty collection initially, which goes against what I though the whole point of preallocating was for.

Simple features are likely different from a memory standpoint, as ex ante one doesn't know if the geometry for each feature will be "small" or "large". Still, it would be great if there was a way to speed this up.

Minimal reprex:

rm(list = ls())
library(sf)
library(tidyverse)

## Initialize size to 100 rows, populate with 100 features ##
start_time100 <- Sys.time()
rsize <- 100

route <- st_sf(id = 1:rsize, geometry = st_sfc(lapply(1:rsize, function(x) st_linestring() )), crs = 4326)

for (i in 1:100) {
  sln <- rbind(c(runif(1),runif(1)), c(runif(1),runif(1)), c(runif(1),runif(1)))
  route_temp <- st_linestring(sln, dim = "XY") %>% 
    st_sfc(crs = 4326)
  (route[i,] <- st_sf(id = i, geometry = route_temp))
}

route <- route[!st_is_empty(route),]
end_time100 <- Sys.time()

## Initialize size to 10000 rows, populate with 100 features ##
start_time10k <- Sys.time()
rsize <- 10000

route <- st_sf(id = 1:rsize, geometry = st_sfc(lapply(1:rsize, function(x) st_linestring() )), crs = 4326)

for (i in 1:100) {
  sln <- rbind(c(runif(1),runif(1)), c(runif(1),runif(1)), c(runif(1),runif(1)))
  
  route_temp <- st_linestring(sln, dim = "XY") %>% 
    st_sfc(crs = 4326)
  (route[i,] <- st_sf(id = i, geometry = route_temp))
}

route <- route[!st_is_empty(route),]
end_time10k <- Sys.time()

end_time100 - start_time100
end_time10k - start_time10k

Running on my machine gives:

> end_time100 - start_time100
Time difference of 0.2343311 secs
> end_time10k - start_time10k
Time difference of 9.813453 secs

Modifying the reprex based on the suggestion from @mdsumner is much faster (prob fast enough for my use case), but still sees substantial slowdown for editing within the "larger" object. Replacing the main portion of the code:

route_list <- vector("list", rsize) 
route_id <- vector("numeric", rsize) 

for (i in 1:100) {
  sln <- rbind(c(runif(1),runif(1)), c(runif(1),runif(1)), c(runif(1),runif(1)))
  route_list[[i]] <- st_linestring(sln, dim = "XY")
  route_id[i] <- i
}

route_list <- route_list %>% 
  st_sfc(crs = 4326)
route <- st_sf(id = route_id, geometry = route_list)

Gives run times as below (and I double checked that it's not the route_id component). sfheaders + template sounds useful, though I haven't tried that yet.

> end_time100 - start_time100
Time difference of 0.01904988 secs
> end_time10k - start_time10k
Time difference of 0.09023905 secs

Answer 1

Without having tried your code, what I see is:

Don't do

route[i ,] <- ... 

because in general modifying a data frame in place can be slow.

Do create a list upfront to hold each geometry, i.e.

vector("list", rsize) 

and put each st_linestring() in that, wrap the entire list after the loop in st_sfc(), and do the st_sf() at the end with the sfc vector.

More advanced (and off the beaten path):

If you have a data frame with all the coordinates and line identifiers, see the sfheaders package - it's not mature/complete yet but usually much faster than the sf constructors (and completion is coming, with the in-dev geometries package.

If it's still too slow you can construct just one st_linestring() and update the coordinate matrix inside it, make a copy for each step. I discuss this here: https://github.com/dcooley/sfheaders/issues/62#issuecomment-599037957 see how mk_direct() creates a template and updates that in-place rather than construct again a new st_linestring.