1

I have a use case that requires looping over a large number of API calls to populate a simple features dataset. Read/write is very slow when I preallocate a large empty collection initially, which goes against what I though the whole point of preallocating was for.

Simple features are likely different from a memory standpoint, as ex ante one doesn't know if the geometry for each feature will be "small" or "large". Still, it would be great if there was a way to speed this up.

Minimal reprex:

rm(list = ls())
library(sf)
library(tidyverse)

## Initialize size to 100 rows, populate with 100 features ##
start_time100 <- Sys.time()
rsize <- 100

route <- st_sf(id = 1:rsize, geometry = st_sfc(lapply(1:rsize, function(x) st_linestring() )), crs = 4326)

for (i in 1:100) {
  sln <- rbind(c(runif(1),runif(1)), c(runif(1),runif(1)), c(runif(1),runif(1)))
  route_temp <- st_linestring(sln, dim = "XY") %>% 
    st_sfc(crs = 4326)
  (route[i,] <- st_sf(id = i, geometry = route_temp))
}

route <- route[!st_is_empty(route),]
end_time100 <- Sys.time()

## Initialize size to 10000 rows, populate with 100 features ##
start_time10k <- Sys.time()
rsize <- 10000

route <- st_sf(id = 1:rsize, geometry = st_sfc(lapply(1:rsize, function(x) st_linestring() )), crs = 4326)

for (i in 1:100) {
  sln <- rbind(c(runif(1),runif(1)), c(runif(1),runif(1)), c(runif(1),runif(1)))
  
  route_temp <- st_linestring(sln, dim = "XY") %>% 
    st_sfc(crs = 4326)
  (route[i,] <- st_sf(id = i, geometry = route_temp))
}

route <- route[!st_is_empty(route),]
end_time10k <- Sys.time()

end_time100 - start_time100
end_time10k - start_time10k

Running on my machine gives:

> end_time100 - start_time100
Time difference of 0.2343311 secs
> end_time10k - start_time10k
Time difference of 9.813453 secs

Modifying the reprex based on the suggestion from @mdsumner is much faster (prob fast enough for my use case), but still sees substantial slowdown for editing within the "larger" object. Replacing the main portion of the code:

route_list <- vector("list", rsize) 
route_id <- vector("numeric", rsize) 

for (i in 1:100) {
  sln <- rbind(c(runif(1),runif(1)), c(runif(1),runif(1)), c(runif(1),runif(1)))
  route_list[[i]] <- st_linestring(sln, dim = "XY")
  route_id[i] <- i
}

route_list <- route_list %>% 
  st_sfc(crs = 4326)
route <- st_sf(id = route_id, geometry = route_list)

Gives run times as below (and I double checked that it's not the route_id component). sfheaders + template sounds useful, though I haven't tried that yet.

> end_time100 - start_time100
Time difference of 0.01904988 secs
> end_time10k - start_time10k
Time difference of 0.09023905 secs
  • I also posted this on the simple features github: github.com/r-spatial/sf/issues/1451 – C Sev Jul 15 at 21:55
  • You are appending geometries. This is not the same as vectorization. – Jeffrey Evans Jul 15 at 23:14
  • Sure, but preallocation should still speed this up, right? Would vectorize if I could... – C Sev Jul 15 at 23:32
  • 1
    yes, don't do route[i ,] <- ... do a raw vector("list", rsize) and put each st_linestring in that, wrap the entire list after the loop in st_sfc(), and do the st_sf() at the end with the sfc vector - if it's still too slow you can construct just one st_linestring() and update the coordinate matrix inside it, make a copy for each step – mdsumner Jul 15 at 23:32
  • 1
    if you have a data frame with all the stuff, see the sfheaders package - it's not mature yet but usually much faster than the sf constructors (and completion is coming, with the in-dev geometries package) – mdsumner Jul 15 at 23:33
4

Without having tried your code, what I see is:

Don't do

route[i ,] <- ... 

because in general modifying a data frame in place can be slow.

Do create a list upfront to hold each geometry, i.e.

vector("list", rsize) 

and put each st_linestring() in that, wrap the entire list after the loop in st_sfc(), and do the st_sf() at the end with the sfc vector.

More advanced (and off the beaten path):

If you have a data frame with all the coordinates and line identifiers, see the sfheaders package - it's not mature/complete yet but usually much faster than the sf constructors (and completion is coming, with the in-dev geometries package.

If it's still too slow you can construct just one st_linestring() and update the coordinate matrix inside it, make a copy for each step. I discuss this here: https://github.com/dcooley/sfheaders/issues/62#issuecomment-599037957 see how mk_direct() creates a template and updates that in-place rather than construct again a new st_linestring.

| improve this answer | |
  • Edited initial post to reflect trying this solution -- list approach is much faster. Can the template + mk_direct() approach work without sfheaders? – C Sev Jul 16 at 15:39
  • I expect so, but note sfheaders has now incorporated this efficiency so there's no benefit to the templating trick if you use it – mdsumner Aug 5 at 2:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.