Clipping LAScatolog with a large polygon shapefile using is extremely slow

Question

I´m using the lidR (Version 4.0.0) package to clip a LAScatalog (with 119 .las files, ~52.9 Gb) using a very large polygon shapefile (~865K buffers with 25m diameter each). I´m stuck in this task because when I run the process without parallelization, the function estimates ~27 days to finish the process. I already tried to parallelize, using plan(multissesion) from future package, but the estimated time to finish the task is even much higher than without parallelization. What I've noticed, and what's intriguing me, is that as time goes by, the time taken to produce a las file clipped per polygon increases. For example, when I start the process, I can produce 244 clipped las files per minute, but after one hour this number decreases to ~120/minute, and after 24 hours to 12/minute. I was wondering if there is a more efficient way to accomplish this task.

Here is my R session and code:

#> R version 4.1.3 (2022-03-10) -- "One Push-Up"
#> Copyright (C) 2022 The R Foundation for Statistical Computing
#> Platform: x86_64-w64-mingw32/x64 (64-bit)

# i) create a lascatalog;
catalog_las<- readLAScatalog(outdir)
opt_independent_files(catalog_las)<- TRUE
opt_merge(catalog_las) = FALSE

# ii) Indexation of the points with lax files
lidR:::catalog_laxindex(catalog_las)

# iii) multiple extraction on disk to write the resulted clipped las files to disk 

shp_path="C:/SIG_project/tests"# path to the polygon shapefile
polygons <- sf::st_read(paste0(shp_path, "//buffers.shp"),quiet = TRUE)
opt_output_files(catalog_las) <- paste0(getwd(), "_{rand_point}") # to designate the path where one las file per "rand_point" ID will be written
clip_roi(catalog_las, polygons) # to clip the las files at the buffers boundaries

Answer 1

Your code is correct. You files are indexed. You can't do much. You are trying to extract 865.000 polygons if I understand well. Your request is gigantic so you must accept that it is going to take time!

You said 25 m diameter. So I'm assuming you are trying to extract discs. In this case use a shapefile of plot centers and use clip_circle which gonna be much more efficient. This is the most obvious potential improvement I can see.

If you really want polygons you may try to simplify the polygons.

Last but not least, decompress all the files in LAS, once for all, if they are LAZ. This is likely to give you another 2 fold speed-up.

I already tried to parallelize, using plan(multissesion) from future package, but the estimated time to finish the task is even much higher than without parallelization

Parallelization is not a magic sorcery that make everything faster. There are overhead costs. Here the cost is higher than the computation itself that is fast.

What I've noticed, and what's intriguing me, is that as time goes by, the time taken to produce a las file clipped per polygon increases. For example, when I start the process, I can produce 244 clipped las files per minute, but after one hour this number decreases to ~120/minute, and after 24 hours to 12/minute.

I've noticed the same. Not with lidR but with bare bone code using future and future.apply and very long computations. This is not related to lidR directly but I hear what you are saying. Currently it seems early to investigate more in depth as I'm not sure if the problem is real or not. But with your testimony I may open an issue on the future repos to discuss about that with the developer. For the moment it seems this happens on windows not linux. Not sure.