Reading remote LASCatalog

Question

Would it be possible to read in a LASCatalog from a URL or vector of URLs?

It is not possible with the standard function, readLASCatalog(). The benefit would be that you don't need to download all of the LAS files together, but only download them as needed, and discard them once they have been processed. Of coursed this would be much slower than storing them on disk, but would save a lot storage space.

Answer 1

In short there is no such feature in lidR and there is no plan for such addition in a close future. las and laz file are read with LASlib which does not have the capability to stream a remote file. Thus lidR cannot do it. However it does not mean you can't find some partial workarounds.

Download only the header of the files

This is something I already made. Hundreds of big files on a server -> almost impossible to download each of them. I downloaded only the first 5 kB of each file with a bash command to get the header + a partial and corrupted payload. This was enough to build a LAScatalog and download only some files of interest.

for f in ('curl -s -l ftp://url.com/data/laz/') do 
  curl --range 0-5000 ftp://url.com/data/laz/$f --output $f
done

Then it was possible to create a LAScatalog with readLAScatalog(). However it was not possible to process the point cloud because the points were not actually downloaded.

Make your own function readLAScatalogRemotely()

You could wrap the previous command in a R system call to make it workable in R

readLAScatalogRemotely = function(remote_dir) {
  filenames <- RCurl::getURL(remote_dir, verbose = FALSE,ftp.use.epsv = TRUE, dirlistonly = TRUE)
  filenames <- strsplit(filenames, "\n")[[1]]
  for (file in filenames) {
    cmd <- paste0("curl --range 0-5000 ", remote_dir, file, " --output ", tempdir(), "/", file)
    system(cmd)
  }

  return(readLAScatalog(tempdir()))
}

remote_dir <- "ftp://url.com/las/"
ctg <- readLAScatalogRemotely(url)

Again, the payload won't be downloaded. You can't process the LAScatalog.

Make your own function readLASremotely()

You can create a function that take some urls in input, download the files, read the file and delete the files.

readLASremotely = function(url, select = "*", filter = "") {
  n <- length(url)
  files <- character(n)

  for (i in 1:n) {
    file <- tempfile(fileext = ".las")
    files[i] <- file
    download.file(url[i], file, mode="wb")
  }

  las <- readLAS(files, select, filter)
  file.remove(files)
  return(las)
}

Combine everything

It could eventually be possible to use our readLASremotely() with the catalog engine through catalog_apply(). This requires to have a deep understanding of the processing engine. I could write an example but in my opinion it is a very bad idea. This would require to download 8 times the point clouds (see also this vignette).