5

Would it be possible to read in a LASCatalog from a URL or vector of URLs?

It is not possible with the standard function, readLASCatalog(). The benefit would be that you don't need to download all of the LAS files together, but only download them as needed, and discard them once they have been processed. Of coursed this would be much slower than storing them on disk, but would save a lot storage space.

1 Answer 1

3

In short there is no such feature in lidR and there is no plan for such addition in a close future. las and laz file are read with LASlib which does not have the capability to stream a remote file. Thus lidR cannot do it. However it does not mean you can't find some partial workarounds.

Download only the header of the files

This is something I already made. Hundreds of big files on a server -> almost impossible to download each of them. I downloaded only the first 5 kB of each file with a bash command to get the header + a partial and corrupted payload. This was enough to build a LAScatalog and download only some files of interest.

for f in ('curl -s -l ftp://url.com/data/laz/') do 
  curl --range 0-5000 ftp://url.com/data/laz/$f --output $f
done

Then it was possible to create a LAScatalog with readLAScatalog(). However it was not possible to process the point cloud because the points were not actually downloaded.

Make your own function readLAScatalogRemotely()

You could wrap the previous command in a R system call to make it workable in R

readLAScatalogRemotely = function(remote_dir) {
  filenames <- RCurl::getURL(remote_dir, verbose = FALSE,ftp.use.epsv = TRUE, dirlistonly = TRUE)
  filenames <- strsplit(filenames, "\n")[[1]]
  for (file in filenames) {
    cmd <- paste0("curl --range 0-5000 ", remote_dir, file, " --output ", tempdir(), "/", file)
    system(cmd)
  }

  return(readLAScatalog(tempdir()))
}

remote_dir <- "ftp://url.com/las/"
ctg <- readLAScatalogRemotely(url)

Again, the payload won't be downloaded. You can't process the LAScatalog.

Make your own function readLASremotely()

You can create a function that take some urls in input, download the files, read the file and delete the files.

readLASremotely = function(url, select = "*", filter = "") {
  n <- length(url)
  files <- character(n)

  for (i in 1:n) {
    file <- tempfile(fileext = ".las")
    files[i] <- file
    download.file(url[i], file, mode="wb")
  }

  las <- readLAS(files, select, filter)
  file.remove(files)
  return(las)
}

Combine everything

It could eventually be possible to use our readLASremotely() with the catalog engine through catalog_apply(). This requires to have a deep understanding of the processing engine. I could write an example but in my opinion it is a very bad idea. This would require to download 8 times the point clouds (see also this vignette).

2
  • Thank you for your tips and feedback. Are you saying that we would need to download 8 times the data because of the 8 neighboring tiles needed to create a buffer? As the vignette points out, indexation by .lax files helps to remediate this, so readLASremotely() or catalog_apply() (I don't have a strong enough understanding of the processing engine) would need to make use of the available .lax files. Are you suggesting it would not be possible to incorporate remote .lax files? This would certainly add more complexity.
    – Lucas
    Oct 16, 2019 at 22:12
  • 1
    You can't read remotely. You need to download the files first. So lax files do not matter here. You need to download the files anyway. If you download lax files as well you can speed up the reading of the buffer. But anyway you have downloaded the neighboring tiles. In my example you download/read/remove so every time you need to download again because you removed the files. Cannot explain more in a comment sorry.
    – JRR
    Oct 16, 2019 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.