r Terra Mosaic Raster uses huge amounts of memory and takes too long - any alternatives?

Question

I built a process using google earth engine code editor to output a raster of urban land cover in Wisconsin. Google exports these as 210 shard which are geotiffs ranging from 2-100mb in size. Most are around 15mb or so. So I would like to mosaic these into one geotiff. The gist of my code is:

raster_files <- list.files("E:file_location", pattern = ".tif$", full.names = TRUE)

rasters <- lapply(raster_files, terra::rast)

raster_sprc <- terra::sprc(rasters)

raster_mosaic<- terra::mosaic(raster_sprc, fun = "max") 

This threw "Error: [mosaic] insufficient disk space (perhaps from temporary files?)" (Note: there is no overlap between tiles, so I really don't care about the function)

So I subsetted it to the first 5 and gave it a whirl.

raster_files <- list.files("E:file_location", pattern = ".tif$", full.names = TRUE)

rasters <- lapply(raster_files, terra::rast)

raster_sprc <- terra::sprc(rasters[1:5])

raster_mosaic<- terra::mosaic(raster_sprc, fun = "max") 

This worked well, but:

1. It created 25 1gb and 1 20gb geotiffs in the terra temp folder. I am guessing the 25 is each combination of the 5 subsetted rasters (5x5).
2. It took about 25 minutes to run (my machine is pretty solid so not a processor/memory issue)
3. The raster that I wrote from the r object was only 100mb

I am not understanding how a seemingly straightforward operation to create a relatively small output takes so long and uses so much hard drive space. It seems like my only option is to loop my operation writing the output and clearing the temp folder (terra::tmpFiles(remove = TRUE)) on each iteration. Then mosaicking together the mosaics. Guessing when all is done, it would take several days to run this with all sorts of ways it could go wrong along the way.

Is there a better way to do this?

I looked but cannot find any way to export a mosaicked geotiff from Google Earth Engine. I tried this on ArcGIS Pro and it ran for 4 days until it threw some unknown error. I tried terra::expand on each raster to get them to the same extent so I could stack them and summarize the stack, but this took too long for a single layer.

Most of the rasters exported by Google Earth Engine are 32768x32768x1 with 1 meter resolution. Some of them at the edges are smaller like 32768x6867x1. The dimensions of the polygon I used to select the region in GEE were about 438000x485000.

Answer 1

Spacedma's VRT solution did the trick. Both of these approaches worked the same.

terra::vrt(raster_files, "E:/raster_file_output") 

or

gdalUtils::gdalbuildvrt(gdalfile = raster_files,
                        output.vrt = "E:/raster_file_output")

I then had problems translating the virtual raster to a real raster using R, so I ended up just using QGIS to convert the VRT to raster. Took a few hours to run but it worked.