3

This question already has an answer here:

I have a few large LiDAR DEMs that I want to merge. Each DEM has some glitches around the borders which need to be removed in order to get a seamless final product. I manually traced around each DEM to create a polygon which excludes the glitchy areas. I am using gdalwarp to clip the raster files, but I am experiencing very slow performance.

The rasters are between 3 and 10 GB, which I understand is pretty big… however I have ~3 ghz of processing power & 4GB of RAM. These rasters load into QGIS quickly and are quick to refresh. I can derive a hillshade or slope raster from one of these DEMS in 2 or 3 minutes.. so it seems unreasonable when it takes 4-6 HOURS to clip one of these rasters with gdalwarp.

Can anybody offer an explanation of what might be going on?

I am also bothered by the very large file sizes (up to 4x larger than the original) which result from the clipping. Research suggests that this is a compression issue, but gdalinfo does not return any compression info for any of the original rasters, and even when I compress the files are still 2x larger than the originals. Could there be any reason for this other than compression?

marked as duplicate by matt wilkie, Vince, Jochen Schwarze, LaughU, KHibma Nov 16 '18 at 15:20

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • A test image and exact GDAL commands would help a lot. – user30184 Apr 16 '16 at 17:06
  • Things to consider are the filetype and compression of your original files, your chosen compression for your output raster and options for the hillshade stuff you did. That the files are refreshing quickly is probably due to pyramids. One general error is when your clipping polygon is in some parts larger than your original raster it takes really long. Also the functions might work probably a different way. Clip, as far a i understand it, puts a lot into RAM. And your files are larger then your (pretty small) Memory. And also after clipping the raster has to be interpolated due to pixel-shift. – Matte Apr 16 '16 at 17:08
  • The command I have been using is: gdalwarp -dstnodata 0 -q -cutline "clip.shp" -crop_to_cutline -dstalpha -tr 5.0 5.0 -of GTiff "county_dem.tif" "county_dem_clip.tif" DEMs can be found here, I am using Vernon, Crawford, Juneau, Sauk, Lacrosse, Monroe and Richland counties. – corvus Apr 16 '16 at 17:10
  • @Matte: Thanks for the response. I am sure that the functions which derive a hillshade or slope are much different than the gdalwarp function.. I was not sure if this could be a normal penomenon, but I thought it seemed unreasonable because of the huge difference in the time it takes to process an equivalent amount of data. "One general error is when your clipping polygon is in some parts larger than your original raster it takes really long" This could be the reason. I'll check back – corvus Apr 16 '16 at 17:12
  • 2
    Shapefile is not converted into raster, it is used as a mask and that is not so expensive as far as I know. – user30184 Apr 16 '16 at 19:02