9

I am trying to clip a large ECW (details below) but the raster file is too big to be processed entirely.

Some details of the ECW below

Driver: ECW/ERDAS Compressed Wavelets (SDK 5.0)

file size: 50gb Size is 450000, 565081 Pixel size: 0.15 0.15 COLORSPACE=RGB COMPRESSION_RATE_TARGET=9 VERSION=2 Number of bands: 4

The area I want to clip is roughly a 1/5 of the original file.

Here are the methods I have tried out with no success:

  1. Used Arcgis to save the ecw to tiff / other formats... (I quickly gave up)

  2. Used Qgis and its clipper tool... the file creation stayed stuck at about 40%.

  3. Used gdal_translate out of OSGeo4W with other options than Qgis. (Tried that thinking that MAYBE freeing some memory not using Qgis would the trick)

  4. Used gdal_retile thinking that I would cut the image into pieces and grab the one I wanted. The command "gdal_retile -ps 10000 10000 -of ecw -tileIndex tile.shp -targetDir input.ecw This crashed even quicker"

Does anyone have an idea?

For info I run windows 7 64bits on an i5-3470 3.2Ghz with 16gb of ram.

5

Other ideas you could try:

  1. gdal_translate with the -srcwin switch
  2. gdalwarp with -cutline and -crop_to_cutline and -wm switches. The last one specifies memory for caching and may get you over the issues you had using clipper in QGIS (as this is essentially the same function)
  3. QGIS raster calculator setting the extent to the area you want (simpler than clipper).
  4. SAGA->Clip grid with polygon - who knows, it might be more memory efficient.
  5. Code a solution using Python and Numpy/SciPy to read just a subset of the raster into memory and save it.

I suspect that doing a very simple crop to extent (either with the raster calculator or -srcwin switch in gdal_translate) will be less memory hungry than cropping with a polygon because you have no geometry checks and conversions. Option 5 should use the least memory as you are only reading in what you need. Have a look at this tutorial if you need a 'how to' (adapt the bit on reading and writing by block).

  • Thanks for the answer! I will try out SAGA and if it doesn't work I will have a go with Numpy. – Blue Oct 16 '13 at 0:25
5

You can cut it directly with gdal's tool gdal_translate if you know the coordinates of your Area of Interest, if its georeferenced:

gdal_translate -projwin [ulx uly lrx lry] infile outfile  

If not use the -srswin flag like this:

gdal_translate -srcwin [xoff yoff xsize ysize] infile outfile.

Another option is to build a 'virtual' raster (of a few kilobytes) that point at your initial dataset, with gdalbuildvrt.

gdalbuildvrt -te [xmin ymin xmax ymax] infile outfile.vrt

Unfortunally you will need a georeferenced file for using gdalbuildvrt.

After you clip, don't forget to create pyramids for easy viewing. The following command will build external pyramids compressed with the DEFLATE (lossless) algorithm:

gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE outfile 2 4 8 16

As a final step your can calculate the statistics as well to avoid some stupid problems with a particular commercial program:

gdalinfo -stats outfile
  • Thanks for the answer nickves. I have built a virtual raster and ran <code>gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE outfile 2 4 8 16</code> I am not too sure why I got the following error message ERROR 1: Cannot create TIFF file due to missing codec for DEFLATE. Overview building failed. – Blue Oct 17 '13 at 6:14
  • Check bugs qgis-bug:8782 and osge4w-bug:382 if they describe your case. In meantime try to create the overviews without any compression algorithms, or choose another one . – nickves Oct 17 '13 at 8:13
  • it looks like I had it working in 2 steps: gdaladdo -ro input.vrt 2 4 8 16 then gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE input.vrt 2 4 8 16. My goal is to end up with an ECW so I tried gdal_translate -of ECW input.vrt output.ecw and got following error : 0ERROR 6: GDALDriver::Create() ... no create method implemented for this format. – Blue Oct 18 '13 at 3:44
  • @blue ECW is a proprietary format and it needs special treatment. Please check first if gdal can write in ECW with gdalinfo --formats. If you don't see the ECW in the list, try those instructions: faunalia.pt/node/438 – nickves Oct 18 '13 at 6:10
0

A straight 'clip' using gdalwarp should work (I know this is a hella-old question: 18 months IRL is like a geological epoch in internet years).

I have a 70Gb aerial (ECW, 94000x81000 pixel at 10cm/px), and GDAL can arbitrarily clip it with a shapefile using

gdalwarp -cutline [clipfile] -crop_to_cutline [infile] [outfile]

at the Windows command-line. (I realise that this solution requires building a shapefile for the target area of interest, but that's not the greatest challenge known to GIS).

To extract a half-suburb-sized chunk takes ~0.4sec; to slice the file into quarters takes 4sec. My machine's specs are not dramatically different (i7-4770 @ 3.4GHz, 16GB RAM, Win7-64 Ultimate).

  • gdalwarp was already mentioned. The info about performance and specific example image size is a welcome addition, but really this should be a comment to gis.stackexchange.com/a/74450/108 (and making a suggested edit with the command line with code formatting would be welcome too; it is easier to read) – matt wilkie Jun 1 '15 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.