2

I need to process many large GeoTiff files with gdalwarp. For this I have used the following command:

gdalwarp -r cubic -s_srs EPSG:25832 -t_srs EPSG:3857 -cutline shapefile.shp -crop_to_cutline -dstalpha file1.tif file2.tif /path/to/output/out.tif

The problem here is that only one CPU core is used and the calculation takes a very long time. Now I've seen that gdalwarp has the option -multi.

So I tried the following command:

gdalwarp -multi -r cubic -s_srs EPSG:25832 -t_srs EPSG:3857 -cutline shapefile.shp -crop_to_cutline -dstalpha file1.tif file2.tif /path/to/output/out.tif -wo NUM_THREADS=4

Now, gdalwarp uses 4 cores, but the utilization per core is only at 25%.

Is there a way to speed up the process or to fully utilize the cores used?

4

If you are using a Linux OS, you could consider using GNU parallel to process multiple files in parallel.

For instance, a simple example using gdalinfo in parallel is:

 cat list_of_images.txt | parallel -j 4 gdalinfo {} 

Where list_of_images.txt, is a text file containing the filename.

To address your above example, you will need to put your command into a script that will accept multiple arguments (e.g. input and output filenames). You could then do something like:

cat list_of_images.txt | parallel -n 2 -j 4 ./script.sh {}

Where -n is the number of arguments and -j is the number of cores you want to use. Leave out -j to run one job per core.

The following list provides a useful set of examples based on GNU parallel. An alternative to GNU parallel, is xargs, which uses a similar syntax.

-1

Once in a seminar at Uni Potsdam I think i was told that there is the possibility to use

-core n or -cores n

(not quite sure about the right spelling) where n is the number of cores gdal should use.

As I remember, it was a way faster but of course leeched every bit of ressources so you can not really use the computer while processing with every core.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy