Full utilization of CPU-cores with gdalwarp

Question

I need to process many large GeoTiff files with gdalwarp. For this I have used the following command:

gdalwarp -r cubic -s_srs EPSG:25832 -t_srs EPSG:3857 -cutline shapefile.shp -crop_to_cutline -dstalpha file1.tif file2.tif /path/to/output/out.tif

The problem here is that only one CPU core is used and the calculation takes a very long time. Now I've seen that gdalwarp has the option -multi.

So I tried the following command:

gdalwarp -multi -r cubic -s_srs EPSG:25832 -t_srs EPSG:3857 -cutline shapefile.shp -crop_to_cutline -dstalpha file1.tif file2.tif /path/to/output/out.tif -wo NUM_THREADS=4

Now, gdalwarp uses 4 cores, but the utilization per core is only at 25%.

Is there a way to speed up the process or to fully utilize the cores used?

Answer 1

If you are using a Linux OS, you could consider using GNU parallel to process multiple files in parallel.

For instance, a simple example using gdalinfo in parallel is:

 cat list_of_images.txt | parallel -j 4 gdalinfo {} 

Where list_of_images.txt, is a text file containing the filename.

To address your above example, you will need to put your command into a script that will accept multiple arguments (e.g. input and output filenames). You could then do something like:

cat list_of_images.txt | parallel -n 2 -j 4 ./script.sh {}

Where -n is the number of arguments and -j is the number of cores you want to use. Leave out -j to run one job per core.

The following list provides a useful set of examples based on GNU parallel. An alternative to GNU parallel, is xargs, which uses a similar syntax.

Answer 2

Once in a seminar at Uni Potsdam I think i was told that there is the possibility to use

-core n or -cores n

(not quite sure about the right spelling) where n is the number of cores gdal should use.

As I remember, it was a way faster but of course leeched every bit of ressources so you can not really use the computer while processing with every core.