Definitive list of QGIS multiprocessing tools?

Question

This question is not as straightforward as it seems, but can be summed up with the following:

Is there a definitive list of geoprocessing tools for QGIS that currently support multithreaded processing? The intent is to use more than a single core.

It is well understood that multithreaded rendering is available in QGIS 3, but this has no effect on geoprocessing operations, and in the question history on the various stacks seems to commonly be confused by respondants.

Background: Datasets are becoming more detailed as acquisition methods increase in detail and density. The requirement for detailed analysis combining multiple datasets and myriad steps to process means that multithreaded processing is an attractive option. For example, ESRI has introduced a small but important handful of multithreaded processing tools in ArcGIS Pro - Pairwise Intersect, Pairwise Buffer, and Pairwise Dissolve. These tools have introduced significant speed enhancements to the traditional intersect, buffer, and dissolve functions when handling large datasets.

QGIS interfaces with a number of GIS toolchains - GDAL, GEOS, Python3 / Cython / Shapely / PostGIS - each which has a different position on whether they support multithreaded or parallel processing. It's really very difficult to know whether a tool you wish to use in QGIS supports multithreaded processing, or which tool from which toolchain might offer a speed improvement without a list of tools that support multithreading. This becomes even more important to users of large datasets when building models that might need to be run with semi regularity.

Despite extensive searching I'm coming up with very little.

So, to answer this question are you either able to point to a preexisting list, or your own list of multithreaded processing options for QGIS?

Answer 1

Thread safety

QGIS Processing algorithms can be thread-safe or not. According to the docs, the latter are :

[...] algorithms which manipulate the current project, layer selections, or with external dependencies which are not thread-safe.

You can run the following code snippet in a QGIS Python console (Ctrl+Alt+P) to obtain a list of QGIS Processing algorithms and whether they are thread-safe or not:

for alg in QgsApplication.processingRegistry().algorithms():
    if alg.flags() & QgsProcessingAlgorithm.FlagNoThreading:
        print("[NOT Thread safe] {}:{} --> {}".format(alg.provider().name(), alg.name(), alg.displayName()))
    else:
        print("[Thread safe] {}:{} --> {}".format(alg.provider().name(), alg.name(), alg.displayName()))

A sample output:

...
[NOT Thread safe] QGIS:selectbyattribute --> Select by attribute
[NOT Thread safe] QGIS:selectbyexpression --> Select by expression
[Thread safe] QGIS:setmvalue --> Set M value
[NOT Thread safe] QGIS:setstyleforrasterlayer --> Set style for raster layer
[NOT Thread safe] QGIS:setstyleforvectorlayer --> Set style for vector layer
[Thread safe] QGIS:setzvalue --> Set Z value
[Thread safe] QGIS:singlesidedbuffer --> Single sided buffer
[Thread safe] QGIS:slope --> Slope
[Thread safe] QGIS:snapgeometries --> Snap geometries to layer
[NOT Thread safe] QGIS:spatialiteexecutesql --> SpatiaLite execute SQL
[Thread safe] QGIS:splitvectorlayer --> Split vector layer
...

Feature-based Algorithms

Algorithms that are thread-safe may be of type QgsProcessingFeatureBasedAlgorithm. These are algorithms where the result for each feature is independent from other features. These are potential multi-threading algorithms, but, according to the docs, it will be implemented in the future.

You can run the following code snippet to get a list of feature-based algorithms:

for alg in QgsApplication.processingRegistry().algorithms():
    if isinstance(alg, QgsProcessingFeatureBasedAlgorithm):
        print("[Feature-based alg] {}:{} --> {}".format(alg.provider().name(), alg.name(), alg.displayName()))

A sample output:

...
[Feature-based alg] QGIS (native c++):boundary --> Boundary
[Feature-based alg] QGIS (native c++):boundingboxes --> Bounding boxes
[Feature-based alg] QGIS (native c++):bufferbym --> Variable width buffer (by m-value)
[Feature-based alg] QGIS (native c++):centroids --> Centroids
[Feature-based alg] QGIS (native c++):convexhull --> Convex hull
[Feature-based alg] QGIS (native c++):deleteholes --> Delete holes
[Feature-based alg] QGIS (native c++):dropgeometries --> Drop geometries
[Feature-based alg] QGIS (native c++):dropmzvalues --> Drop M/Z values
[Feature-based alg] QGIS (native c++):explodelines --> Explode lines
[Feature-based alg] QGIS (native c++):extendlines --> Extend lines
...

---

In summary, (apparently) there are no algorithms that run in multiple threads up to now.

If you need more details, I'd suggest you to ask to the QGIS core devs in the QGIS-developer mailing list about future plans on multi-threading processing algorithms.

Answer 2

Regarding threadsafe operations (but not multiple cores):

This is a partial answer - it addresses what processes are able to run in a threadsafe manner, which is the first step generally required before a multicore implementation can be considered as each thread needs to be aware of the others status to avoid conflicts, etc.

Most importantly, I'll reiterate: This doesn't indicate that a tool listed as threadsafe actually has a multi-core implementation - and that is what I'm still looking for an answer to.

So, lets have a look at threadsafe tools thanks to German Carrillo's little python snippet:

This list is as of QGIS 3.10.2.

Counts of threadsafe tools:

Threadsafe Tools

The list is too long to paste into the body or in a code block. See the GIST instead.