Easy examples for multiprocessing in PyQGIS

Question

I've gone through a large number of forum threads and cannot make sense of how possible it is to easily run two tools at the same time in PyQGIS.

Basically what I'm wanting to do is:

def function1():
    processing.run("gdalsomething"...)
def function2():
    processing.run("gdalsomethingelse"...)

doAtTheSameTime( function1() , function2() )

I've tried the Python threading module (but learnt I was wanting multiprocessing, not multithreading?)

I've tried the multiprocessing module but that spits out an array of errors (including sys.argv not existing), perhaps relating to the fact that __name__ refuses to be __main__, and is __console__ instead (not that I know what that means)

Does anyone have a very simple example of running two QGIS tools at the same time in PyQGIS? (If it is even possible?)

Answer 1

I strongly recommend you follow PyQGIS Developer Cookbook | 15. Tasks - doing heavy work in the background to create your tasks but below is the explanation of your error.

---

In Python functions are objects that you can pass around by name.

QgsTask.fromFunction requires you to pass a function object to it.

Your code currently passes the return values of your functions instead because you execute the functions (the ()s). In Python, functions return None if there is no explicit return statement with a different value.

So instead of getting a function object, QgsTask.fromFunction gets None and the assertion fails.

Try:

import random
from qgis.core import *

def one():
    processing.run("native:slope", {'INPUT':'...Dem.tif','Z_FACTOR':1,'OUTPUT':'...Slope1.tif'})
def two():
    processing.run("native:slope", {'INPUT':'...Dem.tif','Z_FACTOR':2,'OUTPUT':'...Slope2.tif'})

task1 = QgsTask.fromFunction('Please work', one)  # <- changed
task2 = QgsTask.fromFunction('Please work 2', two)  # <- changed

task1.addSubTask(task2)
QgsApplication.taskManager().addTask(task1)

Answer 2

I've made some more progression on this.

---

INITIAL CODE UPDATE following on from bugmenot's help:

The below code works, and as far as I can tell this is the simplest possible way to run two tools at the same time in PyQGIS

Critically, the functions which are used in the QgsTask definition must have a parameter called task

from qgis.core import *

def one(task):
    processing.run("native:slope", {'INPUT':'...Dem.tif','Z_FACTOR':1,'OUTPUT':'...Slope1.tif'})
def two(task):
    processing.run("native:slope", {'INPUT':'...Dem.tif','Z_FACTOR':2,'OUTPUT':'...Slope2.tif'})

task1 = QgsTask.fromFunction('ItWorks', one)
task2 = QgsTask.fromFunction('ItWorks2', two)

task1.addSubTask(task2)
QgsApplication.taskManager().addTask(task1)

Please note that the script does not wait for addTask to complete, and simply powers on, hence why more advanced setups have handling to figure out when the task has completed

---

I've also attempted to try and run several tools in sequence, though with a set up such that I want one task to not start until after two other tasks have completed

def firstProcess(task):
    processing.run("qgis:rastercalculator", ...)
    time.sleep(0.1)

def secondProcess(task):
    processing.run("qgis:rastercalculator", ...)
    time.sleep(0.1)

firstProcessTask = QgsTask.fromFunction('FirstProcess', firstProcess)
secondProcessTask = QgsTask.fromFunction('SecondProcess', secondProcess)

firstProcessTask.addSubTask(secondProcessTask)
QgsApplication.taskManager().addTask(firstProcessTask)

    
####################################


def thirdProcess(task):
    try:
        firstProcessTask.waitForFinished(timeout = 20000000)
    except BaseException as e:
        print("...")
    try:
        secondProcessTask.waitForFinished(timeout = 20000000)
    except BaseException as e:
        print("...")
    processing.run("qgis:rastercalculator", ...)   

thirdProcessTask = QgsTask.fromFunction('ThirdProcess', thirdProcess)
QgsApplication.taskManager().addTask(thirdProcessTask)

It's a little clunky, but it's the best I've been able to do given that if two tasks are run together through addSubTask they appear to trigger task.waitForFinished independently

If anyone has any improvements that would be great