5

I have developed a QGIS plugin that processes an image using a bunch of other information, but it runs slowly. I would like to parallelize, as I can do with Multiprocessing.Process outside a QGIS plugin. I have seen several questions about using Multiprocessing in a QGIS plugin (for example this one), but no real answers or examples. As far as I can tell, the problem is related to management of processes, but I don't understand exactly how.

I'm using the following structure to create multiple (e.g. 5) processes:

def doprocessing():

### Test multiprocessing ###
import multiprocessing as multi
import worker

njobs = 5
jobs = []
for i in range(njobs):
    p = multi.Process(target=worker.worker)
    jobs.append(p)
    p.start()

for p in jobs:
    p.join()
    if str(p.exitcode) != '0':
        print ("Error running process_chunk as process")
        sys.exit(1)

The worker in my test just does something silly:

def worker():
"""worker function"""
#print 'Worker'

import random
import time

value = int(random.random() * 1000000)
time_of_process = time.time()

filename = "D:/Anders/SDB/multiprocessingtest_" + str(value) + ".txt"
output_file = open(filename, "w")
output_file.write(str(time_of_process))
output_file.close()

return 0

I call this from the plugin code like this (just showing you the last bit here):

    def run(self):
    """Run method that performs all the real work"""
    # show the dialog
    self.dlg.show()
    # Run the dialog event loop
    result = self.dlg.exec_()
    # See if OK was pressed
    if result:
        import multiprocessing_test_process as process
        process.doprocessing()

When I run this through my plugin, all kinds of crazy breaks loose, with multiple new instances of QGIS starting up, and the original QGIS freezing.

If I change the first code snippet to the following and run it outside QGIS (in PyCharm), it works as it should:

def doprocessing():

### Test multiprocessing ###
import multiprocessing as multi
import worker

if __name__ == '__main__':
    njobs = 5
    jobs = []
    for i in range(njobs):
        p = multi.Process(target=worker.worker)
        jobs.append(p)
        p.start()

    for p in jobs:
        p.join()
        if str(p.exitcode) != '0':
            print ("Error running process_chunk as process")
            sys.exit(1)

doprocessing()

So it's related to the use of main or not. Any help, pointers, and preferably examples that are known to work in QGIS plugins would be great! Alternatively, if you simply know of a plugin QGIS that successfully uses Multiprocessing, please just point me to it and I will take it from there.

  • 1
    I can't fix your problem, but there is a new framework for multithreading in QGIS 3 called QgsTaskManager, QgsTask available in Python. You can have a preview here youtube.com/watch?v=3KzwNErzkJg&feature=youtu.be It has been merged in QGIS 3 and they started using this framework already, like saving vector layer in the background. You should think about using it, as it will be integrated into QGIS very well. – etrimaille May 4 '17 at 7:38
1

Thanks for the suggestion Gustry, I look forward to trying it out when QGIS 3 comes alive. In the meantime I tried different things, and the only way I could get my plugin to parallelize and speed up was to have it call an external python file, which in turns does the parallel processing. So what I do now in my plugin is:

1) Take all the information passed by the user in the plugin interface, and write it to a file (called argsFile below)

2) Run the following:

import subprocess
basepath = os.path.dirname(os.path.abspath(__file__))  # Get plugin path
fileToRun = basepath + "/my_code_that_uses_parallel_processing.py"
subprocess.Popen(["python", fileToRun, argsFile])

Not elegant, but it gives me the speed-up I want. I'd be happy to see suggestions for improvement, whether this becomes irrelevant in QGIS3 or not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.