I spent quite a bit of time looking on how to use the multiprocessing package, but couldn't find anything on how to use it inside a plugin in QGIS. I am developing a plugin that does some optimization for several elements. I would like to parallelize it.

I found a useful link on multi-threading inside a python plugin (http://snorf.net/blog/2013/12/07/multithreading-in-qgis-python-plugins/), but nothing on using the multiprocessing module, which might be easier?

I have been trying with a very basic example. I am only showing the run function from the plugin here:

def run(self):
    """Run method that performs all the real work"""
    # show the dialog

    # Run the dialog event loop
    result = self.dlg.exec_()

    # See if OK was pressed and run code
    if result:
        #Get number of cores
        nProcs = mp.cpu_count()

        #Start a Process
        p = mp.Pool(nProcs)

        #Define function     
            def cube(x):
                return x**3

        #Run parallel        
        results = p.map(cube, range(1,7))

When I run this code from the plugin in QGIS, it opens several QGIS windows, which then return errors (can't load layers, etc.).

What am I missing?

Do I need to start a worker first on another thread and then use multiprocessing there?

Or would we use another function from multiprocessing?

I am working under windows 7, using QGIS 2.10.

  • I'm not familiar with mp.Pool but have done a bit of multiprocesing in python, C# and C++... it's not easy. It sounds like your subprocesses don't have a reference to your application (QgisApplication) or are trying to modify the same resource at the same time; in C++ you would have a mutex to pass around the threads but your code is too simple for that - are you only trying to cube a number? Jul 28, 2015 at 5:10
  • No, I am trying to do an optimization for several items, but essentially it would work the same, which is why I tested with a simple function first. Using "multiprocessing" in python directly is that simple, I am wondering how it can be adapted to work in python inside qgis.
    – Olivier
    Jul 29, 2015 at 6:50