PyWPS with python multiprocessing

Question

I have a python script that runs a set of queries in parallel using multiprocessing.pool and map_async.

The script from @Aaron works perfectly.

Now I want to integrate the same functions into a PyWPS service, but I do not succeed. The processing does not start and I get no error in return.

Here is my PyWPS script:

import os, sys
from pywps.Process.Process import WPSProcess
import multiprocessing
from multiprocessing import Pool
from pg import DB
from functools import partial....
db = DB(dbname='****', host='****', port=****, user='****', passwd='****')

class Process(WPSProcess):
     def __init__(self):
         # init process
         WPSProcess.__init__(self,...)
         self.input1 = ...
         self.output1 = ...

     def execute(self):
        tup = db.query("""
                SELECT distinct code_cc from comm_pop limit 200
                       """).getresult()

        code_list = list(zip(*tup)[0])

        db.query("""DROP TABLE IF EXISTS tpop.tpop_sample""")
        db.query("""CREATE TABLE IF NOT EXISTS tpop.tpop_sample(...)""")

        pool = multiprocessing.Pool(processes=7)
        func = partial(self.tpop, self.input1.getValue())
        pool.map_async(func, code_list)
        pool.close()
        pool.join() 

        out = db.query("""SELECT * from tpop.tpop_sample""").getresult()
        self.output1.setValue(out)

     def tpop(self, tabcouv, code):
        db.query("""
                INSERT INTO tpop.tpop_sample...do something"""% (tabcouv, code, code))

I have no error message in the pywps log file, I have only an info message:
INFO: Status [processstarted][0.0]: Process pop started

I have already tested without the multiprocessing and I got results.
For information I'am using PyWPS 3.2.5 with Python 2.7

Answer 1

Back in 2012 I spent quite a while trying to get Multiprocessing to work with PyWPS 3. There is some sort of limitation with Python, perhaps related to the way Apache spawns sub-process, it simply does not function. I never managed to find exactly why. There are more recent discussions that can provide more clues.

However, it is possible to execute GRASS calls in parallel. Like so:

workers = multi.cpu_count()
jobs = range(10)

# Check if workers are already being used
if workers is 1 and "WORKERS" in os.environ:
    workers = int(os.environ["WORKERS"])
if workers < 1:
    workers = 1

# Initialize process dictionary
proc = {}
num = 1

# Loop over jobs
for i in jobs:
    # Insert job into dictinoary to keep track of it
    proc[num - 1] = grass.start_command("r.grow.distance", input="map_" + str(i), distance="dist_" + str(i))

    # If the workers are used up, wait for all of them from the last group to finish.
    if num % workers is 0:
        proc[num - 1].wait()
    num += 1

# Make sure all workers are finished.
for i in jobs:
    if proc[i].wait() is not 0:
          grass.fatal(_('Problem running distance on map_' + str(i) + '.'))

More details in the GRASS Wiki. And why not try it with PyWPS 4?