I have a python script that runs a set of queries in parallel using multiprocessing.pool and map_async.

The script from @Aaron works perfectly.

Now I want to integrate the same functions into a PyWPS service, but I do not succeed. The processing does not start and I get no error in return.

Here is my PyWPS script:

import os, sys
from pywps.Process.Process import WPSProcess
import multiprocessing
from multiprocessing import Pool
from pg import DB
from functools import partial....
db = DB(dbname='****', host='****', port=****, user='****', passwd='****')

class Process(WPSProcess):
     def __init__(self):
         # init process
         self.input1 = ...
         self.output1 = ...

     def execute(self):
        tup = db.query("""
                SELECT distinct code_cc from comm_pop limit 200

        code_list = list(zip(*tup)[0])

        db.query("""DROP TABLE IF EXISTS tpop.tpop_sample""")
        db.query("""CREATE TABLE IF NOT EXISTS tpop.tpop_sample(...)""")

        pool = multiprocessing.Pool(processes=7)
        func = partial(self.tpop, self.input1.getValue())
        pool.map_async(func, code_list)

        out = db.query("""SELECT * from tpop.tpop_sample""").getresult()

     def tpop(self, tabcouv, code):
                INSERT INTO tpop.tpop_sample...do something"""% (tabcouv, code, code))

I have no error message in the pywps log file, I have only an info message:
INFO: Status [processstarted][0.0]: Process pop started

I have already tested without the multiprocessing and I got results.
For information I'am using PyWPS 3.2.5 with Python 2.7

  • I'm always looking for a solution!
    – Adam Adam
    Jun 7, 2017 at 14:43

1 Answer 1


Back in 2012 I spent quite a while trying to get Multiprocessing to work with PyWPS 3. There is some sort of limitation with Python, perhaps related to the way Apache spawns sub-process, it simply does not function. I never managed to find exactly why. There are more recent discussions that can provide more clues.

However, it is possible to execute GRASS calls in parallel. Like so:

workers = multi.cpu_count()
jobs = range(10)

# Check if workers are already being used
if workers is 1 and "WORKERS" in os.environ:
    workers = int(os.environ["WORKERS"])
if workers < 1:
    workers = 1

# Initialize process dictionary
proc = {}
num = 1

# Loop over jobs
for i in jobs:
    # Insert job into dictinoary to keep track of it
    proc[num - 1] = grass.start_command("r.grow.distance", input="map_" + str(i), distance="dist_" + str(i))

    # If the workers are used up, wait for all of them from the last group to finish.
    if num % workers is 0:
        proc[num - 1].wait()
    num += 1

# Make sure all workers are finished.
for i in jobs:
    if proc[i].wait() is not 0:
          grass.fatal(_('Problem running distance on map_' + str(i) + '.'))

More details in the GRASS Wiki. And why not try it with PyWPS 4?

  • Thanks for the answer, Grass can not help me because I use PostGIS instead. Now I'm testing PyWPS 4. I'll keep you informed
    – Adam Adam
    Jun 8, 2017 at 7:54
  • 1
    The problem was not with PyWPS, it was in the use of python multiprocessing.pool inside a class. There is many discussions about this stackoverflow.com/questions/27318290/…
    – Adam Adam
    Jun 9, 2017 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.