2

I have a python script that runs a set of queries in parallel using multiprocessing.pool and map_async.

The script from @Aaron works perfectly.

Now I want to integrate the same functions into a PyWPS service, but I do not succeed. The processing does not start and I get no error in return.

Here is my PyWPS script:

import os, sys
from pywps.Process.Process import WPSProcess
import multiprocessing
from multiprocessing import Pool
from pg import DB
from functools import partial....
db = DB(dbname='****', host='****', port=****, user='****', passwd='****')

class Process(WPSProcess):
     def __init__(self):
         # init process
         WPSProcess.__init__(self,...)
         self.input1 = ...
         self.output1 = ...

     def execute(self):
        tup = db.query("""
                SELECT distinct code_cc from comm_pop limit 200
                       """).getresult()

        code_list = list(zip(*tup)[0])

        db.query("""DROP TABLE IF EXISTS tpop.tpop_sample""")
        db.query("""CREATE TABLE IF NOT EXISTS tpop.tpop_sample(...)""")

        pool = multiprocessing.Pool(processes=7)
        func = partial(self.tpop, self.input1.getValue())
        pool.map_async(func, code_list)
        pool.close()
        pool.join() 

        out = db.query("""SELECT * from tpop.tpop_sample""").getresult()
        self.output1.setValue(out)

     def tpop(self, tabcouv, code):
        db.query("""
                INSERT INTO tpop.tpop_sample...do something"""% (tabcouv, code, code))

I have no error message in the pywps log file, I have only an info message:
INFO: Status [processstarted][0.0]: Process pop started

I have already tested without the multiprocessing and I got results.
For information I'am using PyWPS 3.2.5 with Python 2.7

  • I'm always looking for a solution! – Adam Adam Jun 7 '17 at 14:43
1

Back in 2012 I spent quite a while trying to get Multiprocessing to work with PyWPS 3. There is some sort of limitation with Python, perhaps related to the way Apache spawns sub-process, it simply does not function. I never managed to find exactly why. There are more recent discussions that can provide more clues.

However, it is possible to execute GRASS calls in parallel. Like so:

workers = multi.cpu_count()
jobs = range(10)

# Check if workers are already being used
if workers is 1 and "WORKERS" in os.environ:
    workers = int(os.environ["WORKERS"])
if workers < 1:
    workers = 1

# Initialize process dictionary
proc = {}
num = 1

# Loop over jobs
for i in jobs:
    # Insert job into dictinoary to keep track of it
    proc[num - 1] = grass.start_command("r.grow.distance", input="map_" + str(i), distance="dist_" + str(i))

    # If the workers are used up, wait for all of them from the last group to finish.
    if num % workers is 0:
        proc[num - 1].wait()
    num += 1

# Make sure all workers are finished.
for i in jobs:
    if proc[i].wait() is not 0:
          grass.fatal(_('Problem running distance on map_' + str(i) + '.'))

More details in the GRASS Wiki. And why not try it with PyWPS 4?

  • Thanks for the answer, Grass can not help me because I use PostGIS instead. Now I'm testing PyWPS 4. I'll keep you informed – Adam Adam Jun 8 '17 at 7:54
  • 1
    The problem was not with PyWPS, it was in the use of python multiprocessing.pool inside a class. There is many discussions about this stackoverflow.com/questions/27318290/… – Adam Adam Jun 9 '17 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.