0

I am processing a bunch of GPS data using a Qgis python script. The processing takes a LONG time to execute and when it is executing I notice that only a single CPU is pinned at 100%, the remaining CPUs are idle.

Before I get too deep into the details of my issue, I am trying to figure out how to create several processes to increase CPU utilization/processing speed for a task that is CPU-bound.

reading this question here: https://stackoverflow.com/questions/15143837/how-to-multi-thread-an-operation-within-a-loop-in-python I moved the contents of a loop in my script into a function then attempted to call it in using this form:

executor = concurrent.futures.ProcessPoolExecutor(10)
futures = [executor.submit(try_my_operation, item) for item in items]
concurrent.futures.wait(futures)

When I executed my script I got an error in the python error panel:

AttributeError: module 'sys' has no attribute 'argv'

After googling around a bit I found that using this at the top of my script should be a work around:

if not hasattr(sys, 'argv'):
    sys.argv  = ['']

When I execute my script now another instance of QGIS opens up and an error message with title "Private qgis.db" displays a message:

Creation of missing tbl_srs in the private qgis.db failed.
table tbl_srs already exists

Click OK and another error message witht he same title pops up:

Update of view in private qgis.db failed.
table vw_srs already exists

This then repeats its self 9 more times and I end up with 10 instances of QGIS open, no of which are actually doing anything...

  • Any reason why you could not use a multiprocessing Pool in python? Very easy to implement. There are a lot of examples out there, but look at the following link. stackoverflow.com/questions/20190668/multiprocessing-a-for-loop – Olivier Feb 21 at 8:33
  • If the multiprocessing class works then I will try and implement with that. I don't have enough python experience to know the pros/cons of one system over the other... Hopefully that works with Qgis because I expected that either method would have worked. – Eanema Feb 22 at 18:45
  • So I have tried to implement a test program and run it in the python console of QGIS. The results are not exactly the same, but quite similar. Again there is a complaint that module 'sys' has not attribute 'argv' and so I tried again to set sys.argv=['']. This caused a new window to open up and give me some critical messages about an invalid data source. It seems to me like perhaps Qgis has done something to the environment to prohibit mulitprocessing? – Eanema Feb 24 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.