Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I am interested in learning methods to utilize the full extent of multicore processing power available on a desktop computer. Arc states that background geoprocessing allows the user to utilize multiple cores, however, tasks essentially have to wait in line for the previous task to be completed.

Has anyone developed parallel or multithreaded geoprocessing methods in Arc/Python? Are there hardware bottlenecks that prevent multicore processing on individual tasks?

I found an interesting example in Stackoverflow that caught my interest, although it is not a geoprocessing example:

from multiprocessing import Pool
import numpy

numToFactor = 976

def isFactor(x):
    result = None
    div = (numToFactor / x)
    if div*x == numToFactor:
        result = (x,div)
    return result

if __name__ == '__main__':
    pool = Pool(processes=4)
    possibleFactors = range(1,int(numpy.floor(numpy.sqrt(numToFactor)))+1)
    print 'Checking ', possibleFactors
    result = pool.map(isFactor, possibleFactors)
    cleaned = [x for x in result if not x is None]
    print 'Factors are', cleaned
share|improve this question
1  
In my Arc experience, it almost always boils down to either 1) splitting your data up into {number of core} chunks, processing and reassembling or 2) reading everything into memory and letting x API handle the threading. note that this is not meant to discourage. –  valveLondon Jun 26 '12 at 18:35
    
Thanks valveLondon. Perhaps newer Ivy Bridge technology and the Kepler GPU will allow for more sophisticated processing approaches. –  Aaron Jun 27 '12 at 3:04
    
Here's a link to a useful blog about python multiprocessing from a product engineer on ESRIs Analysis and Geoprocessing team. blogs.esri.com/esri/arcgis/2011/08/29/multiprocessing –  Aaron Jun 27 '12 at 4:23

2 Answers 2

up vote 8 down vote accepted

Here is an example of a multicore arcpy script. The process is very CPU-intensive so it scales very well: Port "Producing Building Shadows" Avenue code to ArcGIS 10

Some more general info in this answer: Can concurrent processes be run in a single model?

share|improve this answer

In my experience the biggest problem is managing stability. If you do six weeks of processing in a single night you will also have six weeks of inexplicable errors and bugs.

An alternative approach is to develop standalone scripts that can run independently and fail without causing problems:

  • Split the data into chunks that a single core can process in <20 minutes (tasks).
  • Build a standalone Arcpy script that can process a single task and is as simple as possible (worker).
  • Develop a mechanism to run tasks. Lots of pre-existing python solutions exist. Alternatively you can make your own with a simple queue.
  • Write some code to verify that tasks have been completed. This could be as simple as checking that an output file has been written.
  • Merge data back together.
share|improve this answer
    
Excellent words of wisdom and good advice. I like the segmented, worker approach. –  Aaron Jun 27 '12 at 3:00
    
I've found that this approach, which can include using the multiprocessing module, is a good one - some extensions, such as spatial analyst, don't work very well if you have multiple copies of the same function running simultaneously, so something like what you describe that allows for a user-controlled form of queuing (ie, avoids scheduling those tasks at the same time or avoids using the same geodatabase at once for file locking reasons) is going to be best. –  nicksan Jul 3 '12 at 19:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.