Methods for optimizing multcore processing in ArcGIS

Question

I am interested in learning methods to utilize the full extent of multicore processing power available on a desktop computer. Arc states that background geoprocessing allows the user to utilize multiple cores, however, tasks essentially have to wait in line for the previous task to be completed.

Has anyone developed parallel or multithreaded geoprocessing methods in Arc/Python? Are there hardware bottlenecks that prevent multicore processing on individual tasks?

I found an interesting example in Stackoverflow that caught my interest, although it is not a geoprocessing example:

from multiprocessing import Pool
import numpy

numToFactor = 976

def isFactor(x):
    result = None
    div = (numToFactor / x)
    if div*x == numToFactor:
        result = (x,div)
    return result

if __name__ == '__main__':
    pool = Pool(processes=4)
    possibleFactors = range(1,int(numpy.floor(numpy.sqrt(numToFactor)))+1)
    print 'Checking ', possibleFactors
    result = pool.map(isFactor, possibleFactors)
    cleaned = [x for x in result if not x is None]
    print 'Factors are', cleaned

Answer 1

Here is an example of a multicore arcpy script. The process is very CPU-intensive so it scales very well: Porting Avenue code for Producing Building Shadows to ArcPy/Python for ArcGIS Desktop?

Some more general info in this answer: Can concurrent processes be run in a single model?

Answer 2

In my experience the biggest problem is managing stability. If you do six weeks of processing in a single night you will also have six weeks of inexplicable errors and bugs.

An alternative approach is to develop standalone scripts that can run independently and fail without causing problems:

• Split the data into chunks that a single core can process in <20 minutes (tasks).
• Build a standalone Arcpy script that can process a single task and is as simple as possible (worker).
• Develop a mechanism to run tasks. Lots of pre-existing python solutions exist. Alternatively you can make your own with a simple queue.
• Write some code to verify that tasks have been completed. This could be as simple as checking that an output file has been written.
• Merge data back together.