3

I'm attempting to speed up a process which is currently running synchronously, by using the python multiprocessing module.

I'm having trouble sending a feature layer to a function which is called by multiprocessing, as demonstrated in this simple script:

import multiprocessing, arcpy

def doProcess(lyr):
    print(lyr.name)

if __name__ == '__main__':

    #Create an array of feature layers
    arcpy.env.workspace = "C:\Program Files (x86)\ArcGIS\Desktop10.2\TemplateData\TemplateData.gdb"
    featureLayers = []
    fcs = arcpy.ListFeatureClasses("*","All","World")
    for fc in fcs:
        arcpy.Delete_management(fc + "_lyr")
        lyrName = fc + "_lyr"
        arcpy.MakeFeatureLayer_management(fc, lyrName)
        featureLayers.append(arcpy.mapping.Layer(lyrName))

    #This works when not using multiprocessing:
    for featureLayer in featureLayers:
        doProcess(featureLayer)

    #This fails with "UnpickleableError: Cannot pickle <type 'geoprocessing Layer object'> objects"
    pool = multiprocessing.Pool()
    pool.map(doProcess, featureLayers)
    pool.close()
    pool.join()

When iterating over the array manually, rather than using multiprocessing, the function has access to the feature layer. But when using multiprocessing, this error message is shown:

UnpickleableError: Cannot pickle type 'geoprocessing Layer object' objects

What is the correct syntax/approach to handle a feature layer within the multiprocessing environment? I based the above script on the example on the Esri blog Multiprocessing with ArcGIS

  • 2
    There is a GeoNet thread worth looking at on this one: geonet.esri.com/thread/31027 where Jason Scheirer says "many objects in arcpy/arcgisscripting are not pickleable, so sending them directly across the wire in multiprocessing will not work. There are workarounds, however." – PolyGeo Sep 18 '14 at 5:47
  • 1
    Maybe you should build an ad-hoc object using python built-in type to make sure it would be "pickleable" – Peter Peterson Sep 18 '14 at 7:22
3

I finally found the time to look into this. I don't fully understand the "unpickleable" error message, but a workaround is to pass only strings into the multiprocessor. Something like this:

import multiprocessing, arcpy, os

def doProcess(fClass):
    #This function doesn't do anything, it's just to show that accessing arcpy methods is possible
    print("in do process function for " + fClass)
    arcpy.env.workspace = "C:\Program Files (x86)\ArcGIS\Desktop10.2\TemplateData\TemplateData.gdb"
    arcpy.Delete_management(fClass + "_lyr")
    lyrName = fClass + "_lyr"
    arcpy.MakeFeatureLayer_management(fClass, lyrName)
    desc = arcpy.Describe(lyrName)
    print("Finished " + desc.Name)

if __name__ == '__main__':

    #Create an array of feature class names
    arcpy.env.workspace = "C:\Program Files (x86)\ArcGIS\Desktop10.2\TemplateData\TemplateData.gdb"
    fClasses = []
    fcs = arcpy.ListFeatureClasses("*","All","World")
    for fc in fcs:
        fClasses.append(fc)

    #Multiprocessing approach
    pool = multiprocessing.Pool()
    pool.map(doProcess, fClasses)
    pool.close()
    pool.join()

(Interestingly, this script takes a lot longer to complete when I use the multiprocessing approach, compared to just running:

for fClass in fClasses:
    doProcess(fClass)

Presumably there's a lot more overhead in setting up the environments for each thread. Hopefully in a more complicated scenario involving long geoprocessing tasks, the payoff would be faster overall completion of all tasks.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.