Using MultiProcessing With Update Cursor?

Question

I am trying to use multiprocessing to help me run a tool that will clip a feature and sum an attribute in it. The tool needs to iterate through about 24,000 rows of a featureclass while doing this. I can run it without multiprocessing but it takes a long time (over 20 hours). Because of that, I am trying to use multiprocessing to get it to run faster, as I will likely have to use this type of tool again on larger sets of data. However, when I run it, it runs and finishes, but there is no output and when I look at the shapefiles created during it, the updatecursor has not run on them. I am hoping someone can help me code it better so that it works.

import arcpy
import os
import multiprocessing

#this toolbox is full of tools I've made, I use one of them later
arcpy.ImportToolbox( 
r'C:\Users\joe.chestnut\Documents\PythonScripts\JoesTools.pyt')
#geodatabase that my files are all in
arcpy.env.workspace = r"D:\Joe.Chestnut\Dallas\Dallas.gdb"




#This is the actual function that I need to run.  
def ClipCount(stuffpassed):
    #ranges are based on the oid.  they stratify the data so that it can be multiprocessed. 
    ranges = stuffpassed[0]
    #this is my file with about 25,000 polygon features (service areas for census blocks in Dallas
    inputfile = stuffpassed[1]
    #This file is the census blocks in dallas, and has some job data in it
    inputfile2 = stuffpassed[2]
    print(ranges)
    print"RunningClipCount)"
    arcpy.env.workspace =r"D:\Joe.Chestnut\Dallas" 
    i, j = ranges[0], ranges[1]
    #This copies my file into a folder and creates 4 different copies of it, for the four different processors to run on
    infile = arcpy.management.CopyFeatures(inputfile,  'layer{0}'.format(i), """OID>= {0} AND OID <= {1}""".format(i, j))
    print(infile)
    #makes a layer out of my census blocks so that they can be geoprocessed
    cliplayer = arcpy.management.MakeFeatureLayer(inputfile2, "clipmedallas")
    print(cliplayer)
    #this cursor runs through each of the rows of the service areas file, and for each one will clip
    #the census block file.  Using a searchcursor on the clipped output, it will then sum the values in the 'newpop' field
    #the summed values are then written into the 'jobcount' field of the service area file using the update cursor. 
    with arcpy.da.UpdateCursor(infile, ["Name", "jobcount"]) as cursor:
        print "updatecursorrunning"
        for row in cursor:
            value = row[0]
            field = "Name"
            exp = field+"='" + value + "'"
            infile3 = arcpy.MakeFeatureLayer_management(infile, "infile3")
            rowselect = arcpy.SelectLayerByAttribute_management(infile3, "NEW_SELECTION", exp)
            clipedlyr = arcpy.LEHDClipper(cliplayer, rowselect, "in_memory", "in_memory\clipedlyr")
            with arcpy.da.SearchCursor(clipedlyr, "newpop") as newcursor:
                stationpop = []
                for srow in newcursor:
                    value = float(srow[0])
                    stationpop.append(value)

                print(stationpop)

            totalpop= 0
            for num in stationpop:
                totalpop = totalpop + num

            print(totalpop)
            row[1] = totalpop
            cursor.updateRow(row)
            arcpy.Delete_management(clipedlyr)
#This is the main function where I give the inputs for the ClipCount and then actually run the multiprocessing module. 
def main():

    ranges = [[1, 6000], [6001, 12000], [12001, 18000], [18001, 24000]]
    inputfile =r'D:\Joe.Chestnut\Dallas\Dallas.gdb\gtfs\BlockAccess'
    inputfile2 = r'D:\Joe.Chestnut\Dallas\Dallas.gdb\gtfs\Dallas_1clip'
    stuffpassed = [ranges, inputfile, inputfile2]
    pool = multiprocessing.Pool(processes=4, initializer=None, maxtasksperchild = 1)
    result = pool.map_async(ClipCount, stuffpassed)
    pool.close()
    pool.join()    

    print result



if __name__ == '__main__':
    main()

When I run this, I get the following result:

<multiprocessing.pool.MapResult object at 0x1467CAD0>

It also creates four copies of inputfile as shapefiles. However, apart from this, I don't think know if it is doing anything.

Does anyone know what I am doing wrong?

How do I get the actual update cursor to run with the multiprocessing?

Answer 1

My answer is not about resolving your multiprocessor issue but the general approach to count overlapping parts. I advised a method using spatial join in this answer to How to remove duplicate features with the same geometry in ArcMap? which uses ‘ARE_IDENTICAL’ operator with field mapping. In summary if you self union your input, it will keep all overlapping part then spatial join will find identical ones and tally them.

Answer 2

Your multiprocessing call is all wrong which is the crux(or at least first part) of your issue. I have no idea how you're getting four copies of input file as it seems like you should only be making 3, one for each element of your malformed stuffpassed list.

Map_async should make a call for every element of the argument list so your current code should only ever be running clipcount 3 times(one for each element of your current Stuffpassed list). Its also failing because you're saying get oid between two ranges that are two lists not two numbers. So how to fix this:

Stuffpassed is not at all correctly formatted for a multiprocessing call map_async. If you want to keep the way you're parsing the variables inside of clipcount you need stuffpassed to be a list of lists with each element of the outer list being ALL of the variables you wish you pass to each call of clipcount.

This means you need stuffpassed to be empty and then you need to loop through every element of ranges to populate it.

 stuffpassed = list()
 for range in ranges:
      stuffpassed.append([range,infile1,infile2])

Now your multiprocessing call will, at least, be getting the variables it needs and that you think you're sending.

I'm also fairly certain copyfeatures doesn't return anything and you cannot set infile that way but I could be wrong about that.