Arcpy Freezing on Field Calculation

Question

I am trying to populate a grid with the number of features in each grid cell. I've gotten this script to work with a subset of the data, but when I try to run it on the full (52,000+) cell feature class, it freezes when doing the Field Calculation and won't progress past it. I want to run it outside of Arc so it processes faster than the 20 minutes it took to run inside Arc.

I'm using Identity to split these features by the grid cells, so I get an accurate count, then spatial joining that to the Grid, then joining the FID from the spatial join to the original grid and field calculating the Join Count over. This is done for several feature classes, so I need it to calculate into the original Grid FC.

Additionally can arcpy do feature layer outputs like running the script inside Arc?

Whenever I have my input as "Input" I get an error 000210 cannot create output.

arcpy.MakeFeatureLayer_management(Grid, "Grid_Lyr")
Feature_Split = "C:\Users\Testing.gdb\Feature_Split"
Feature_Join = "C:\Users\Testing.gdb\Feature_Join"
print "Counting Features"
arcpy.Identity_analysis(Ways, Grid, Feature_Split)
print "Identity Analysis Done"
arcpy.SpatialJoin_analysis("Grid_Lyr", Feature_Split, Feature_Join , "JOIN_ONE_TO_ONE", "KEEP_ALL", "", "CONTAINS")
print "Spatial Join Done"
arcpy.AddJoin_management("Grid_Lyr", "OID", Feature_Join, "TARGET_FID", "KEEP_ALL")
print "Join Added"
arcpy.CalculateField_management("Grid_Lyr", "Feature_Count", "!Feature_Join.Join_Count!", "PYTHON_9.3", "")
arcpy.RemoveJoin_management ("Grid_Lyr")
print "Fields Calculated"

Answer 1

Since your working in arcpy and doing a simple data transfer, use a search cursor to load your Spatial Join count results as the value in a dictionary associated with the FID key and then use and update cursor to read the FID from the current feature being updated and lookup the count from the dictionary to post it. It takes a few more lines of code to set up, but with a da cursor it will process your data 20 times faster than a field calculation using a join. See my Blog on Turbo Charging Data Manipulation with Python Cursors and Dictionaries.

Replace the Add Join and Field Calculation steps with this:

import arcpy  

arcpy.MakeFeatureLayer_management(Grid, "Grid_Lyr")
Feature_Split = r"C:\Users\Testing.gdb\Feature_Split"
Feature_Join = r"C:\Users\Testing.gdb\Feature_Join"
print "Counting Features"
arcpy.Identity_analysis(Ways, Grid, Feature_Split)
print "Identity Analysis Done"
arcpy.SpatialJoin_analysis("Grid_Lyr", Feature_Split, Feature_Join , "JOIN_ONE_TO_ONE", "KEEP_ALL", "", "CONTAINS")
print "Spatial Join Done"

sourceFC = Feature_Join  

sourceFieldsList = ["TARGET_FID", "JOIN_COUNT" ]  

# Use list comprehension to build a dictionary from a da SearchCursor  
valueDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, sourceFieldsList)}  

updateFC = Feature_Split  

updateFieldsList = ["OID@", "Feature_Count"]  

with arcpy.da.UpdateCursor(updateFC, updateFieldsList) as updateRows:  
    for updateRow in updateRows:  
        # store the Join value of the row being updated in a keyValue variable  
        keyValue = updateRow[0]  
        # verify that the keyValue is in the Dictionary  
        if keyValue in valueDict:  
            # transfer the values stored under the keyValue from the dictionary to the updated fields.  
            for n in range (1,len(sourceFieldsList)):  
                updateRow[n] = valueDict[keyValue][n-1]  
            updateRows.updateRow(updateRow)  

del valueDict

print "Fields Updated"

If you are going to do Add Join you need to add a step to add an Attribute Index to the TARGET_FID field in the Spatial Join output before joining the data to improve performance. The script above does not require an index to improve performance.