Improving processing time of ArcPy script?

Question

Background: I work for an electrical utility, and we have a point feature class called "meters" that contains approximately 35,000 records. These records represent the actual power meter, and contain the customer's account number among other things.

Approximately 2000 of these records are missing account numbers, and I've been tasked with filling in this information which is required for another project the utility is working on.

I selected out the meters that were missing accounts, and created a feature class containing only those problem meters. I then filled in the missing information using a separate script that I wrote.

Now, I'm trying to take the completed account numbers from the subset feature class and transfer them into the "live" Meter feature class. I wrote a fairly simple script to accomplish this, however it seems to be taking a very long time to run. It ran all night last night without finishing (over 12 hours long) and stopped responding after my PC's connection to the server was booted for backup/synch. So this morning I created a local (non SDE) copy of the Meters feature class and am running the script against that to see if it makes a difference. It's been running now for three hours and I'm feeling like this is taking too long.

I'm hoping someone might be able to take a look at the code below and give me some tips on how to possibly improve the processing time.

import arcpy, time, csv
from arcpy import env
import datetime as dt

logfile = r"U:\Python Scripts\scriptLog.csv"
scriptName = "Meters_AttributeTransfer"
startTime = time.clock()
clockTime = time.time()

workspace = arcpy.GetParameterAsText(0)
arcpy.env.overwriteOutput = True

targetMeters = arcpy.GetParameterAsText(1)
goodMeters = arcpy.GetParameterAsText(2)

tMeterFields = ["LOC_NBR"]
gMeterFields = ["LOC_NBR"]

arcpy.MakeFeatureLayer_management(targetMeters, "tMeters_lyr")
arcpy.MakeFeatureLayer_management(goodMeters, "gMeters_lyr")

with arcpy.da.UpdateCursor(targetMeters, tMeterFields) as uCursor:
    for uRow in uCursor:
        arcpy.SelectLayerByLocation_management("gMeters_lyr", "INTERSECT",     targetMeters, None, "NEW_SELECTION")
        with arcpy.da.SearchCursor("gMeters_lyr", gMeterFields) as bCursor:
            for bRow in bCursor:
                if uRow[0] == bRow[0]:
                    pass
                elif uRow[0] != bRow[0]:
                    uRow[0] = bRow[0]
                    uCursor.updateRow(uRow)

#----Write Log File---------------------------------------------------------------------------------------------------------------
duration = ((time.clock() - startTime))/60
dateObj = dt.datetime.now()
dateStr = dateObj.strftime( '%Y-%m-%d' )
newRow = [scriptName, str(duration), dateStr, clockTime, "V0.1"]
with open(logfile, 'a') as edit:
    writer = csv.writer(edit)
    writer.writerow(newRow)

The idea for this script is fairly simple:

An update cursor is created against the "target meters", and for each row in the target meter feature class, the matching/intersecting point in the "good meters" feature class (which contains the missing account information) is selected and searched to see if there is a matching number in the target feature class. If there is no match, then it copies the account number from the good meter to the target meter and moves on to the next one.

There is also code included which updates a log file that I maintain, which tracks when I run scripts and how long they take, and which version of the script I'm running.

I hope someone has some insight into this.

PS: I'm using ArcGIS 10.2.1 with Python 2.7.

Answer 1

May not be the approach you were looking for, but if you're just using a spatial intersection to get the match, then you could always take your new/updated feature class and run a spatial join against the original feature class. The output from that Spatial Join should have the ObjectID of the original feature class in it along with the attributes from the matching updated feature class (depending on Spatial Join tool/parameter options and settings). From there you should be able to do a table to table join, joining the SPJoin output to the original feature class. And then you can do a select by attribute for all the ones with no meter number in the original feature class and a meter number in the SPJoin output. Finally, do a Calculate Field on the selected records copying the value of the updated meter number into the original feature class.

Yes, I know a lot of people like arcpy.da.... cursors, but sometimes, if you are iterating over every feature and doing other processes repeatedly for every individual feature, it can be rather slow. In these cases, make sure you examine if there are already built in GP tools that can accomplish a part of your task in a single run, rather than having to run a select by tool 35,000 times for example.

Let me know if I need to clarify anything.