I am running an ArcPy script with IDLE 2.7
For each line in a geometric network, it traces the path to a fixed location from that input line. For each input line, it does this trace with all other lines as a barrier one at a time.
So 400 nodes will each perform 399 traces. Or 159,600 traces
I have found that each loop takes longer, on average, than the previous. When I started the testing, I saw that they were taking ~2.5 min to do the 399 traces for each input node. This put me at ~17 hours to finish. Now I'm seeing a steady increase in completion time for each complete loop of the 399 traces.
I'm at the 55th iteration right now and it's pushing 6 min now to complete each iteration.
this trend projects the 400th iteration to take 23 minutes! So, on average, I estimate 12 min each * 400 iterations = 80 hours to run!
The chart below shows the completion time of each loop vs the loop number. it seems to be a pretty steady increase.
The number of processes does not increase with each loop; it does the same thing, just for 400 different points.
I don't think I'm looking for an improvement to the code right now, but rather as a non-computer-scientist, I'm wondering if this is simply expected functionality of how For loops execute.
Does it eat up memory that slows it down each time?
weight = 1
mains_cur =arcpy.SearchCursor(mains)
for main_cur in mains_cur:
start_timeloop1 = datetime.datetime.now()
ID = int(main_cur.getValue(("OBJECTID")))
whereclause = ("OBJECTID" + ' = ' + repr(ID))
whereclause_NOT = ("OBJECTID" + ' <> ' + repr(ID))
arcpy.SelectLayerByAttribute_management(mains_Lyr,"NEW_SELECTION", whereclause)
arcpy.FeatureVerticesToPoints_management(mains_Lyr, r"in_memory\traceflag_A", "MID")
arcpy.Merge_management([r"in_memory\traceflag_A",source],r"in_memory\traceflag")
traceflag = r"in_memory\traceflag"
arcpy.SelectLayerByAttribute_management(mains_Lyr,"NEW_SELECTION", whereclause_NOT)
arcpy.MakeFeatureLayer_management(mains_Lyr, r"in_memory\NOTmains_Lyr", "","","")
NOT_mains = r"in_memory\NOTmains_Lyr"
# For each OTHER pipe, make a barrier at mid point
not_mains_cur =arcpy.SearchCursor(NOT_mains)
for NOT_main in not_mains_cur:
ID2 = int(NOT_main.getValue(("OBJECTID")))
whereclause = ("OBJECTID" + ' = ' + repr(ID2))
arcpy.SelectLayerByAttribute_management(NOT_mains,"NEW_SELECTION", whereclause)
arcpy.MakeFeatureLayer_management(NOT_mains, r"in_memory\not_main4point")
arcpy.FeatureVerticesToPoints_management(r"in_memory\not_main4point",
r"in_memory\Not_main_pt", "MID")
bar = r"in_memory\Not_main_pt"
arcpy.TraceGeometricNetwork_management(geomnet, r"in_memory\outNet",
traceflag, "FIND_PATH", bar,"", "", "", "", "NO_TRACE_ENDS",
"TRACE_INDETERMINATE_FLOW", "", "","AS_IS", "", "", "", "AS_IS")
for layer in arcpy.mapping.Layer(r"in_memory\outNet"):
if str(layer)== "Water_Mains":
calcstr = "["+ str(fieldname)+"]" +r"+" + str(weight)
arcpy.CalculateField_management(layer, fieldname, calcstr, "VB", "")
end_timeloop1 = datetime.datetime.now()
print('Duration: {}'.format(end_timeloop1 - start_timeloop1))
Edit: in the above, this appears to be what really adds time to each step:
for layer in arcpy.mapping.Layer(r"in_memory\outNet"):
if str(layer)== "Water_Mains":
calcstr = "["+ str(fieldname)+"]" +r"+" + str(weight)
arcpy.CalculateField_management(layer, fieldname, calcstr, "VB", "")
I hypothesize that even though it is only adding +1 to the traced records, after a while, it's adding +1 to existing values of 100,000+ storing and calulating with progressively larger numbers may lead to more writes and time? the field it's calculating to is formatted as long.
if os.path.exists('stop.now'): break
in any long-running job, allowing it to end gracefully.arcpy.SearchCursor
is deprecated because it's so inefficient. Nesting search cursors is something to avoid; all the more so with old-style cursors. You don't appear to be deleting intermediate products, and altering theNOT_mains
selection set inside the NOT_mains cursor may produce random results.