1

I am running an ArcPy script with IDLE 2.7

For each line in a geometric network, it traces the path to a fixed location from that input line. For each input line, it does this trace with all other lines as a barrier one at a time.

So 400 nodes will each perform 399 traces. Or ‭159,600‬ traces

I have found that each loop takes longer, on average, than the previous. When I started the testing, I saw that they were taking ~2.5 min to do the 399 traces for each input node. This put me at ~17 hours to finish. Now I'm seeing a steady increase in completion time for each complete loop of the 399 traces.

I'm at the 55th iteration right now and it's pushing 6 min now to complete each iteration.

this trend projects the 400th iteration to take 23 minutes! So, on average, I estimate 12 min each * 400 iterations = 80 hours to run!

The chart below shows the completion time of each loop vs the loop number. it seems to be a pretty steady increase.

The number of processes does not increase with each loop; it does the same thing, just for 400 different points.

I don't think I'm looking for an improvement to the code right now, but rather as a non-computer-scientist, I'm wondering if this is simply expected functionality of how For loops execute.

Does it eat up memory that slows it down each time?

enter image description here

weight = 1
mains_cur =arcpy.SearchCursor(mains)
for main_cur in mains_cur:
    start_timeloop1 = datetime.datetime.now()
    ID = int(main_cur.getValue(("OBJECTID")))
    whereclause = ("OBJECTID" + ' = ' + repr(ID))
    whereclause_NOT =  ("OBJECTID" + ' <> ' + repr(ID))
    arcpy.SelectLayerByAttribute_management(mains_Lyr,"NEW_SELECTION", whereclause)
    arcpy.FeatureVerticesToPoints_management(mains_Lyr, r"in_memory\traceflag_A", "MID")
    arcpy.Merge_management([r"in_memory\traceflag_A",source],r"in_memory\traceflag")
    traceflag = r"in_memory\traceflag"
    arcpy.SelectLayerByAttribute_management(mains_Lyr,"NEW_SELECTION", whereclause_NOT)
    arcpy.MakeFeatureLayer_management(mains_Lyr, r"in_memory\NOTmains_Lyr", "","","")
    NOT_mains = r"in_memory\NOTmains_Lyr"

    # For each OTHER pipe, make a barrier at mid point

    not_mains_cur =arcpy.SearchCursor(NOT_mains)
        for NOT_main in not_mains_cur:
        ID2 = int(NOT_main.getValue(("OBJECTID")))
        whereclause = ("OBJECTID" + ' = ' + repr(ID2))
        arcpy.SelectLayerByAttribute_management(NOT_mains,"NEW_SELECTION", whereclause)
        arcpy.MakeFeatureLayer_management(NOT_mains, r"in_memory\not_main4point")
        arcpy.FeatureVerticesToPoints_management(r"in_memory\not_main4point",
                     r"in_memory\Not_main_pt", "MID")
        bar = r"in_memory\Not_main_pt"
        arcpy.TraceGeometricNetwork_management(geomnet, r"in_memory\outNet", 
                     traceflag, "FIND_PATH", bar,"", "", "", "", "NO_TRACE_ENDS", 
                     "TRACE_INDETERMINATE_FLOW", "", "","AS_IS", "", "", "", "AS_IS")
        for layer in arcpy.mapping.Layer(r"in_memory\outNet"):
            if str(layer)== "Water_Mains":
                calcstr = "["+ str(fieldname)+"]"  +r"+" + str(weight)
                arcpy.CalculateField_management(layer, fieldname, calcstr, "VB", "")
    end_timeloop1 = datetime.datetime.now()
    print('Duration: {}'.format(end_timeloop1 - start_timeloop1))

Edit: in the above, this appears to be what really adds time to each step:

for layer in arcpy.mapping.Layer(r"in_memory\outNet"):
        if str(layer)== "Water_Mains":
            calcstr = "["+ str(fieldname)+"]"  +r"+" + str(weight)
            arcpy.CalculateField_management(layer, fieldname, calcstr, "VB", "")

I hypothesize that even though it is only adding +1 to the traced records, after a while, it's adding +1 to existing values of 100,000+ storing and calulating with progressively larger numbers may lead to more writes and time? the field it's calculating to is formatted as long.

9
  • 1
    Without seeing the code, it's difficult to guess where inefficiencies may lie. Can you restart your process in the middle? I always make sure I have an if os.path.exists('stop.now'): break in any long-running job, allowing it to end gracefully.
    – Vince
    Commented Apr 16, 2020 at 1:08
  • 3
    arcpy.SearchCursor is deprecated because it's so inefficient. Nesting search cursors is something to avoid; all the more so with old-style cursors. You don't appear to be deleting intermediate products, and altering the NOT_mains selection set inside the NOT_mains cursor may produce random results.
    – Vince
    Commented Apr 16, 2020 at 3:55
  • 2
    Dont just paste code without describing the task you are trying to solve.
    – Bera
    Commented Apr 16, 2020 at 5:37
  • 1
    You hypothesize its slowing down in a certain chunk of code. The only real way to confirm this is to start putting time statements after every tool/function call. I count 10 tools plus 2 cursors. The previous comments have suggested some ways to possibly speed it up, but if you want to know where the problem is, you're going to need to take a scalpel to your workflow and find (which I assume will be) the one tool that's running into a problem.
    – KHibma
    Commented Apr 16, 2020 at 14:49
  • 3
    I would convert your SearchCursors to the da.SearchCursor as @Vince suggest, you will get an immediate performance boost. Also he suggests nesting loops within loops can slow things down. Re-think your logic and try to pass through searchcursors once. So if this was me I would create a dictionary of barrier points and grab the point from that which will be blistering fast compared to the logic you have now.
    – Hornbydd
    Commented Apr 16, 2020 at 19:00

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.