arcpy.da.UpdateCursor "iteration not started" when using pairwise iteration through cursor

Question

Without getting too much into the context of this workflow, I'm trying to find whether a stream segment is flowing in the right direction. If the "FROM_NODE" or the "TO_NODE" values of two adjacent stream segments are equal, then one of the segments isn't digitized in the correct direction, and that record is tagged with an error code (i.e. 6).

I am trying to do this by iterating through a file geodatabase table using da.UpdateCursor, iterating two cursor rows at a time.

I'm using a function called "pairwise" to iterate through two rows at a time, which I got from the Python 2.7 itertools documentation here:

https://docs.python.org/2/library/itertools.html#recipes

When stepping through the code, everything works fine until I get to one of the cursor.updateRow(row1) lines, then I get the following error:

StopIteration: iteration not started

Here's the code:

# iterates over two items at a time (recipe from python documentation)
def pairwise(iterable):
    a, b = tee(iterable)
    next(b, None)
    return zip(a,b)

# Find flow direction errors
def flow_direction(tmp_network_tbl):
    arcpy.AddMessage("... flow direction")
    fields = ["ReachID", "UpstreamID", "FROM_NODE", "TO_NODE", "ERROR_CODE"]
    with arcpy.da.UpdateCursor(tmp_network_tbl, fields) as cursor:
        for row1, row2 in pairwise(cursor):
            if (row1[2] == row2[2]): # upstream has same from_node as reach
                row1[4] = 6
                cursor.updateRow(row1)
            elif (row1[3] == row2[3]): # reach has same to_node as upstream
                row1[4] = 6
                cursor.updateRow(row1)
            elif (row1[4] != 0):
                pass

I'm assuming that the fact that the cursor is being run through the pairwise function is causing the problem, so I suppose my whole effort to try iterating two rows at a time isn't a good idea... Or is there just something simple I'm missing?

Answer 1

I consider this approach a very bad idea. Cursors cannot move backwards. The update using row1 will occur after the cursor is no longer accessing row1, since it has moved on to row2. This should never work.

I would load the entire feature set to a dictionary with the ReachID as the key and the rest as the value in a list. Then process an update cursor on the same feature class and find the ReachID in the dictionary that matches the upstream ID of the current feature (if any). Then you can run your comparisons of the current row with the dictionary record list for that UpstreamID. Then you can set the Error code on the current cursor record. It will be more flexible, since it does not need to make the incorrect assumption that rows will return with a particular sort order (which is not guaranteed by a da cursor) and will process much faster than a sorted cursor anyway.

All paired cursor operations should be done this way. See my Blog on Turbo Charging Data Manipulation with Python Cursors and Dictionaries

The basic code should be something like:

from time import strftime  

print "Start script: " + strftime("%Y-%m-%d %H:%M:%S")  

import arcpy  

sourceFC = r"C:\Path\SourceFeatureClass"  

sourceFieldsList = ["ReachID", "OID@", "FROM_NODE", "TO_NODE"]  

# Use list comprehension to build a dictionary from a da SearchCursor  
valueDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, sourceFieldsList)}  

updateFC = r"C:\Path\SourceFeatureClass"  

updateFieldsList = ["UpstreamID", "OID@", "FROM_NODE", "TO_NODE", "ERROR_CODE"]  

with arcpy.da.UpdateCursor(updateFC, updateFieldsList) as updateRows:  
    for updateRow in updateRows:  
        # store the Join value of the row being updated in a keyValue variable  
        keyValue = updateRow[0]  
         # verify that the keyValue is in the Dictionary  
        if keyValue in valueDict:
            if updateRow[1] != valueDict[keyValue][0]): # Make sure rows are different
                # check nodes.  
                if (updateRow[2] == valueDict[keyValue][1]): # upstream has same from_node as reach
                    updateRow[4] = 6
                    updateRows.updateRow(updateRow)
                elif (updateRow[3] == valueDict[keyValue][2]): # reach has same to_node as upstream
                    updateRow[4] = 6
                    updateRows.updateRow(updateRow)

del valueDict  

print "Finished script: " + strftime("%Y-%m-%d %H:%M:%S")

I believe this code also is better than the pairwise approach since it is possible (probable) that multiple downstream branches will share the same upstream segment. The pairwise approach will only compare one of the downstream branches to any given upstream segment, while my code will compare all of the downstream branches to a given upstream segment no matter how many downstream branches that segment may have. Also, this code will work without requiring the records in the underlying database to be sorted in any particular order.