3

Without getting too much into the context of this workflow, I'm trying to find whether a stream segment is flowing in the right direction. If the "FROM_NODE" or the "TO_NODE" values of two adjacent stream segments are equal, then one of the segments isn't digitized in the correct direction, and that record is tagged with an error code (i.e. 6).

I am trying to do this by iterating through a file geodatabase table using da.UpdateCursor, iterating two cursor rows at a time.

I'm using a function called "pairwise" to iterate through two rows at a time, which I got from the Python 2.7 itertools documentation here:

https://docs.python.org/2/library/itertools.html#recipes

When stepping through the code, everything works fine until I get to one of the cursor.updateRow(row1) lines, then I get the following error:

StopIteration: iteration not started

Here's the code:

# iterates over two items at a time (recipe from python documentation)
def pairwise(iterable):
    a, b = tee(iterable)
    next(b, None)
    return zip(a,b)

# Find flow direction errors
def flow_direction(tmp_network_tbl):
    arcpy.AddMessage("... flow direction")
    fields = ["ReachID", "UpstreamID", "FROM_NODE", "TO_NODE", "ERROR_CODE"]
    with arcpy.da.UpdateCursor(tmp_network_tbl, fields) as cursor:
        for row1, row2 in pairwise(cursor):
            if (row1[2] == row2[2]): # upstream has same from_node as reach
                row1[4] = 6
                cursor.updateRow(row1)
            elif (row1[3] == row2[3]): # reach has same to_node as upstream
                row1[4] = 6
                cursor.updateRow(row1)
            elif (row1[4] != 0):
                pass

I'm assuming that the fact that the cursor is being run through the pairwise function is causing the problem, so I suppose my whole effort to try iterating two rows at a time isn't a good idea... Or is there just something simple I'm missing?

  • I think code snippets should only be posted here once they have had any try/except statements removed, and retested, because they can mask error messages which would otherwise be helpful. As it stands this does not qualify as a code snippet that works up to where you are stuck because it consists only of two functions with no call to your flow_direction function to run it. – PolyGeo Aug 23 '16 at 21:17
  • Although you have been a user of this site for quite some time I notice that you have not yet taken the 2-minute Tour that leads via the help center to advice on question structure at meta.gis.stackexchange.com/questions/3349/… – PolyGeo Aug 23 '16 at 21:23
  • @PolyGeo - I removed the try/except statements. I didn't include the code to call the function because it is embedded within a much larger function. Also, without the input data, the function call wouldn't work anyway. I'm assuming it is better to keep the questions short and to-the-point. – tinyplanet00 Aug 23 '16 at 22:25
  • It is better to keep questions short and to the point but if they cannot be illustrated by a working code snippet then they are likely unsuited to focussed Q&A. The act of extracting a working code snippet rather than simply copy/pasting a function or two often enables the person who is the most familiar with the code (the asker) to spot their own error and/or a better way forward. – PolyGeo Aug 23 '16 at 22:32
5

I consider this approach a very bad idea. Cursors cannot move backwards. The update using row1 will occur after the cursor is no longer accessing row1, since it has moved on to row2. This should never work.

I would load the entire feature set to a dictionary with the ReachID as the key and the rest as the value in a list. Then process an update cursor on the same feature class and find the ReachID in the dictionary that matches the upstream ID of the current feature (if any). Then you can run your comparisons of the current row with the dictionary record list for that UpstreamID. Then you can set the Error code on the current cursor record. It will be more flexible, since it does not need to make the incorrect assumption that rows will return with a particular sort order (which is not guaranteed by a da cursor) and will process much faster than a sorted cursor anyway.

All paired cursor operations should be done this way. See my Blog on Turbo Charging Data Manipulation with Python Cursors and Dictionaries

The basic code should be something like:

from time import strftime  

print "Start script: " + strftime("%Y-%m-%d %H:%M:%S")  

import arcpy  

sourceFC = r"C:\Path\SourceFeatureClass"  

sourceFieldsList = ["ReachID", "OID@", "FROM_NODE", "TO_NODE"]  

# Use list comprehension to build a dictionary from a da SearchCursor  
valueDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, sourceFieldsList)}  

updateFC = r"C:\Path\SourceFeatureClass"  

updateFieldsList = ["UpstreamID", "OID@", "FROM_NODE", "TO_NODE", "ERROR_CODE"]  

with arcpy.da.UpdateCursor(updateFC, updateFieldsList) as updateRows:  
    for updateRow in updateRows:  
        # store the Join value of the row being updated in a keyValue variable  
        keyValue = updateRow[0]  
         # verify that the keyValue is in the Dictionary  
        if keyValue in valueDict:
            if updateRow[1] != valueDict[keyValue][0]): # Make sure rows are different
                # check nodes.  
                if (updateRow[2] == valueDict[keyValue][1]): # upstream has same from_node as reach
                    updateRow[4] = 6
                    updateRows.updateRow(updateRow)
                elif (updateRow[3] == valueDict[keyValue][2]): # reach has same to_node as upstream
                    updateRow[4] = 6
                    updateRows.updateRow(updateRow)

del valueDict  

print "Finished script: " + strftime("%Y-%m-%d %H:%M:%S")

I believe this code also is better than the pairwise approach since it is possible (probable) that multiple downstream branches will share the same upstream segment. The pairwise approach will only compare one of the downstream branches to any given upstream segment, while my code will compare all of the downstream branches to a given upstream segment no matter how many downstream branches that segment may have. Also, this code will work without requiring the records in the underlying database to be sorted in any particular order.

  • Thanks @Richard. Your code doesn't quite do what I want... I'm actually using a single geodatabase table as the input (not two separate feature classes), and the OID@ field is being compared to the ReachID, which results in the wrong records being tagged as an error. But the approach is great, and I adapted it to work. Also, you're right that it's a huge boost in speed! – tinyplanet00 Aug 23 '16 at 23:30
  • I did want to say that the code assumes that ReachID is unique for every segment. If it is not unique the code would have be modified to create values with a list of lists for each ReachID segment and iterate those lists for the comparison. – Richard Fairhurst Aug 23 '16 at 23:32
  • I just forgot to point the source and the update to the same feature class. I modified the code to work for a single feature class. The UpstreamID should be comparing to the ReachID, not the OID. The indexes stagger since the key is not in the list of the dictionary, but is in the field value list. So updateRow[1] != valueDict[keyValue][0] etc. are the correct comparisons. However, you perhaps want to flip the UpstreamID to the dictionary and the ReachID to the update cursor to mark the upstream as the error and not the downstream. – Richard Fairhurst Aug 23 '16 at 23:37
  • Yes, the ReachID is unique (as is the UpstreamID). I wasn't complaining about your solution... it's great. I see what you're saying about the pairwise approach... but the time you get to the second row, you can't go back and update the first row (which is why the UpdateCursor wasn't working). – tinyplanet00 Aug 23 '16 at 23:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.