1

I often see code that is comparing 2 shapefiles where there needs to be 2 with statements in the same loop. In my head I always thought both with statements should be placed first before starting looping the rows in each cursor like:

with arcpy.da.SearchCursor(fc2, field4) as sCursor1:
    with arcpy.da.SearchCursor(fc1, field5) as sCursor2:
        for row1 in sCursor1:
            for row2 in sCursor2:
                #do stuff

However I have seen code that looks like this:

with arcpy.da.SearchCursor(fc2, field4) as sCursor1:
    for row1 in sCursor1:
        with arcpy.da.SearchCursor(fc2, field5) as sCursor2:            
            for row2 in sCursor2:
                #do stuff

Purely just looking at this I would assume that the second version reopens the cursor for every row in sCursor1, and therefore the first method would be better. I have read http://preshing.com/20110920/the-python-with-statement-by-example/ and this suggests that :

The above with statement will automatically close the file after the nested block of code.

Therefore it looks like I might be correct, however is there a preferred method of the 2 or another way that would be better, such as creating the cursors first and the using the del statements at the end?

Some timings on small, med, large data would be good if possible.

  • 6
    I would try to avoid this by loading the contents of the second cursor into a dictionary. That can return values to the first cursor far quicker than repeatedly opening the second cursor. – PolyGeo Jul 6 '16 at 23:37
  • @PolyGeo loading the values into a dictionary is a much more effective way of doing a comparison provided you have the memory (I've done it that way many times before), if not because the 2nd dataset is too large, then you need to get a bit more clever with your 2nd where clause than just polling each and every record arbitrarily or your process would take forever to finish. The question is, however, can you do it that way and the answer is 'yes, and there's ways to performance tune the operation'. – Michael Stimson Jul 7 '16 at 0:20
5

Logically, they do effectively the same thing. For every row in one data set, loop through every line in one dataset (presumably until some criteria is satisfied, then break). Without a break on an if statement, if both datasets are 10,000 rows long, you would have to iterate through 100,000,000 rows.

EDIT:

However, the With With For For methodology doesn't work in practice because, as the documentation says, an arcpy.da.SearchCursor:

Returns an iterator of tuples.

And

Search cursors also support with statements to reset iteration and aid in removal of locks.

This means when you create a cursor you can only iterate through it once, meaning you have to delete it and re-create it for multiple iterations through the same dataset. Iterators raise a StopIteration if it reaches the end of the iterable object (http://anandology.com/python-practice-book/iterators.html) (http://pro.arcgis.com/en/pro-app/arcpy/data-access/searchcursor-class.htm)

Sometimes a problem will require this type of workflow, but the best way to optimize it is to reduce the number of times you iterate through your data. For example, if you use dict comprehension to build a dictionary of values from one table, you can call on the components of that dictionary when iterating through the other table, so you would only have to iterate through 20,000 rows to accomplish your goal, like so:

import arcpy
fc1 = "feature1"
fc2 = "feature2"
data_dict = {row[0]: row[1] for row in arcpy.da.SearchCursor(fc1, ["ID", "FIELD1"])}
with arcpy.da.SearchCursor(fc2, ["ID", "FIELD2"]) as cursor:
    for row in cursor:
        relevant_data = data_dict[row[0]]
        #Do something else

If you absolutely have to iterate through one dataset for every row in another dataset, you can use multiprocessing to significantly reduce the time. On an 8 core machine iterating through a 10,000 row dataset for every row in a 10,000 row dataset, you would accomplish the same amount of work as if you were iterating through only 12.5 million rows instead of 100 million. For example,

import arcpy
import multiprocessing
from _functools import partial

def working_function(fc1, row):
    fc1row = row
    with arcpy.da.SearchCursor(fc1, ["field1"]) as cursor:
        for row in cursor:
            ##Do some work/matching/etc

if __name__ == "__main__":
    fc1 = "feature1"
    fc2 = "feature2"
    the_data = [row for row in arcpy.da.SearchCursor(fc2, ["field2"])]
    partial_function = partial(working_function, fc1) 
    ''' creates a new function where the first parameter is always fc1'''
    pool = multiprocessing.Pool()
    pool.map(partial_function, the_data)
    pool.close()
    pool.join()

EDIT: The above example doesn't really do anything, but if you store the values you want to update or use for later in a python dictionary, you could use an Update Cursor to modify the fc1 row data after multiprocessing based on whatever criteria you specify. I've used multiprocessing on file geodatabase feature classes before with great success.

You can also copy your feature classes to the "in_memory" workspace, which stores your features in RAM and significantly speeds up the read/write time.

In_memory workspace: http://pro.arcgis.com/en/pro-app/help/analysis/geoprocessing/modelbuilder/the-in-memory-workspace.htm

Multiprocessing (https://docs.python.org/2/library/multiprocessing.html)

EDIT: You can also combine the two methodologies to further optimize, like this:

import arcpy
import multiprocessing
from _functools import partial

def working_function(data_dict, row):
    fc1row = row
    wanted_value = data_dict[row[0]]
    return wanted_value
if __name__ == "__main__":
    fc1 = "feature1"
    fc2 = "feature2"
    data_dict = {row[0]: row[1] for row in arcpy.da.SearchCursor(fc1, ["ID", "field1"])}
    the_data = [row for row in arcpy.da.SearchCursor(fc2, ["id", "field2"])]
    partial_function = partial(working_function, data_dict) 
    ''' creates a new function where the first parameter is always data_dict'''
    pool = multiprocessing.Pool()
    pool.map(partial_function, the_data)
    pool.close()
    pool.join()
  • That's interesting, Esri objects (ArcObjects) aren't guaranteed to be thread safe so you could assume python objects to be the same, attempts to use multiprocessing usually have to be well conceived and isolated STA threads with a mutex or two thrown in to avoid locking situations. Have you tried multiprocessing like this with other objects? did it work or crash and burn? – Michael Stimson Jul 7 '16 at 0:12
  • I have used this methodology before and it significantly reduced my processing time without any locking issues (on feature classes in the same file geodatabase). If you have an editing task as part of your function (that requires arcpy.da.Editor) you can't use multiprocessing. However, you could use multiprocessing if you were making your edits to a feature class on an SDE. – crld Jul 7 '16 at 0:19
  • I suspect arcpy.da.InsertCursor to be in the same boat as editor. A process can (and does) lock itself out, you may not see much of it in python but in ArcObjects I do. I suppose for a search cursor or succinct datasets it could work, what about temporary files? attempts I've made to use multiprocessing in python crash because two (or more) threads try to create the same temp file.. if this avoids self locking/clashing then this is a positive gem of information and the most interesting thing I've read in a while. – Michael Stimson Jul 7 '16 at 0:26
  • 2
    ok I've double checked a script I wrote where I used multiprocessing to speed up an update cursor. Looks like I'm only using search cursors and storing the information to update in a dictionary (with objectid as key) and after multiprocessing finishes, perform a single update cursor to update those values in the dict. Sorry for the confusion. I'm also storing the feature classes for multiprocessing in the in_memory workspace – crld Jul 7 '16 at 15:52
  • I just a quick test, and the first method doesn't work! The loop breaks after one iteration. Can you change your answer a little so I can accept it because everything else is great. – JamesLeversha Jul 7 '16 at 23:07
1

In response to @crld I have done some tests, Im not sure if im doing the timings right but, the first method doesnt actually work, it breaks out of the loop after 1 iteration

import arcpy
import itertools

fc = r'D:\randomstuff\test2.gdb\Point0' #5146 points
fc1 = r'D:\randomstuff\test2.gdb\points2' #484 points

def printCount(start,count):
    end = time.time()
    total = end - start
    print 'Count: ' + str(count), 'Time: ' + str(total)

print 'With with for for'
for x in range(3):
    start = time.time()
    count = 0
    with arcpy.da.SearchCursor(fc,'OID@') as cursor:
        with arcpy.da.SearchCursor(fc1,'OID@') as cursor1:
            for row in cursor:
                for row1 in cursor1:
                    count +=1
    printCount(start,count)

print 'With for with for'
for x in range(3):
    start = time.time()
    count = 0
    with arcpy.da.SearchCursor(fc,'OID@') as cursor:
        for row in cursor:
            with arcpy.da.SearchCursor(fc1,'OID@') as cursor1:
                for row1 in cursor1:
                    count +=1
    printCount(start,count)

print 'Storing one of the cursors to memory'
cursor1 =[row for row in arcpy.da.SearchCursor(fc1,'OID@')]
for x in range(3):
    start = time.time()
    count = 0
    with arcpy.da.SearchCursor(fc,'OID@') as cursor:
        for row in cursor:
            for row1 in cursor1:
                count +=1
    printCount(start,count)

#Not in the question but worthy to note
print 'Comparing one dataset against itself using combinations'

for x in range(3):
    start = time.time()
    count = 0
    with arcpy.da.SearchCursor(fc,'OID@') as cursor:
        for row,row1 in itertools.combinations(cursor,2):
                count +=1
    printCount(start,count)

These were the timings

With with for for
Count: 484 Time: 1.875
Count: 484 Time: 0.039999961853
Count: 484 Time: 0.0409998893738

With for with for
Count: 2490664 Time: 53.0940001011
Count: 2490664 Time: 53.2479999065
Count: 2490664 Time: 53.2400000095

Storing one of the cursors to memory
Count: 2490664 Time: 0.193000078201
Count: 2490664 Time: 0.191999912262
Count: 2490664 Time: 0.194000005722

#Not really in the question but worthy to note
Comparing one dataset against itself using combinations
Count: 13238085 Time: 1.39700007439
Count: 13238085 Time: 1.39999985695
Count: 13238085 Time: 1.39700007439
  • Weird! At the for row1 in cursor1 put the next line count+=1 on the same line after the :,, then below it add continue and see if that works. – crld Jul 9 '16 at 11:01
  • Also, the times for storing one of the cursors to memory section are kind of amazing compared to before. – crld Jul 9 '16 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.