1

I'm running this chunk of code (below) inside a larger script. This section is supposed to find the time for each boat (mmsi) that is closest before and closest after a single detection time. It first finds all the boat times before the detection time and puts them into a fc, then it does the same with all the boat times after the detection time. It then finds the max and min (respectively) of those two fcs and put them into yet another fc. However, this chunk of code is a) messy and b) taking forever to run.

I know from reading about speeding up python scripts that for-loops and cursors can slow things down. However, because of the way they are all nested in code, I can't figure out how to rewrite this in a simpler, faster way. Any thoughts?

arcpy.env.workspace = r'L:\gathr\indonesia\Sara\Date20160104\Detections_UTM.gdb'
workspace = r'L:\gathr\indonesia\Sara\Date20160104\Detections_UTM.gdb'
Geodatabase = arcpy.ListFeatureClasses()
detection_datalist = []
print Geodatabase
for fc in Geodatabase:
    fc_name = str(fc)
    print fc_name
    detection_name = fc_name
    fc_path = os.path.join(workspace, fc_name)
    cursor = arcpy.SearchCursor(fc_path)
    for row in cursor:
        time1 = row.getValue("Date_Mscan")
        time2 = time1.strftime("%Y-%m-%d %H:%M:%S")
        print time2
        time = datetime.strptime(time2,"%Y-%m-%d %H:%M:%S")
        print time
    del cursor  
    arcpy.env.workspace = r'L:\gathr\indonesia\Sara\Date20160104\fc_processed.gdb'
    Geodatabase = arcpy.ListFeatureClasses()
    for fc in Geodatabase:
        fc_str = str(fc)
        suffix = detection_name
        if fc_str.endswith(suffix):
            where_clause = '"ts_pos_utc_Converted" <= date' + "'%s'" %time
            before = os.path.join(r'L:\gathr\indonesia\Sara\Date20160104\fc_processed.gdb', fc + "_before")
            arcpy.Select_analysis(fc, before, where_clause)

            listdates = []
            rows = arcpy.SearchCursor(before)
            row = rows.next()
            while row:
                value = row.ts_pos_utc_Converted
                print value
                listdates.append(value)
                row = rows.next()
            print listdates
            try:
                maxdate = max(listdates)
            except:
                maxdate = None 
            print maxdate

            after = os.path.join(r'L:\gathr\indonesia\Sara\Date20160104\fc_processed.gdb', fc + "_after")
            where_clause = '"ts_pos_utc_Converted" > date' + "'%s'" %time
            arcpy.Select_analysis(fc, after, where_clause)

            listdates = []
            rows = arcpy.SearchCursor(after)
            row = rows.next()
            while row:
                value = row.ts_pos_utc_Converted
                print value
                listdates.append(value)
                row = rows.next()

            print listdates
            try:
                mindate = min(listdates)
            except:
                mindate = None
            print mindate
            new_fc = os.path.join(r'L:\gathr\indonesia\Sara\Date20160104\fc_processed.gdb', fc + "_final")
            layer = os.path.join(r'L:\gathr\indonesia\Sara\Date20160104\fc_processed.gdb', fc)                               
            arcpy.CopyFeatures_management(layer, new_fc)
            with arcpy.da.UpdateCursor(new_fc,"ts_pos_utc_Converted") as cursor:
                for row in cursor:
                    if row[0] not in [mindate, maxdate]:
                        cursor.deleteRow()

del row, cursor
  • 3
    use list comprehension and the da cursors – ziggy Oct 25 '16 at 16:37
  • Maybe implied by using da.SearchCursor, but you can skip the entire 'Select_analysis/write new feature class' steps by using your where clause in the da.SearchCursor. That should keep you from writing intermediate data to disk, but if you find you need to do so, whenever possible write intermediate data to the in_memory workspace rather than disk. – phloem Oct 26 '16 at 20:50
  • All the advice here already is good. Bear in mind that the actual arcpy tools you're calling will run at their own speed - I have a script about 4x as long as yours that would take ~7 minutes tops if there wasn't a step with a Dissolve loop in it. That loop takes about 45 minutes, even using \in_memory. Arcpy is amazing for making scripting easy, but the tools aren't always fast. – obrl_soil Oct 26 '16 at 22:02
2

Use arcpy.da cursors instead of old ones:

The arcpy.da cursors (arcpy.da.SearchCursor, arcpy.da.UpdateCursor, and arcpy.da.InsertCursor) were introduced with ArcGIS 10.1 to provide significantly faster performance over the previously existing set of cursor functions (arcpy.SearchCursor, arcpy.UpdateCursor, and arcpy.InsertCursor). The original cursors are provided only for continuing backward compatibility.

And as mentioned by @Ziggy you can use list comprehensions:

List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition.

For example in combination with a da.SearchCursor:

datelist = [date for date in arcpy.da.SearchCursor(infc, ['DATEFIELD'])]
  • I've been trying to figure out how to use list comprehensions. So could I use something like this: listdates = [row[0] for row in arcpy.da.SearchCursor(fc, fields)] to accomplish the same thing as: 'listdates = [] fields = "ts_pos_utc_Converted" rows = arcpy.da.SearchCursor(before, fields) listdates = [row[0] for row in rows] for row in rows: value = row[0] print value listdates.append(value)' – confused_coder Oct 28 '16 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.