1

I saw some examples with for loop to find duplicates.

I had 1.5 million records in feature class, what would be fastest way to find duplicates and delete that particular rows.

Based on three fields, I'm trying to find duplicates[ID,S_NAME,TIME] using ArcPy Python 2.7.

EDIT-1: Delete identical getting error with sde connection-

arcpy.DeleteIdentical_management (r"C:\FolderConn\DEV\master.sde\DatasetName\FeatureClassName", ["ID","S_NAME","TIME"])`

2 Answers 2

3

You can use Delete Identical for this task.

Deletes records in a feature class or table which have identical values in a list of fields. If the field Shape is selected, feature geometries are compared

1
  • will the delete identical works with sde connection like-arcpy.DeleteIdentical_management (r"C:\FolderConn\DEV\master.sde\DatasetName\FeatureClassName", ["ID","S_NAME","TIME"])-@Aaron
    – Mr_gemini
    Feb 5, 2020 at 22:02
0

If you really wanted to use Python you could do it with an update cursor:

fields = ('ID', 'S_NAME', 'TIME')

seen = set()
with arcpy.da.UpdateCursor(fc, fields) as cursor:
    for row in cursor:
        row = tuple(row)  # lists aren't hashable so we can't add them to a set
        if row in seen:
            cursor.deleteRow()
        else:
            seen.add(row)

This will keep the first occurrence, sort the cursor in reverse order if you want the last.

Whether or not that's quicker than Delete Identical for 1.5M rows I'm not sure, I'll leave that up to you to test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.