2

I have a field which is supposed to hold a unique identifier, however due to issues with Attribute Assistant I have multiple features which have the same unique ID. I want to write a script which will select duplicates and remove the originals from the selection. The duplicated field is FID, the originals will have a lower OID (also, there can be more than one duplicate).

I have used SelectLayersByAttribute_management to select features where there are more than one feature with the same FID:

arcpy.SelectLayerByAttribute_management("MyLayer", "NEW_SELECTION", "FID In (SELECT FID FROM MyFeatureClass GROUP BY FID HAVING Count(*)>1))

This selects both originals and duplicates, but how do I now remove the features which have the original FID value?

The originals will have a lower OID value:

OID   FID
 1    x101    ORIGINAL
 2    x101    DUPLICATE
 3    x102    ORIGINAL
 4    x102    DUPLICATE
 5    x102    DUPLICATE

I'm using ArcMap 10.2.1

  • Does the solution have to be a script? What you're trying to do can be accomplished with a summarize, then join, then select by attributes, then switch selection. – Rayner May 30 '17 at 13:43
  • Yes I'm afraid so. I can fix the problem manually pretty easily but I'm trying to automate the process. This is just a small section of what will be a larger script, it's just what I'm currently stuck on. – TeddyTedTed May 30 '17 at 15:25
  • Welcome to GIS SE! As a new user please take the tour to learn about our focused Q&A format. What version of ArcGIS Desktop are you using? Also I have removed your "PS" about Attribute Assistant. If you have a another question, please ask it separately. – Midavalo May 30 '17 at 18:06
  • 1
    @user3596332: Which reply matched what you were looking for? Please accept the passing answer ; else maybe comment further. – Hélène Jun 12 '17 at 9:02
  • @Hélène Hi, I'm awaiting a further comment from Midavalo who has been very helpful so far. – TeddyTedTed Jun 19 '17 at 10:58
3

Something like the following should loop through the features in your layer using a Search Cursor, and keep track of when a value already exists, adding subsequent feature OIDs to a list to select.

lyr = arcpy.mapping.Layer("MyLayer")
selOID = list()
existingValue = list()

with arcpy.da.SearchCursor(lyr, ['OID@', 'FID']) as cursor:
    for row in cursor:
        if row[1] in existingValue:
            selOID.append(row[0])
        else:
            existingValue.append(row[1])

lyr.setSelectionSet('NEW', selOID)

For more info on lyr.setSelectionSet() see the setSelectionSet() method for the Layer class


As the setSelectionSet() isn't available until 10.3, you can use this method to select the list of ObjectIDs. Replace the last line in the code sample above with the last two lines below (the rest is the same code)

lyr = arcpy.mapping.Layer("MyLayer")
selOID = list()
existingValue = list()

with arcpy.da.SearchCursor(lyr, ['OID@', 'FID']) as cursor:
    for row in cursor:
        if row[1] in existingValue:
            selOID.append(row[0])
        else:
            existingValue.append(row[1])

query = 'OBJECTID IN (' + ','.join(map(str, selOID)) + ')' 
arcpy.SelectLayerByAttribute_management(lyr, "NEW_SELECTION", query)
  • 1
    Small caveat: setSelectionSet() doesn't seem to be available prior to 10.3. – Richard Morgan May 30 '17 at 19:12
  • @RichardMorgan You may be right - a quick look in the 10.2 docs didn't find it. I have requested the OP tell us the version. The mention of Attribute Assistant makes me think it's not too old, although the AA does still work on a minimum release of 10.2.1, so if that is the case my answer may not work – Midavalo May 30 '17 at 22:23
  • 10.2.1 it is I'm afraid – TeddyTedTed May 31 '17 at 8:20
  • @user3596332 I have updated my answer to include an option that should work pre 10.3 – Midavalo May 31 '17 at 14:02
  • @Midavalo This is so close but not quite there. It returns the correct number of records however I need it to specifically avoid the record with the lowest OID. I think this excludes the first record it comes across but that may not necessarily be the one with the lowest OID. Thank you for getting it this far. – TeddyTedTed May 31 '17 at 15:41
1

I'd explore Delete Identical (Data Management Toolbox), which removes all but one of the identical records found. As a ArcToolbox tool, you can use it in scripts, or directly from the GUI.

To list the duplicates without deleting them, the Find Identical tool is also available, using the "Output only duplicated records" parameter. However, this outputs a new table, rather than creating a selection in your original data.

  • I don't want to delete either of the records. I just want to deselect the original so that I can update the others' attributes. – TeddyTedTed May 30 '17 at 15:24
1

This Select By Attribute, Where Clause seems to work, only selecting the DUPLICATE rows when the same FID has ORIGINAL also.

Status = 'DUPLICATE' and Fid in (SELECT FID FROM dups where Status = 'ORIGINAL') and  Fid in (SELECT FID FROM dups where Status = 'DUPLICATE')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.