0

I am attempting to create a python script (ArcGIS 10.1) that will identify duplicated records within a points shapefile with a Y or N (possibility of over 5000 records). Similar to this:

xyCombine | dplicate

E836814.148873 N814378.125749 |

E836814.148873 N814378.125749 |

E836815.033548 N814377.614688 |

E836815.033548 N814377.614688 |

E836815.033548 N814377.614688 |

E836818.016542 N814371.411850 |

I wish to process the field xyCombine for duplicates and update another field (dplicate) with a Y or N if it's a duplicate or not. With the desired outcome as (it does not have to be sorted):

xyCombine | dplicate

E836814.148873 N814378.125749 | Y

E836814.148873 N814378.125749 | Y

E836815.033548 N814377.614688 | Y

E836815.033548 N814377.614688 | Y

E836815.033548 N814377.614688 | Y

E836818.016542 N814371.411850 | N

Below is my attempt:

# Process: Searches xyCombine field for any duplicates
duplicateCount = 0
inShapefile = pointsShapefile
fieldName = "xyCombine"
shpFRows = arcpy.UpdateCursor(inShapefile)
shpFRow = shpFRows.next()
fieldList = []
while shpFRow:
    if shpFRow.isNull(fieldName) == False and len(str(shpFRow.getValue(fieldName)).strip()) > 1:
        fieldList.append(shpFRow.getValue(fieldName))
shpFRow = shpFRows.next()
duplicateList = [x for x, y in collections.Counter(fieldList).items() if y > 1]
print duplicateList
selectFile = pointsShapefile
selectFields = ('xyCombine','dupCHK')
shpFRows = arcpy.UpdateCursor(selectFile,selectFields)
shpFRow1 = shpFRows.next()
while shpFRow1:
    if shpFRow1.isNull(fieldName) == False and len(str(shpFRow1.getValue(fieldName)).strip()) > 1:
        for row in duplicateList:
            if shpFRow1.getValue(fieldName) == row:
                duplicate += 1
                row[1] = "Y"
            else:
                row[1] = "N"
            cursor.updateRow(row)
        shpFRow1 = shpFRows.next()
if duplicateCount > 0:
    print ""
    print "*** "+str(duplicate)+" duplicated points. ***"
    print ""

If I do not include:

    row[1] = "Y"
else:
    row[1] = "N"
cursor.updateRow(row)

The script executes properly printing the total amount of duplicates, however does not update the field duplicates with either Y or N values which is important as it will provide a csv error report later down the script.

However when I do include it, I get the following error message:

Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32

[u'E836814.148873 N814378.125749', u'E836815.033548 N814377.614688', u'E836818.016542 N814371.41185']

Traceback (most recent call last): File "C:\ Duplicate Points Check\Python Scripts\DuplicatePointsCheck_TEST1.py", line 458, in DuplicatePointsCheck() File "C:\ Duplicate Points Check\Python Scripts\DuplicatePointsCheck_TEST1.py", line 94, in DuplicatePointsCheck row[1] = "N" TypeError: 'unicode' object does not support item assignment>>>

I understand that there are tools within ArcGIS that will provide possible solutions through field calculator. However I want to strengthen my understanding of Python as I am quite new to Python. I apologise if this question has been posed before, but I have trawled the internet and the only results of my search included locating and removing duplicated records.


After numerous attempts (with the suggestion of @Michael Miles-Stimson), I tried to simply and strip down the code as much as I can and came up with this:

inShapefile = pointsShapefile
inShapefileFields = ('xyCombine','dplicate')
valueList = list()
duplicatePointsCount = 0
with arcpy.da.UpdateCursor(inShapefile,inShapefileFields) as cursor:
    for row in cursor:
        if (valueList.count(row[0]) > 1):
            row[1] = "Y"
            duplicatePointsCount += 1
        else:
            row[1] = "N"
        cursor.updateRow(row)
print ""
print "*** "+str(duplicatePointsCount)+" duplicated points."

However, for some reason I could not get the below part to work, and it continued to write "N" in the dplicate field for all records.

if (valueList.count(row[0]) > 1):
            row[1] = "Y"
            duplicatePointsCount += 1

I then attempted @Emil Brundage's code. And it worked a treat!

  • I think you're getting a little confused with arcpy.UpdateCursor and arcpy.da.UpdateCursor - little change but BIG difference in operation. You're iterating correctly but assigning using .da. cursor styles. It's OK, you can use classic cursor, you don't have to use the new one, but to assign you should use row.setValue(Field,Value) to assign to the field and you also need to del(row) and del(shpFRows) at the end of the script. – Michael Stimson Jan 12 '15 at 4:33
  • Thanks for the heads up, @MichaelMiles-Stimson. I had another crack at it, but it was unsucessful (see above). – gisPlatypus Jan 12 '15 at 9:07
  • Cross-posted at stackoverflow.com/q/27857489/820534 – PolyGeo Jan 12 '15 at 11:01
3

Your code looks a bit too complex for me to decipher, but allow me to present a simplified script that should accomplish your goal. It makes use of an initial SearchCursor to determine duplicates. It does so by populating lists. It populates one list of all values that occur once, and a second list of values that occur more than once. It then initiates an UpdateCursor to populate your Y/N field.

from arcpy import *

inShapefile = pointsShapefile

checkField = "xyCombine"
updateField = "dplicate"

#List of values found once
occursOnce = []
#list of values found twice
occursTwice = []

cursor = da.SearchCursor (inShapefile, [checkField])
for row in cursor:
    #Check value is not null
    if row[0]:
        #If not already found to occur twice, proceed
        if not row[0] in occursTwice:
            #If hasn't occured once yet
            if not row[0] in occursOnce:
                #Add to occurs once list
                occursOnce.append (row[0])
            #If value has already been found once
            else:
                #Add to occurs twice list (duplicates)
                occursTwice.append (row[0])
del cursor

cursor = da.UpdateCursor (inShapefile, [checkField, updateField])
for row in cursor:
    #Check value is not null
    if row[0]:
        #check if value in occursTwice list (i.e. is duplicate)
        if row[0] in occursTwice:
            row[1] = "Y"
        else:
            row[1] = "N"
        cursor.updateRow(row)
del cursor

I hope this helps!

  • Thanks for all your help @Emil Brundage! The code you provided worked a treat! It's a lot simpler than my previous attempt and executed quicker. Cheers again. – gisPlatypus Jan 12 '15 at 9:11
  • Glad to be of help. – Emil Brundage Jan 12 '15 at 9:15
1

You do not state which version of arcmap you are using. If you have an advance version you could try to use the Find Identical tool.

http://resources.arcgis.com/en/help/main/10.1/index.html#//001700000054000000

It can take multiple fields so no need to concatenate text.

  • Hi @Hornbydd, apologies for not stating the version. Currently using ArcInfo / Advanced (10.1) for the situation. Thanks for the link. I wanted to get a better grasp of Python and give it a crack so decided not to employ the Find Identical tool. Also the code could assist others who do not have advanced licenses. – gisPlatypus Jan 12 '15 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.