2

I am trying to calculate a field that marks duplicates and sequentially numbers them. I am using the code below that I found here: Generating sequential numbering for duplicate values in a field sorted by ObjectID.

It works, but I get a duplicate starting value before it begins sequentially numbering unless I only run the field calculator on the selected values. When I calculate after manually selecting the duplicate rows and running it on only those values, it works fine.

Here's the code:

prevFieldValue = ''
counter = 0
def GetDuplicateCounter(myFieldValue):
  global prevFieldValue
  global counter
  if myFieldValue == prevFieldValue:
    counter += 1
  else:
    counter = 1
    prevFieldValue = myFieldValue
  return counter

Results running it on all roads (nothing selected):

calculated without selection

Results running it after manually selecting the duplicated fields (not too efficient!).

calculated on selected records

  • I just noticed that the records do not have sequential object ids and start renumbering from zero where they pick up again. – indybrown Sep 9 '13 at 20:32
  • 1
    What sort of database is your feature class/table stored in? Field Calculator does not respect the attribute table sort state, but rather operates on rows in the order they are returned from the underlying DBMS, which may be non-deterministic. I would suggest doing this with a sorted update cursor, or one of the built-in find duplicate geoprocessing tools. – blah238 Sep 9 '13 at 21:30
  • Thanks! It is in a file geodatabase. I will start research those options... – indybrown Sep 9 '13 at 21:45
3

Here's how I solved the problem:

Pre-Logic Script Code:
d = {} # creates an empty dictionary the first time

def find_duplicates(val):
    d[val] = d.setdefault(val, -1) + 1
    return d[val]

Expression: dup =
find_duplicates(!Unique_ID!)

Unfortunately, I can't remember where I found the answer but if I do, I will post the credit.

2

If you don't mind working with a 2nd data set, you can find unique values by using a python dictionary:

values = {}
with da.SearchCursor(<ds>, ["unique_field", "id_field"]) as rows:
   for row in rows:
      if values.has_key(row[1]):
         values[row[1]] += 1
      else:
         values[row[1]] = 1

or you could turn your feature class into a numpy array, calculate the count on a column, then convert it back to a feature class.

1

within field calculator I use this code to find my dupes, assigning a '1' value to the actual dupes.

uList = []
def isDupe(inValue):
  if inValue in uList:
    return 1
  else:
    uList.append(inValue)
    return 0

isDupe(!field_name!)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.