How to find and delete identical points?

Question

I recently performed Integrate on a merged point feature class in ArcGIS 10.1. I specified "1 meter" XY tolerance for the Integrate command. The resulting point feature appears to be what I want except points that were integrated are now stacked on top of each other, which will be problematic in future processing steps. The attributes for my dataset consist of an OID, Shape, x, y, z. Where x and y = UTM coordinates and z = buffer distance. The command Delete Identical (Data Management) does not appear to work for my dataset (i.e based on Fields: OID, Shape, x, y, z and XY tolerance = 1m) as the output still has stacked points. I am also working with big data--the integrate output has 37 million points.

There is a related thread, titled Remove spatially duplicate features using ModelBuilder, although I would like to open the question up to python solutions too.3

What is the best way to find and delete (spatially) duplicate points?

Is there a better solution to integrating such that the output does not contain overlapping points?

Answer 1

Use just the Shape field as the compare fields in the Delete Identical tool which:

Deletes records in a feature class or table which have identical values in a list of fields. If the field Shape is selected, feature geometries are compared.

Answer 2

WARNING! "Find Identical" and "Delete Identical" require an "Advanced" license.

I was burned by this yesterday because I delivered a python tool to a client who only has a Basic license. Fortunately I was able to use a simple point comparison instead (it took less code than calling the tool!) and the resulting tool is about 3x faster now.

You can iterate over a file using arcpy.da.SearchCursor and specify "SHAPE" as one of the fields. When you do that you can directly access the x,y location as a tuple. I read the points and put them into a dictionary of lists, using the shape as the dictionary index and putting the object id's into the lists.

If you have an advanced license and you use Find Identical, you need to look carefully at the results table -- it creates a row for each shape from the source, with a "group" column. So EVERY row will be found in the output, not just rows with duplicates. It's up to you to read the table and figure out which "groups" have more than one entry, which is how duplicates are flagged.

I think it's kind of a sketchy tool - hard to use and like most ESRI tools, you end up having to write output to a new file that you then have to read and interpret that file.

It does have the advantage of accepting a tolerance as a parameter. For the tool I wrote that did not matter.