1

I have a shapefile with more than 6 million points, most of which are identical. I used the built-in tool Delete Identical by using Shape property to do it, which takes almost 10 hours and only complete 20%.

The point shapefile only contains three fields, including ID, x-coordinate and y-coordinate. I want to delete the duplicate records from the existing shapefile based on the condition that the points have same x,y coordinates. I can also save the unique records to another shapefile, depending on which method is much faster.

I am using ArcGIS 10.3.1 Advanced License.

5
  • Do you need to retain the table attributes?
    – HDunn
    Oct 9, 2016 at 15:52
  • I only need the shape property, i.e. x,y coordinates, of the unique point.
    – Tianxin Li
    Oct 9, 2016 at 15:54
  • 1
    What version of ArcGIS? What are the features stored in (shapefile, file gdb, enterprise gdb)? Do you want it saved to new feature class or just removed from existing?
    – Midavalo
    Oct 9, 2016 at 17:03
  • 3
    It will always be faster to write a new 3m row table than to delete 3m rows from a 6m row table. It would even be faster to write to a temp table, then truncate and copy those rows back into the source table.
    – Vince
    Oct 9, 2016 at 19:55
  • Thanks. Does the "temp table" mean to use "in_memory"?
    – Tianxin Li
    Oct 9, 2016 at 20:22

1 Answer 1

6

I found that this script copies unique points into separate originally empty shapefile about 5 times faster, compare to delete identical. I tested it on 200,000 points, with 100,000 of them being a duplicate.

import arcpy, time,os
from arcpy import env
env.overwriteoutput=True
allPoints=r'C:\...\ScratchFolder\POINTs.shp'
outFC=r'C:\...\ScratchFolder\OUTPUT\Block_00000.shp'
curT=arcpy.da.InsertCursor(outFC,"Shape@")

result=arcpy.GetCount_management(allPoints)
nF=int(result.getOutput(0))

aDict={}
with arcpy.da.SearchCursor(allPoints, "SHAPE@XY") as cursor:
  arcpy.SetProgressor("step", "", 0, nF,1)
  for row in cursor:
    arcpy.SetProgressorPosition()
    v=row[0]
    if aDict.has_key(v):continue
    aDict[v]=1
    curT.insertRow((row[0],))

It confirms @Vince point.

Use fastest drive to store input and output, network drives=NO GO. I suggest to run it from mxd or ArcCatalog, you can watch progress and cancel at any time.

UPDATE TO HANDLE TOLERANCE:

import arcpy, time,os
from arcpy import env
env.overwriteoutput=True

def truncate(f, n):
  s = '{}'.format(f)
  i, p, d = s.partition('.')
  return '.'.join([i, (d+'0'*n)[:n]])        

allPoints=r'C:\...Data\ScratchFolder\POINTs.shp'
outFC=r'C:\...Data\ScratchFolder\OUTPUT\Block_00000.shp'
curT=arcpy.da.InsertCursor(outFC,"Shape@")
result=arcpy.GetCount_management(allPoints)
nF=int(result.getOutput(0))
aDict={}

with arcpy.da.SearchCursor(allPoints, "SHAPE@XY") as cursor:
  arcpy.SetProgressor("step", "", 0, nF,1)
  for row in cursor:
    arcpy.SetProgressorPosition()
    x,y=row[0]
    v='%s_%s' %(truncate(x,3),truncate(y,3))
    if aDict.has_key(v):continue
    aDict[v]=1
    curT.insertRow((row[0],))

This will consider points to be the same if their coordinates are identical to 3 decimal places. I badly want to hope that you at least working with projected data

19
  • Thanks. I am testing it now. By the way, how long did it cost you when you run the 200,000 points?
    – Tianxin Li
    Oct 9, 2016 at 21:45
  • 24 seconds compared to about 2 mins with delete identical
    – FelixIP
    Oct 9, 2016 at 22:09
  • As always it took ArcView 3.2 only 5 seconds to accomplish the same task (including delete !). This is why I keep it in my pocket. Not sure how long it will take with your set... ArcGIS performance on geometries is very-very far from perfect
    – FelixIP
    Oct 9, 2016 at 22:23
  • 1
    If it works for you please mark it as solved and ask separate question on defining projection
    – FelixIP
    Oct 9, 2016 at 22:55
  • 1
    a set would work better than a dictionary in this case
    – jbalk
    Oct 10, 2016 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.