1

I have a shapefile with more than 6 million points, most of which are identical. I used the built-in tool Delete Identical by using Shape property to do it, which takes almost 10 hours and only complete 20%.

The point shapefile only contains three fields, including ID, x-coordinate and y-coordinate. I want to delete the duplicate records from the existing shapefile based on the condition that the points have same x,y coordinates. I can also save the unique records to another shapefile, depending on which method is much faster.

I am using ArcGIS 10.3.1 Advanced License.

  • Do you need to retain the table attributes? – HDunn Oct 9 '16 at 15:52
  • I only need the shape property, i.e. x,y coordinates, of the unique point. – Tianxin Li Oct 9 '16 at 15:54
  • 1
    What version of ArcGIS? What are the features stored in (shapefile, file gdb, enterprise gdb)? Do you want it saved to new feature class or just removed from existing? – Midavalo Oct 9 '16 at 17:03
  • 3
    It will always be faster to write a new 3m row table than to delete 3m rows from a 6m row table. It would even be faster to write to a temp table, then truncate and copy those rows back into the source table. – Vince Oct 9 '16 at 19:55
  • Thanks. Does the "temp table" mean to use "in_memory"? – Tianxin Li Oct 9 '16 at 20:22
6

I found that this script copies unique points into separate originally empty shapefile about 5 times faster, compare to delete identical. I tested it on 200,000 points, with 100,000 of them being a duplicate.

import arcpy, time,os
from arcpy import env
env.overwriteoutput=True
allPoints=r'C:\...\ScratchFolder\POINTs.shp'
outFC=r'C:\...\ScratchFolder\OUTPUT\Block_00000.shp'
curT=arcpy.da.InsertCursor(outFC,"Shape@")

result=arcpy.GetCount_management(allPoints)
nF=int(result.getOutput(0))

aDict={}
with arcpy.da.SearchCursor(allPoints, "SHAPE@XY") as cursor:
  arcpy.SetProgressor("step", "", 0, nF,1)
  for row in cursor:
    arcpy.SetProgressorPosition()
    v=row[0]
    if aDict.has_key(v):continue
    aDict[v]=1
    curT.insertRow((row[0],))

It confirms @Vince point.

Use fastest drive to store input and output, network drives=NO GO. I suggest to run it from mxd or ArcCatalog, you can watch progress and cancel at any time.

UPDATE TO HANDLE TOLERANCE:

import arcpy, time,os
from arcpy import env
env.overwriteoutput=True

def truncate(f, n):
  s = '{}'.format(f)
  i, p, d = s.partition('.')
  return '.'.join([i, (d+'0'*n)[:n]])        

allPoints=r'C:\...Data\ScratchFolder\POINTs.shp'
outFC=r'C:\...Data\ScratchFolder\OUTPUT\Block_00000.shp'
curT=arcpy.da.InsertCursor(outFC,"Shape@")
result=arcpy.GetCount_management(allPoints)
nF=int(result.getOutput(0))
aDict={}

with arcpy.da.SearchCursor(allPoints, "SHAPE@XY") as cursor:
  arcpy.SetProgressor("step", "", 0, nF,1)
  for row in cursor:
    arcpy.SetProgressorPosition()
    x,y=row[0]
    v='%s_%s' %(truncate(x,3),truncate(y,3))
    if aDict.has_key(v):continue
    aDict[v]=1
    curT.insertRow((row[0],))

This will consider points to be the same if their coordinates are identical to 3 decimal places. I badly want to hope that you at least working with projected data

  • Thanks. I am testing it now. By the way, how long did it cost you when you run the 200,000 points? – Tianxin Li Oct 9 '16 at 21:45
  • 24 seconds compared to about 2 mins with delete identical – FelixIP Oct 9 '16 at 22:09
  • As always it took ArcView 3.2 only 5 seconds to accomplish the same task (including delete !). This is why I keep it in my pocket. Not sure how long it will take with your set... ArcGIS performance on geometries is very-very far from perfect – FelixIP Oct 9 '16 at 22:23
  • 1
    If it works for you please mark it as solved and ask separate question on defining projection – FelixIP Oct 9 '16 at 22:55
  • 1
    a set would work better than a dictionary in this case – jbalk Oct 10 '16 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.