What is the most efficient method to delete points?

Question

I have a script that takes a point featureclass (FC) through the following workflow:

1. Select points that lie within individual polygons
2. From the selected points within each polygon, randomly subset X% of points
3. Merge the selected points from each polygon feature sub-selection
4. Erase the original points with the merged subset and write to a new FC

I am using arcpy.Erase_analysis() to do the point deletion.

The subset of data I am testing this script on has ~35000 point features. The point subset that I would like to use as the erase feature has ~3000 point features. It takes approximately 3-5 minutes to complete processing. Seeing that I will be scaling this up to a dataset containing ~30 million points, is there a more efficient method to delete points than by using the arcpy.Erase_analysis() method? Perhaps, programmatically deleting points in an editing session? Or, a method that does not involve writing a new point FC?

Answer 1

You can't efficiently delete large numbers of features from a large table, period.

You'd be much better off just marking them as deleted (available varchar(1), values 'Y' or 'N') and adding an attribute constraint to your selection logic (or using a view). Building a parallel table to house a foreign key and available field is an adequate solution, though the join performance is likely to drag down processing on larger tables.

Be sure to use "Y" and "N" values, rather than nulls, because changing a 1-character value to another 1-character value isn't going to have the same fragmentation impact as changing a null to a string.

Answer 2

If you do not NEED to delete the points, then you could just use the Select By Attribute tool, select the points that you don't need, switch selection and create a new feature class. That will help your data maintain its original integrity also.