arcpy.Select_analysis choosing only a fraction of the record

Question

I'm trying to divide up a feature class into a number of smaller sets. I have been using Select_analysis with objectid to divide up the featureclass so far but know that this is not the best method.

So for example, currently I have something like:

query = "objectid > 0 +  AND objectid <= 1000"

arcpy.Select_analysis(r"C:\Users\CHOK\Desktop\mulithreading\multiprocessing.gdb\Contours",r"C:\Users\CHOK\Desktop\mulithreading\multiprocessing.gdb\Contours_Mereged",query_statement)

What I want to know is if I can use the number of rows in the feature class to generate the query rather than relying on the objectid? So say I get the number of rows of the feature class using arcpy.GetCount_management(featureclass) and it outputs 1.9 million, is there a way I can divide that 1.9 million into 10 parts based on the count rather than objectid?

Answer 1

I'm not aware of any way to do what you're asking directly. But a work around might be...

Can you add a new field to the feature class? If so, add an integer field, and use the field calculator to add sequential numbers to this field for each feature. One way to do this is explained here: https://support.esri.com/en/technical-article/000011137

You should modify this code to reset to 1 each time it reaches the number of datasets required. Eg,

sets=10
rec=0 
def autoIncrement(): 
   global rec
   global sets
   pStart = 1  
   pInterval = 1 
   if (rec == 0 or rec > sets):  
       rec = pStart  
   else:  
       rec += pInterval  
   return rec

Then you can use Split By Attributes to split into multiple feature classes based on the value of this field (which would be 1-10 for 10 sets). See: http://desktop.arcgis.com/en/arcmap/latest/tools/analysis-toolbox/split-by-attributes.htm

Of course, if it is a dynamic set of data, you would have to recalculate the field each time you wanted to do the same operation again.

Note that the code for the field calculator above would not have the consecutive records kept together (ie, not the first 10% of records, but 1 record in every 10). If you need the consecutive records kept together, this could be done with a tweak to the above code.

Answer 2

Creating and populating new field could significantly slow things down.

This is why I tested 2 alternatives below on 150k long point dataset. They use similar approach - splitting the list of OBJECTIDs into equal size chunks:

import arcpy, time
from arcpy import env
env.overwriteOutput = True

infc="many_points"
OIDs=[row[0] for row in arcpy.da.TableToNumPyArray(infc,"objectid")]
n=len(OIDs)/10
t0 = time.time()
N=65
for i in xrange(0, len(OIDs), n):
    chunk=OIDs[i:i + n]
    q='OBJECTID >=%i AND OBJECTID <=%i' %(chunk[0],chunk[-1])
    arcpy.Select_analysis(infc, "C:/scratch/scratch.gdb/%s"%chr(N), q)
    N+=1
arcpy.AddMessage("Seconds %i using query" %int(time.time()-t0))

t0 = time.time()
mxd = arcpy.mapping.MapDocument("CURRENT")
lyr = arcpy.mapping.ListLayers(mxd,infc)[0]
N=65
for i in xrange(0, len(OIDs), n):
    chunk=OIDs[i:i + n]
    lyr.setSelectionSet ("NEW",chunk)
    arcpy.CopyFeatures_management(lyr, "C:/scratch/scratch.gdb/%s"%chr(N))
    N+=1
arcpy.AddMessage("Seconds %i using layer selection" %int(time.time()-t0))

OUTPUT:

Seconds 57 using query
Seconds 34 using using layer selection

It seems using setSelectionSet method on a layer coupled with Copy features works much faster than Select with query.

Note: Populating integer field in this sample set took 21 second.