Split millions of points in shp with 100'000 points each [closed]

Question

I havie over 8 million points in a shapefile. Now I would like to split these 8 mio points into parts of 100'000 points each, in order to facilitate further calculations with these points.

In the end I need a table containing the r,g,b,nir values of every pixel of an image, at the location of these points AND within a specific radius around these points (buffer).

I intended to use the "sample" tool to get this table. But unfortunately it seems that sample can't handle millions of points (gave me an empty table after 20 hours of calculating when I tried to sample 2 million of my 8 million points (worked perfectly fine for 10'000 points)). This is why I'm looking for a method to automatically iterate my 8 mio points and split them into smaller parts, over which I then can iterate in further steps to do the buffering, rasterizing and sampling.

The smaller shapefiles should still contain the attributes: x,y,Radii and NewID.

As I'm new to Python, I have no idea how this could be done, but I seek a Python solution if possible.

Answer 1

Code below will split a shapefile into separate shapefiles depending on a specified chunk size. Do you want shapefile outputs? I have not tried it on millions of Points.

import arcpy,os
input_shape=r"C:\infolder\points.shp"
outpath=r"C:\outfolder"
chunksize=100000
fields_to_keep=['x','y','Radii', 'NewID']

#Add all points to a list
pointlist=[]
with arcpy.da.SearchCursor(input_shape,fields_to_keep+['SHAPE@']) as scursor:
    for row in scursor:
        pointlist.append(row)

#Create a generator to split list of points into chunks of specified size (last chunk will contain whats left)
def chunks(l, n):
    for i in xrange(0, len(l), n):
        yield l[i:i + n]
x=chunks(pointlist,chunksize)

#For each chunk create a shapefile and insert the points using the insertcursor:
counter=1
fieldobjects=[f for f in arcpy.ListFields(input_shape) if f.name in fields_to_keep]
for y in x:
    outname='Pointset_{0}.shp'.format(counter)
    arcpy.CreateFeatureclass_management(out_path=outpath, out_name=outname, geometry_type='POINT',spatial_reference=arcpy.Describe(input_shape).spatialReference)
    for field in fieldobjects:
        arcpy.AddField_management(in_table=os.path.join(outpath,outname), field_name=field.name,field_type=field.type, field_length=field.length)
    cursor = arcpy.da.InsertCursor(os.path.join(outpath,outname),fields_to_keep+['SHAPE@'])
    for point in y:
        cursor.insertRow(point)
    del cursor
    counter+=1