2

I havie over 8 million points in a shapefile. Now I would like to split these 8 mio points into parts of 100'000 points each, in order to facilitate further calculations with these points.

In the end I need a table containing the r,g,b,nir values of every pixel of an image, at the location of these points AND within a specific radius around these points (buffer).

I intended to use the "sample" tool to get this table. But unfortunately it seems that sample can't handle millions of points (gave me an empty table after 20 hours of calculating when I tried to sample 2 million of my 8 million points (worked perfectly fine for 10'000 points)). This is why I'm looking for a method to automatically iterate my 8 mio points and split them into smaller parts, over which I then can iterate in further steps to do the buffering, rasterizing and sampling.

The smaller shapefiles should still contain the attributes: x,y,Radii and NewID.

As I'm new to Python, I have no idea how this could be done, but I seek a Python solution if possible.

15
  • Which GIS software are you using? – BERA May 9 '17 at 9:10
  • Sorry, completely forgot to mention that - ArcGIS 10.2 – Sibir May 9 '17 at 9:14
  • 1
    Have a look at ArcGIS' "focal statistics" tool. It sounds like this will get you the results you ultimately need for statistics within a moving window (I.e. Your buffer distance), then extract those values back to your original points with the "extract values to points" tool. – Jae May 9 '17 at 9:46
  • Combining Python and GDAL (OGR) you can use SQL to select a range of points and export those as the smaller parts. – Tins May 9 '17 at 9:53
  • 1
    If you use "focal statistics" with a circular window (of your defined radius), you can extract stats for max, min, mean, whatever you choose, within a "buffer" around each cell location. You can then use "extract multivalues to points" to extract raster values for the original r,g,b, nir as well as the summary raster values derived from the focal statistics tool, which would extract summary stats based on your defined radius, only at the original point locations. If your points reside within a geodatabase, you should have no concerns working with millions of points. – Jae May 9 '17 at 13:26
4

Code below will split a shapefile into separate shapefiles depending on a specified chunk size. Do you want shapefile outputs? I have not tried it on millions of Points.

import arcpy,os
input_shape=r"C:\infolder\points.shp"
outpath=r"C:\outfolder"
chunksize=100000
fields_to_keep=['x','y','Radii', 'NewID']

#Add all points to a list
pointlist=[]
with arcpy.da.SearchCursor(input_shape,fields_to_keep+['SHAPE@']) as scursor:
    for row in scursor:
        pointlist.append(row)

#Create a generator to split list of points into chunks of specified size (last chunk will contain whats left)
def chunks(l, n):
    for i in xrange(0, len(l), n):
        yield l[i:i + n]
x=chunks(pointlist,chunksize)

#For each chunk create a shapefile and insert the points using the insertcursor:
counter=1
fieldobjects=[f for f in arcpy.ListFields(input_shape) if f.name in fields_to_keep]
for y in x:
    outname='Pointset_{0}.shp'.format(counter)
    arcpy.CreateFeatureclass_management(out_path=outpath, out_name=outname, geometry_type='POINT',spatial_reference=arcpy.Describe(input_shape).spatialReference)
    for field in fieldobjects:
        arcpy.AddField_management(in_table=os.path.join(outpath,outname), field_name=field.name,field_type=field.type, field_length=field.length)
    cursor = arcpy.da.InsertCursor(os.path.join(outpath,outname),fields_to_keep+['SHAPE@'])
    for point in y:
        cursor.insertRow(point)
    del cursor
    counter+=1
3
  • Thank you very much for your help! I've edited my question and added the field names which I need for the further processing. – Sibir May 9 '17 at 12:40
  • I have changed the code so that the fields will remain – BERA May 9 '17 at 13:08
  • Unfortunately this error message always occurs (my computer has 32 GB RAM and I never had memory issues so far...): Runtime error Traceback (most recent call last): File "<string>", line 11, in <module> MemoryError Any ideas of how to solve this? – Sibir May 9 '17 at 20:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.