0

I have a large point data set that has irregular geometry (random sampling) that I want to bin in 2D XY space. I've attached a sketch below to illustrate what I mean. Apologies for hand drawing, hope no one minds.

So far I've been using the fishnet option in ArcGIS+QGIS and SAGA tool polygon attribute to points. However I have reached the workable file size limit for my shapefiles (>2GB). When attempting to use feature classes ArcGIS is incredibly slow at performing the required task, so I have been mostly focusing on SAGA for its speed.

It's been suggested to me by a colleague that I could use GMT's blockmean, blockmedian, blockmode programs. However I have limited experience in this but would be willing to learn IF it is the right direction

I can see the maths of all this as a matrix operation, but the problem I am having is with the large data size. My point files are >700mb in just XY coordinates (billions of points).

I am not looking for interpolation, just looking to assign the points a 'bin' reference in XY space.

enter image description here

3

R would be a good place to do this, if you're willing to copy a little code. I think in general you are looking for a raster function, essentially. If you need it as a polygon, you can convert the raster to polygon when you're finished.

library(rgdal)
library(raster)
library(plyr)

# load your points
points <- readOGR("points.shp", "points")

# create a blank raster that has the same extent as the points layer
r <- raster(ext=extent(points))

# set the resolution of the raster to the size of the cell you want
res(r) <- 1000

# use the count function from the plyr package
re <- extract(r, y, fun='count')

R can handle the 700MB points layer, it shouldn't be too slow.

|improve this answer|||||
  • Going to give this a go in the next 2-3 hours, will get back on how things work out later tonight or tomorrow (UK time) – 8556732 Jul 26 '16 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.