The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [big-data]

A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

Filter by
Sorted by
Tagged with
20
votes
2answers
2k views

What is the best hack for importing large datasets into PostGIS?

I have to import large Shapefiles (> 1 million records) into PostGIS, and I have been wondering about the best way to do it. In my question I used the word "hack", instead of tool, on purpose because ...
6
votes
2answers
7k views

How to find and delete identical points?

I recently performed Integrate on a merged point feature class in ArcGIS 10.1. I specified "1 meter" XY tolerance for the Integrate command. The resulting point feature appears to be what I want ...
34
votes
2answers
2k views

Streamlining Python Code for Big Data

I have Python code which is designed to take point shapefiles through the following workflow: Merge points Integrate points, such that any points within 1 m of each other become one point Create ...
3
votes
1answer
172 views

Filtering (lasfilter on) huge LiDAR dataset

My dataset is about 8.3 GB after loaded into R environment by using lidR package. Some points which have return number greater than number of returns need to be removed. With a small dataset, I ...
2
votes
0answers
826 views

Large point to Polygon by buffer-join-buffer-dissolve issues

I have a few large datasets (about 10 million points) which I am buffering, then joining, dowgrading and finally buffering and dissolving. This is all in a model which works fine with sample smaller ...
1
vote
0answers
165 views

How to getValues for raster with many values?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...