Questions tagged [big-data]

A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

12 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
426 views

Doing spatial regression using big data when Spatial weight matrix is too large for R?

I am currently doing an analysis on a quite huge data set - 250,000 polygons. I have figured out the general Moran's I for a linear model, but when I was trying to do a spatial lag or spatial error ...
2
votes
0answers
103 views

create neighborhood list in R with large dataset

I want to create a weight matrix based on distance. My code for the moment looks as follows and functions for a smaller sample of the data. However, with the large dataset (569424 individuals in 24077 ...
2
votes
0answers
709 views

How to store large time series of raster images

I've got a large time-series of rasters, derived from satellite images. Each raster is about 2000 x 2000 pixels, and I've got about 10,000 individual images. I'm trying to work out what the best way ...
2
votes
0answers
826 views

Large point to Polygon by buffer-join-buffer-dissolve issues

I have a few large datasets (about 10 million points) which I am buffering, then joining, dowgrading and finally buffering and dissolving. This is all in a model which works fine with sample smaller ...
1
vote
0answers
160 views

How to getValues for raster with many values?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
1
vote
0answers
120 views

How can I limit a Hive spatial binning query to a specific map extent?

I'm trying to do a HQL query based off a map extent. I cant get this to work, what am I doing wrong? I'm using the taxi_demo in the ESRI example. SELECT ST_Bin(0.001, ST_Point(dropoff_longitude,...
1
vote
0answers
62 views

Point data grouping based on location + huge dataset in QGIS

I need community help to solve my puzzle. I have one huge dataset of points that represent places where a set of trucks stopped. Points go to the database if the reported speed of a truck was 0. The ...
1
vote
0answers
287 views

Finding nearest points to POI with big data at Yelp?

Businesses often need to find nearest points to Points of Interest (POI). For example, use Yelp to find restaurants within 10 mile distance. As a normal GIS practice, we solve this problem by using ...
1
vote
0answers
39 views

Sending commands into FGDB

I have a large .gdb of all blocks in the United States, available from Census. I would like to create a point shape file of all census tract centroids in the US. I have QGIS with FileGDB enabled, as ...
1
vote
0answers
311 views

Big data rastering using ArcGIS Platform?

I have the daunting task doing raster analysis on a record set that runs into quite a few millions, and worse still is that the data is in Hive. So far I have managed to use record set of 2 million ...
0
votes
0answers
52 views

Why does myRaster[1] <- 5 takes a very long time for a large raster?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
0
votes
0answers
46 views

Transferring data from file geodatabase (.gdb) to cassandra?

I want to store my spatial data in cassandra. I want to import data from an existing file geodatabase(.gdb) which has a table containing boundaries of polygons and a table containing point data. I ...