Questions tagged [big-data]

A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

Filter by
Sorted by
Tagged with
0
votes
0answers
21 views

Big data and spatial regression: how to make spml more efficient in R?

I am running a spatial panel regression in R using spml from package splm. When I conduct this on about 1/200th of my data, the program executes. But when I try to do so on a third of my data, R uses ...
0
votes
0answers
41 views

GEE - Reprojecting before exporting causes “User memory exceeded” error

I am studying ecosystem distributions in Norway using satellite data and figured that the Google Earth Engine is a very viable tool for this purpose. I am well aware that processing data with a high ...
1
vote
1answer
210 views

Google Earth Engine: Error exporting a large Feature Collection to CSV

On GEE, I want to export daily weather variables for 18 years for about 30,000 rectangular polygons. That would be exporting a FeatureCollection to a csv file with ~200 million rows (30000 places * ...
1
vote
0answers
40 views

Spatio-temporal clustering Algorithm

I'm working with a dataset with latitude, longitude and date-time. I have millions of points per day and I want to group them using a clustering algorithm. I read this question Clustering geographic ...
0
votes
0answers
24 views

Publish large data volume(PB) of point cloud data using portal

Make the las point cloud into SLPK, and then publish through portal, However, when dealing with the point cloud data with a large amount of data in this way, the process of publishing and loading is ...
6
votes
1answer
401 views

Multi-threaded compression in rasterio?

For writing large raster files (e.g. satellite image GeoTIFFs) to disk, the throughput rate tends to be limited by the compression algorithm rather than by the raw disk IO. (Benchmarks: 1, 2.) This is ...
4
votes
2answers
336 views

Display millions of locations very, very fast

We're building a mapping application for voter data using PostGIS and Leaflet. It has a web interface. Our voter file will have several million entries. In our app you can select a subset of the voter ...
1
vote
0answers
45 views

Storing earth observation derived attributes for polygons [closed]

I want to generate a number of vegetation related indices (NDVI, NDTI, etc., min/max/avg/std (pixel values)) from earth observation data for a large number of polygons during the entire growing season,...
0
votes
1answer
295 views

Fast road snapping for big data

I hope this question is not too general. I'm dealing with a lot of data from vehicles (1.000.000 GPS points per day) and I want to snap the data to a big city's street network. The timely order is not ...
3
votes
1answer
279 views

Compute pixel area of a large raster in Google Earth Engine

I want to compute a total area of a large raster dataset using .reduceRegion(), but receive the "Earth Engine capacity exceeded" Error. Is it possible to overcome it? For example create a vector grid ...
1
vote
5answers
601 views

Is there anything faster than PostGIS ST_Intersects for seeing where features intersect (or something wrong with my approach)?

Basically have 2 tables. 1 is buildings (1809555 rows called oregon). Another is flood zone (164 rows called oregonslr1). Want to see where they intersect to get number of buildings flooded. I am ...
0
votes
1answer
54 views

Second-level neighbors of large polygon (neighbors' neighbors)

I have a large-ish set of polygons and am after identifying second-level neighbors of each, that is, the neighbors of the neighbors of each polygon (distinctly, i.e. the 2nd-level neighbors cannot ...
3
votes
1answer
409 views

Filtering (lasfilter on) huge LiDAR dataset

My dataset is about 8.3 GB after loaded into R environment by using lidR package. Some points which have return number greater than number of returns need to be removed. With a small dataset, I ...
4
votes
1answer
1k views

Using Google BigQuery GIS table as a data source to QGIS

Google BigQuery recently added GIS functionality (still in beta), see here. I have a dataset in BigQuery that I would like to use as a data source in QGIS - much the same way as I would do with a ...
1
vote
1answer
1k views

Extracting mean values in Earth Engine

I am trying to extract the dates and mean NDVI values of a processed Google Earth Engine image collection thru Python. All goes well until I try to use getInfo(), where I always encounter this error: ...
-1
votes
1answer
45 views

Merging/combining huge point (layer) data using ArcMap?

i have 2 point feature layer (file GDB), each layer have a lot of point feature (records with depth value), around 80.000.000-100.000.000 records/point. there are overlapping area between those layer (...
3
votes
1answer
1k views

Iterating an Earth Engine calculation (reducer) over every country in the world: a simple example

I'm an EE newbie, and would like to create a detailed 'resource' for the following challenge. The problem: What is the best way to iterate a simple calculation (e.g. a Reducer, or a simple user-...
0
votes
1answer
770 views

Creating climate maps in QGIS? [closed]

I have millions of points having temperature and humidity information covering the Middle East and North Africa (MENA). How can I create animated heatmaps/hotspot maps of change over time? The data ...
3
votes
4answers
564 views

Calculate length of huge vector lines dataset

I have a MultiLineString vector layer from which I would like to calculate the length. I have tried to calculate the length with the Field Calculator by putting in the expression $length. The ...
0
votes
1answer
271 views

Processing Problems with the Difference (QGIS) and Erase (ArcGIS) Tool with big data

I am struggling with processing a huge data set. I downloaded it from a German geodata site (https://download.geofabrik.de/europe/germany.html). I downloaded the ZIP of Bavaria and clipped the eight ...
2
votes
2answers
860 views

Dice a polygon with over 21 million vertices in ArcMap?

Based on posts like this I suspect I'm bumping up against issues when trying to work with a feature class that contains features with a single very complicated polygon. These errors include 000426, ...
0
votes
0answers
74 views

Why does myRaster[1] <- 5 takes a very long time for a large raster?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
1
vote
0answers
240 views

How to getValues for raster with many values?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
2
votes
0answers
137 views

create neighborhood list in R with large dataset

I want to create a weight matrix based on distance. My code for the moment looks as follows and functions for a smaller sample of the data. However, with the large dataset (569424 individuals in 24077 ...
1
vote
0answers
140 views

How can I limit a Hive spatial binning query to a specific map extent?

I'm trying to do a HQL query based off a map extent. I cant get this to work, what am I doing wrong? I'm using the taxi_demo in the ESRI example. SELECT ST_Bin(0.001, ST_Point(dropoff_longitude,...
1
vote
0answers
70 views

Point data grouping based on location + huge dataset in QGIS

I need community help to solve my puzzle. I have one huge dataset of points that represent places where a set of trucks stopped. Points go to the database if the reported speed of a truck was 0. The ...
0
votes
0answers
50 views

Transferring data from file geodatabase (.gdb) to cassandra?

I want to store my spatial data in cassandra. I want to import data from an existing file geodatabase(.gdb) which has a table containing boundaries of polygons and a table containing point data. I ...
3
votes
1answer
652 views

Exporting Multiple Rasters to a Table: working with large dataset

I have 20 rasters that cover the same geographic extent, are in the same projection, have the same cell size etc. Instead of a stack of rasters, I want a table where the columns are the different ...
2
votes
1answer
691 views

read and write geotiffs bigger than memory?

The geotiff format is a versatile and easy to understand file format. The only limitation is when I try to do raster calculations I prefer to use GDAL (over ArcPy) which comes with the limitation that ...
2
votes
2answers
1k views

Converting coordinate and raster values of each pixel to csv of large raster using Python?

There should be a simple way to convert a large raster to a csv or dataframe by reading the raster in as chunks, converting raster chunk to table of just three columns (x, y, and values), and ...
2
votes
1answer
177 views

Feature class to feature class on huge data - iterate loop in ArcGIS pro?

I have a very large dataset containing a very large area. I have two files; one that is area specific shapefile and a standalone table that I have joined to the shapefile. Earlier, I have run feature ...
0
votes
0answers
375 views

Finding nearest points to POI with big data at Yelp?

Businesses often need to find nearest points to Points of Interest (POI). For example, use Yelp to find restaurants within 10 mile distance. As a normal GIS practice, we solve this problem by using ...
3
votes
0answers
873 views

How to store large time series of raster images

I've got a large time-series of rasters, derived from satellite images. Each raster is about 2000 x 2000 pixels, and I've got about 10,000 individual images. I'm trying to work out what the best way ...
1
vote
2answers
336 views

How to split a large point vector layer in QGIS?

I need to split my shapefiles into much smaller pieces so that I can upload them onto the Postgis server. Shapefiles have DBF files with 14 Go (1,000,000 rows) and I need them to be under 2 Go. I've ...
3
votes
0answers
496 views

Doing spatial regression using big data when Spatial weight matrix is too large for R?

I am currently doing an analysis on a quite huge data set - 250,000 polygons. I have figured out the general Moran's I for a linear model, but when I was trying to do a spatial lag or spatial error ...
1
vote
2answers
2k views

Displaying big data set on Google Maps? [closed]

I have a pretty large polygon shapefile (almost 1.5 million buildings) that I want to display in Google Maps. As the KML would be far too big, I have to create tiles, right? I don't want to host my ...
-1
votes
1answer
143 views

Best program or tool for visualising big data sets really fast 10-20 GB data if the users don't know squat about GIS? [closed]

I have between 10-20 GB various GIS-data that there is a need to set up och and visualise for various users who want to be able to pan, zoom and switch between various layers and groups of layers ...
1
vote
1answer
120 views

Save millions of points in the server side

I have asked a similar question about millions of points a few days ago:Display millions of points in the browser. It focused on how to display these data in the client(browser), however I think it ...
2
votes
1answer
10k views

Any tool to view large tiff images?

I'm looking for some application to view some large TIFF images and large mosaics consist of tiff images. At the moment I have VLIV for viewing large tiff files. And tools based on HD view: HD View ...
7
votes
2answers
194 views

Remote geoprocessing solutions?

I have a relatively old laptop (i5 8Gb) but it is fine for most of my GIS needs. Infrequently I would like to run processes which requires larger amounts of RAM (16G+) but I cannot justify forking out ...
1
vote
0answers
40 views

Sending commands into FGDB

I have a large .gdb of all blocks in the United States, available from Census. I would like to create a point shape file of all census tract centroids in the US. I have QGIS with FileGDB enabled, as ...
23
votes
2answers
3k views

What is the best hack for importing large datasets into PostGIS?

I have to import large Shapefiles (> 1 million records) into PostGIS, and I have been wondering about the best way to do it. In my question I used the word "hack", instead of tool, on purpose because ...
5
votes
2answers
1k views

Moving PostGIS database for Social Media to Cassandra? [closed]

I currently have a large and growing database of geo-social media. I'm using Postgres with postgis, and so far it's been quite good. My use case thus far has been to query the database for all posts ...
1
vote
0answers
319 views

Big data rastering using ArcGIS Platform?

I have the daunting task doing raster analysis on a record set that runs into quite a few millions, and worse still is that the data is in Hive. So far I have managed to use record set of 2 million ...
2
votes
2answers
320 views

What simple solutions are available for sharing big data with end users?

I have approximately 1700 raster tiles showing classified land cover with a total size of approximately 300GB. I am looking for a simple solution to privately serve these data to non-technical end ...
1
vote
1answer
216 views

Arcpy scripts fail on big datasets - anyone has experienced similar behavior?

Here a question that is not extremely specific but maybe someone has had similar experiences... I have a few Arcpy scripts that work perfectly when used with small datasets but that silently fail ...
4
votes
1answer
257 views

Limits to torque and cartodb

Bit of a theoretical one this, but I was wondering if anyone had tried to use CartoDB and Torque with any big(ish) data. What I am talking about is having the location of 75,000 people at 1 minute ...
4
votes
1answer
381 views

Should slow refresh rate be expected while viewing 1000 points out of 100 million using ArcSDE/Oracle?

Is it normal to experience very slow refresh rate in ArcMap with 1000 points shown out of 100 million using ArcSDE/Oracle? I’am testing a setup with ArcSDE/Oracle database to contain geographical ...
9
votes
3answers
9k views

Best way to clip a large raster ECW file ?

I am trying to clip a large ECW (details below) but the raster file is too big to be processed entirely. Some details of the ECW below Driver: ECW/ERDAS Compressed Wavelets (SDK 5.0) file size: ...
2
votes
2answers
2k views

What is faster for geoprocessing (intersect) big shapefiles?

I have 200 Mb shapefiles of polylines downloaded from OSM. I've been working with QGIS 2.0 in Ubuntu 64 Bit 8 Gb RAM to make some geoprocessing tasks and it's taking 2 days for processing. Once I've ...