Questions tagged [big-data]
A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.
72
questions
0
votes
0
answers
31
views
Measuring the motorable distance between two points by open Source technology
I'm trying to generate a solution to get the Shortest motorable distance between two points by open source GIS technology ?
Our Input layer: Road layer for the AOI (Include flow direction and ...
- 1
1
vote
0
answers
45
views
Zonal statistics for very large point layer with buffer
I have a point layer and I need to calculate, for each point, the area of overlapping raster layer within 500m distance. This could be done fairly quick by generating a 500m-buffered vector layer from ...
- 11
0
votes
0
answers
205
views
Apache Sedona issue: "Cannot convert this array to unsafe format as it's too big"
OS: Ubuntu 20.04
Scala: 2.12.10
Spark/pyspark: 3.0.3
GDAL: 3.6.2
I am trying to work with raster files in apache spark/sedona. The rasterfiles can be downloaded from here. The jp2 file are translated ...
- 79
0
votes
1
answer
107
views
What can be done to avoid Computation timed out for large area reduceRegion?
The reduceRegion below returns the value of the computation after a bit of waiting.
var largeRegionReduced = ee.Image.constant(0).eq(ee.Image.constant(1)).selfMask()
....
- 756
1
vote
4
answers
311
views
Most efficient way to perform rastercalculations with GeoTIFF and/or ErdasImagine files greater than 100gb
I have many GeoTIFFs ranging from 50-300gb and expect I'll have some even larger in the near future. I running a simple subtraction on a pair 77gb 1-band files, but I will also have similar 6-band ...
- 151
3
votes
2
answers
484
views
Compressing large vector data in QGIS
I work on QGIS 3.10.3 and would like to know if it is possible to compress a large vector data (size: 66298KB). The GIS software is still loading every time I touch the mouse.
Is it possible to ...
- 171
1
vote
1
answer
165
views
Iteratively plot large shapefiles using matplotlib
I am working with the National Wetlands Inventory, a set of shapefiles that describe all wetlands in the United States. There are 50 states, and each state has at least one shapefile (some have more ...
- 43
0
votes
0
answers
75
views
QGIS count points in polygon by category
This question is following to this answer point count in polygon by category
array_length(overlay_intersects('pointlayername',$id,filter:="fieldcontainingcategory"='nameofcategory'))
This ...
- 353
1
vote
1
answer
232
views
Finding rasters overlapping polygon shapefile
I have 10,000 rasters but I only need 1000 of them but do not know which 1000.
The 1,000 I need to utilize overlap a vector polygon shapefile and another single raster I have (well the value of 1 in ...
3
votes
1
answer
200
views
Exporting large txt file as point shapefile in QGIS
I have a large txt file with 10 million rows and 16 columns and I already have imported the table as a delimited text file in QGIS. In row 7 and row 8, the coordinates are stored (I also allocated x ...
- 83
3
votes
0
answers
312
views
Which Big Data analytics platform has most extensive GIS support? [closed]
I've recently had to deal with datasets that are simply too large to be efficiently processed using PostgreSQL and so I've turned to Google's BigQuery for those tasks. BigQuery, while offering ...
- 3,073
2
votes
2
answers
4k
views
Getting "Non-successful status code 502" error when reverse geocoding of large data in GeoPy
I have a data size of 1.87GB containing 14962000 rows and 9 columns. And I want to apply GeoPy but I got the errors.
My code:
import pandas as pd
from geopy.geocoders import Nominatim
from geopy....
3
votes
1
answer
404
views
What is the most efficient way to create millions of lines using Python from coordinates? [closed]
I am trying to create a flow map and I have 150 million rows of data. I have tried to use QGIS's Virtual Layer (described in Creating virtual layers using Python and export them without using QGIS UI) ...
- 569
0
votes
1
answer
244
views
Load big wfs layer in R
i´m trying to load the layer Mroad_Lden_17 into R with the code below:
proj_folder <- "C:/Users/... "
setwd(proj_folder)
# helper function for querying uba
load_laerm <- function(what)...
- 9
0
votes
1
answer
217
views
Big data and spatial regression: how to make spml more efficient in R?
I am running a spatial panel regression in R using spml from package splm. When I conduct this on about 1/200th of my data, the program executes. But when I try to do so on a third of my data, R uses ...
- 395
1
vote
1
answer
888
views
Google Earth Engine: Error exporting a large Feature Collection to CSV
On GEE, I want to export daily weather variables for 18 years for about 30,000 rectangular polygons. That would be exporting a FeatureCollection to a csv file with ~200 million rows (30000 places * ...
- 111
3
votes
0
answers
181
views
Spatio-temporal clustering Algorithm
I'm working with a dataset with latitude, longitude and date-time. I have millions of points per day and I want to group them using a clustering algorithm.
I read this question Clustering geographic ...
7
votes
1
answer
2k
views
Multi-threaded compression in rasterio?
For writing large raster files (e.g. satellite image GeoTIFFs) to disk, the throughput rate tends to be limited by the compression algorithm rather than by the raw disk IO. (Benchmarks: 1, 2.) This is ...
- 476
5
votes
2
answers
1k
views
Display millions of locations very, very fast
We're building a mapping application for voter data using PostGIS and Leaflet. It has a web interface. Our voter file will have several million entries. In our app you can select a subset of the voter ...
- 297
1
vote
0
answers
54
views
Storing earth observation derived attributes for polygons [closed]
I want to generate a number of vegetation related indices (NDVI, NDTI, etc., min/max/avg/std (pixel values)) from earth observation data for a large number of polygons during the entire growing season,...
- 686
0
votes
1
answer
783
views
Fast road snapping for big data
I hope this question is not too general. I'm dealing with a lot of data from vehicles (1.000.000 GPS points per day) and I want to snap the data to a big city's street network.
The timely order is not ...
- 73
4
votes
1
answer
427
views
Compute pixel area of a large raster in Google Earth Engine
I want to compute a total area of a large raster dataset using .reduceRegion(), but receive the "Earth Engine capacity exceeded" Error. Is it possible to overcome it? For example create a vector grid ...
- 3,533
1
vote
5
answers
3k
views
Is there anything faster than PostGIS ST_Intersects for seeing where features intersect (or something wrong with my approach)?
Basically have 2 tables. 1 is buildings (1809555 rows called oregon). Another is flood zone (164 rows called oregonslr1). Want to see where they intersect to get number of buildings flooded. I am ...
- 13
0
votes
1
answer
166
views
Second-level neighbors of large polygon (neighbors' neighbors)
I have a large-ish set of polygons and am after identifying second-level neighbors of each, that is, the neighbors of the neighbors of each polygon (distinctly, i.e. the 2nd-level neighbors cannot ...
- 655
3
votes
1
answer
793
views
Filtering (lasfilter on) huge LiDAR dataset
My dataset is about 8.3 GB after loaded into R environment by using lidR package. Some points which have return number greater than number of returns need to be removed.
With a small dataset, I ...
- 163
5
votes
1
answer
3k
views
Using Google BigQuery GIS table as a data source to QGIS
Google BigQuery recently added GIS functionality (still in beta), see here.
I have a dataset in BigQuery that I would like to use as a data source in QGIS - much the same way as I would do with a ...
- 53
1
vote
1
answer
2k
views
Extracting mean values in Earth Engine
I am trying to extract the dates and mean NDVI values of a processed Google Earth Engine image collection thru Python. All goes well until I try to use getInfo(), where I always encounter this error:
...
- 31
-1
votes
1
answer
78
views
Merging/combining huge point (layer) data using ArcMap?
i have 2 point feature layer (file GDB), each layer have a lot of point feature (records with depth value), around 80.000.000-100.000.000 records/point. there are overlapping area between those layer (...
- 23
6
votes
1
answer
2k
views
Iterating an Earth Engine calculation (reducer) over every country in the world: a simple example
I'm an EE newbie, and would like to create a detailed 'resource' for the following challenge.
The problem: What is the best way to iterate a simple calculation (e.g. a Reducer, or a simple user-...
- 138
0
votes
1
answer
2k
views
Creating climate maps in QGIS? [closed]
I have millions of points having temperature and humidity information covering the Middle East and North Africa (MENA).
How can I create animated heatmaps/hotspot maps of change over time? The data ...
- 1,103
3
votes
4
answers
814
views
Calculate length of huge vector lines dataset
I have a MultiLineString vector layer from which I would like to calculate the length.
I have tried to calculate the length with the Field Calculator by putting in the expression $length. The ...
- 373
0
votes
1
answer
449
views
Processing Problems with the Difference (QGIS) and Erase (ArcGIS) Tool with big data
I am struggling with processing a huge data set.
I downloaded it from a German geodata site (https://download.geofabrik.de/europe/germany.html). I downloaded the ZIP of Bavaria and clipped the eight ...
2
votes
2
answers
1k
views
Dice a polygon with over 21 million vertices in ArcMap?
Based on posts like this I suspect I'm bumping up against issues when trying to work with a feature class that contains features with a single very complicated polygon. These errors include 000426, ...
- 341
0
votes
0
answers
96
views
Why does myRaster[1] <- 5 takes a very long time for a large raster?
I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
- 756
1
vote
0
answers
451
views
How to getValues for raster with many values?
I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
- 756
2
votes
0
answers
273
views
create neighborhood list in R with large dataset
I want to create a weight matrix based on distance. My code for the moment looks as follows and functions for a smaller sample of the data. However, with the large dataset (569424 individuals in 24077 ...
- 21
1
vote
0
answers
177
views
How can I limit a Hive spatial binning query to a specific map extent?
I'm trying to do a HQL query based off a map extent. I cant get this to work, what am I doing wrong? I'm using the taxi_demo in the ESRI example.
SELECT ST_Bin(0.001, ST_Point(dropoff_longitude,...
- 11
1
vote
0
answers
106
views
Point data grouping based on location + huge dataset in QGIS
I need community help to solve my puzzle.
I have one huge dataset of points that represent places where a set of trucks stopped. Points go to the database if the reported speed of a truck was 0. The ...
- 111
0
votes
0
answers
73
views
Transferring data from file geodatabase (.gdb) to cassandra?
I want to store my spatial data in cassandra.
I want to import data from an existing file geodatabase(.gdb) which has a table containing boundaries of polygons and a table containing point data.
I ...
- 623
3
votes
1
answer
871
views
Exporting Multiple Rasters to a Table: working with large dataset
I have 20 rasters that cover the same geographic extent, are in the same projection, have the same cell size etc. Instead of a stack of rasters, I want a table where the columns are the different ...
user87619
3
votes
1
answer
1k
views
read and write geotiffs bigger than memory?
The geotiff format is a versatile and easy to understand file format. The only limitation is when I try to do raster calculations I prefer to use GDAL (over ArcPy) which comes with the limitation that ...
- 3,245
3
votes
2
answers
2k
views
Converting coordinate and raster values of each pixel to csv of large raster using Python?
There should be a simple way to convert a large raster to a csv or dataframe by reading the raster in as chunks, converting raster chunk to table of just three columns (x, y, and values), and ...
- 339
2
votes
1
answer
243
views
Feature class to feature class on huge data - iterate loop in ArcGIS pro?
I have a very large dataset containing a very large area. I have two files; one that is area specific shapefile and a standalone table that I have joined to the shapefile. Earlier, I have run feature ...
- 61
0
votes
0
answers
463
views
Finding nearest points to POI with big data at Yelp?
Businesses often need to find nearest points to Points of Interest (POI).
For example, use Yelp to find restaurants within 10 mile distance. As a normal GIS practice, we solve this problem by using ...
- 291
4
votes
0
answers
1k
views
How to store large time series of raster images
I've got a large time-series of rasters, derived from satellite images. Each raster is about 2000 x 2000 pixels, and I've got about 10,000 individual images. I'm trying to work out what the best way ...
- 3,976
1
vote
2
answers
488
views
How to split a large point vector layer in QGIS?
I need to split my shapefiles into much smaller pieces so that I can upload them onto the Postgis server. Shapefiles have DBF files with 14 Go (1,000,000 rows) and I need them to be under 2 Go.
I've ...
- 2,278
3
votes
0
answers
647
views
Doing spatial regression using big data when Spatial weight matrix is too large for R?
I am currently doing an analysis on a quite huge data set - 250,000 polygons. I have figured out the general Moran's I for a linear model, but when I was trying to do a spatial lag or spatial error ...
- 31
1
vote
2
answers
2k
views
Displaying big data set on Google Maps? [closed]
I have a pretty large polygon shapefile (almost 1.5 million buildings) that I want to display in Google Maps. As the KML would be far too big, I have to create tiles, right?
I don't want to host my ...
- 123
-1
votes
1
answer
169
views
Best program or tool for visualising big data sets really fast 10-20 GB data if the users don't know squat about GIS? [closed]
I have between 10-20 GB various GIS-data that there is a need to set up och and visualise for various users who want to be able to pan, zoom and switch between various layers and groups of layers ...
- 7
1
vote
1
answer
167
views
Save millions of points in the server side
I have asked a similar question about millions of points a few days ago:Display millions of points in the browser.
It focused on how to display these data in the client(browser), however I think it ...
- 953