Skip to main content

Questions tagged [big-data]

A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

Filter by
Sorted by
Tagged with
0 votes
1 answer
75 views

QGIS and use of big data

I have a database with more than 110 million records (points), in the geopackage format it is 11 gigabytes and in the Geoparquet format it is 2 gigabytes. Viewing takes more than 5 minutes. I need ...
Denilson's user avatar
  • 695
2 votes
1 answer
150 views

Using QGIS to obtain lat/long coordinates from large data with addresses

In QGIS, I have a requirement to obtain lat/long coordinates for an US address, and I'm looking for low-cost alternatives to the Google Maps API. I have a (very) large dataset; right now I'm looking ...
Mel Riffe's user avatar
1 vote
0 answers
66 views

Zonal statistics for very large point layer with buffer

I have a point layer and I need to calculate, for each point, the area of overlapping raster layer within 500m distance. This could be done fairly quick by generating a 500m-buffered vector layer from ...
Janne's user avatar
  • 11
0 votes
1 answer
247 views

What can be done to avoid Computation timed out for large area reduceRegion?

The reduceRegion below returns the value of the computation after a bit of waiting. var largeRegionReduced = ee.Image.constant(0).eq(ee.Image.constant(1)).selfMask() ....
Andrei's user avatar
  • 766
1 vote
4 answers
513 views

Most efficient way to perform rastercalculations with GeoTIFF and/or ErdasImagine files greater than 100gb

I have many GeoTIFFs ranging from 50-300gb and expect I'll have some even larger in the near future. I running a simple subtraction on a pair 77gb 1-band files, but I will also have similar 6-band ...
kpierce8's user avatar
  • 151
3 votes
2 answers
831 views

Compressing large vector data in QGIS

I work on QGIS 3.10.3 and would like to know if it is possible to compress a large vector data (size: 66298KB). The GIS software is still loading every time I touch the mouse. Is it possible to ...
GIS Noh's user avatar
  • 321
1 vote
1 answer
210 views

Iteratively plot large shapefiles using matplotlib

I am working with the National Wetlands Inventory, a set of shapefiles that describe all wetlands in the United States. There are 50 states, and each state has at least one shapefile (some have more ...
sdg's user avatar
  • 43
0 votes
0 answers
117 views

QGIS count points in polygon by category

This question is following to this answer point count in polygon by category array_length(overlay_intersects('pointlayername',$id,filter:="fieldcontainingcategory"='nameofcategory')) This ...
Rii Pii's user avatar
  • 464
1 vote
1 answer
246 views

Finding rasters overlapping polygon shapefile

I have 10,000 rasters but I only need 1000 of them but do not know which 1000. The 1,000 I need to utilize overlap a vector polygon shapefile and another single raster I have (well the value of 1 in ...
If you do not know- just GIS's user avatar
3 votes
1 answer
229 views

Exporting large txt file as point shapefile in QGIS

I have a large txt file with 10 million rows and 16 columns and I already have imported the table as a delimited text file in QGIS. In row 7 and row 8, the coordinates are stored (I also allocated x ...
Kevin_S2021's user avatar
3 votes
0 answers
360 views

Which Big Data analytics platform has most extensive GIS support? [closed]

I've recently had to deal with datasets that are simply too large to be efficiently processed using PostgreSQL and so I've turned to Google's BigQuery for those tasks. BigQuery, while offering ...
Encomium's user avatar
  • 3,143
2 votes
2 answers
5k views

Getting "Non-successful status code 502" error when reverse geocoding of large data in GeoPy

I have a data size of 1.87GB containing 14962000 rows and 9 columns. And I want to apply GeoPy but I got the errors. My code: import pandas as pd from geopy.geocoders import Nominatim from geopy....
Ibrahim Mohamed's user avatar
3 votes
1 answer
454 views

What is the most efficient way to create millions of lines using Python from coordinates? [closed]

I am trying to create a flow map and I have 150 million rows of data. I have tried to use QGIS's Virtual Layer (described in Creating virtual layers using Python and export them without using QGIS UI) ...
sjp_1989's user avatar
  • 569
0 votes
1 answer
294 views

Load big wfs layer in R

i´m trying to load the layer Mroad_Lden_17 into R with the code below: proj_folder <- "C:/Users/... " setwd(proj_folder) # helper function for querying uba load_laerm <- function(what)...
lunam's user avatar
  • 9
0 votes
1 answer
330 views

Big data and spatial regression: how to make spml more efficient in R?

I am running a spatial panel regression in R using spml from package splm. When I conduct this on about 1/200th of my data, the program executes. But when I try to do so on a third of my data, R uses ...
C. Ashley's user avatar
  • 405
1 vote
1 answer
1k views

Google Earth Engine: Error exporting a large Feature Collection to CSV

On GEE, I want to export daily weather variables for 18 years for about 30,000 rectangular polygons. That would be exporting a FeatureCollection to a csv file with ~200 million rows (30000 places * ...
vpk's user avatar
  • 111
3 votes
0 answers
214 views

Spatio-temporal clustering Algorithm

I'm working with a dataset with latitude, longitude and date-time. I have millions of points per day and I want to group them using a clustering algorithm. I read this question Clustering geographic ...
Yannick Arthur DOUNGMO's user avatar
7 votes
1 answer
2k views

Multi-threaded compression in rasterio?

For writing large raster files (e.g. satellite image GeoTIFFs) to disk, the throughput rate tends to be limited by the compression algorithm rather than by the raw disk IO. (Benchmarks: 1, 2.) This is ...
benjimin's user avatar
  • 476
5 votes
2 answers
1k views

Display millions of locations very, very fast

We're building a mapping application for voter data using PostGIS and Leaflet. It has a web interface. Our voter file will have several million entries. In our app you can select a subset of the voter ...
ccleve's user avatar
  • 297
1 vote
0 answers
56 views

Storing earth observation derived attributes for polygons [closed]

I want to generate a number of vegetation related indices (NDVI, NDTI, etc., min/max/avg/std (pixel values)) from earth observation data for a large number of polygons during the entire growing season,...
GISinHelsinki's user avatar
0 votes
1 answer
874 views

Fast road snapping for big data

I hope this question is not too general. I'm dealing with a lot of data from vehicles (1.000.000 GPS points per day) and I want to snap the data to a big city's street network. The timely order is not ...
Andi Anderle's user avatar
4 votes
1 answer
474 views

Compute pixel area of a large raster in Google Earth Engine

I want to compute a total area of a large raster dataset using .reduceRegion(), but receive the "Earth Engine capacity exceeded" Error. Is it possible to overcome it? For example create a vector grid ...
Basile's user avatar
  • 3,583
1 vote
5 answers
4k views

Is there anything faster than PostGIS ST_Intersects for seeing where features intersect (or something wrong with my approach)?

Basically have 2 tables. 1 is buildings (1809555 rows called oregon). Another is flood zone (164 rows called oregonslr1). Want to see where they intersect to get number of buildings flooded. I am ...
user10271755's user avatar
0 votes
1 answer
171 views

Second-level neighbors of large polygon (neighbors' neighbors)

I have a large-ish set of polygons and am after identifying second-level neighbors of each, that is, the neighbors of the neighbors of each polygon (distinctly, i.e. the 2nd-level neighbors cannot ...
MichaelChirico's user avatar
3 votes
1 answer
889 views

Filtering (lasfilter on) huge LiDAR dataset

My dataset is about 8.3 GB after loaded into R environment by using lidR package. Some points which have return number greater than number of returns need to be removed. With a small dataset, I ...
wnursal's user avatar
  • 163
5 votes
1 answer
3k views

Using Google BigQuery GIS table as a data source to QGIS

Google BigQuery recently added GIS functionality (still in beta), see here. I have a dataset in BigQuery that I would like to use as a data source in QGIS - much the same way as I would do with a ...
joaopcoelho's user avatar
1 vote
1 answer
2k views

Extracting mean values in Earth Engine

I am trying to extract the dates and mean NDVI values of a processed Google Earth Engine image collection thru Python. All goes well until I try to use getInfo(), where I always encounter this error: ...
crftdlve's user avatar
-1 votes
1 answer
102 views

Merging/combining huge point (layer) data using ArcMap?

i have 2 point feature layer (file GDB), each layer have a lot of point feature (records with depth value), around 80.000.000-100.000.000 records/point. there are overlapping area between those layer (...
Budi Badu's user avatar
6 votes
1 answer
2k views

Iterating an Earth Engine calculation (reducer) over every country in the world: a simple example

I'm an EE newbie, and would like to create a detailed 'resource' for the following challenge. The problem: What is the best way to iterate a simple calculation (e.g. a Reducer, or a simple user-...
antifrax's user avatar
  • 138
0 votes
1 answer
2k views

Creating climate maps in QGIS? [closed]

I have millions of points having temperature and humidity information covering the Middle East and North Africa (MENA). How can I create animated heatmaps/hotspot maps of change over time? The data ...
GIS Man's user avatar
  • 1,123
3 votes
4 answers
842 views

Calculate length of huge vector lines dataset

I have a MultiLineString vector layer from which I would like to calculate the length. I have tried to calculate the length with the Field Calculator by putting in the expression $length. The ...
Lutz's user avatar
  • 393
0 votes
1 answer
464 views

Processing Problems with the Difference (QGIS) and Erase (ArcGIS) Tool with big data

I am struggling with processing a huge data set. I downloaded it from a German geodata site (https://download.geofabrik.de/europe/germany.html). I downloaded the ZIP of Bavaria and clipped the eight ...
Lara Leonie's user avatar
2 votes
2 answers
2k views

Dice a polygon with over 21 million vertices in ArcMap?

Based on posts like this I suspect I'm bumping up against issues when trying to work with a feature class that contains features with a single very complicated polygon. These errors include 000426, ...
jac's user avatar
  • 341
0 votes
0 answers
103 views

Why does myRaster[1] <- 5 takes a very long time for a large raster?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
Andrei's user avatar
  • 766
1 vote
0 answers
471 views

How to getValues for raster with many values?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
Andrei's user avatar
  • 766
2 votes
0 answers
283 views

create neighborhood list in R with large dataset

I want to create a weight matrix based on distance. My code for the moment looks as follows and functions for a smaller sample of the data. However, with the large dataset (569424 individuals in 24077 ...
Kerstin's user avatar
  • 21
1 vote
0 answers
184 views

How can I limit a Hive spatial binning query to a specific map extent?

I'm trying to do a HQL query based off a map extent. I cant get this to work, what am I doing wrong? I'm using the taxi_demo in the ESRI example. SELECT ST_Bin(0.001, ST_Point(dropoff_longitude,...
Matthew Gray's user avatar
1 vote
0 answers
117 views

Point data grouping based on location + huge dataset in QGIS

I need community help to solve my puzzle. I have one huge dataset of points that represent places where a set of trucks stopped. Points go to the database if the reported speed of a truck was 0. The ...
Sergey K's user avatar
  • 111
0 votes
0 answers
79 views

Transferring data from file geodatabase (.gdb) to cassandra?

I want to store my spatial data in cassandra. I want to import data from an existing file geodatabase(.gdb) which has a table containing boundaries of polygons and a table containing point data. I ...
Sandeep Kumar's user avatar
3 votes
1 answer
893 views

Exporting Multiple Rasters to a Table: working with large dataset

I have 20 rasters that cover the same geographic extent, are in the same projection, have the same cell size etc. Instead of a stack of rasters, I want a table where the columns are the different ...
user avatar
3 votes
1 answer
1k views

read and write geotiffs bigger than memory?

The geotiff format is a versatile and easy to understand file format. The only limitation is when I try to do raster calculations I prefer to use GDAL (over ArcPy) which comes with the limitation that ...
RutgerH's user avatar
  • 3,295
3 votes
2 answers
2k views

Converting coordinate and raster values of each pixel to csv of large raster using Python?

There should be a simple way to convert a large raster to a csv or dataframe by reading the raster in as chunks, converting raster chunk to table of just three columns (x, y, and values), and ...
timpjohns's user avatar
  • 339
2 votes
1 answer
248 views

Feature class to feature class on huge data - iterate loop in ArcGIS pro?

I have a very large dataset containing a very large area. I have two files; one that is area specific shapefile and a standalone table that I have joined to the shapefile. Earlier, I have run feature ...
Rolf's user avatar
  • 61
0 votes
0 answers
490 views

Finding nearest points to POI with big data at Yelp?

Businesses often need to find nearest points to Points of Interest (POI). For example, use Yelp to find restaurants within 10 mile distance. As a normal GIS practice, we solve this problem by using ...
enaJ's user avatar
  • 311
4 votes
0 answers
1k views

How to store large time series of raster images

I've got a large time-series of rasters, derived from satellite images. Each raster is about 2000 x 2000 pixels, and I've got about 10,000 individual images. I'm trying to work out what the best way ...
robintw's user avatar
  • 4,006
1 vote
2 answers
513 views

How to split a large point vector layer in QGIS?

I need to split my shapefiles into much smaller pieces so that I can upload them onto the Postgis server. Shapefiles have DBF files with 14 Go (1,000,000 rows) and I need them to be under 2 Go. I've ...
Luffydude's user avatar
  • 2,318
3 votes
0 answers
696 views

Doing spatial regression using big data when Spatial weight matrix is too large for R?

I am currently doing an analysis on a quite huge data set - 250,000 polygons. I have figured out the general Moran's I for a linear model, but when I was trying to do a spatial lag or spatial error ...
Elvis Wu's user avatar
1 vote
2 answers
2k views

Displaying big data set on Google Maps? [closed]

I have a pretty large polygon shapefile (almost 1.5 million buildings) that I want to display in Google Maps. As the KML would be far too big, I have to create tiles, right? I don't want to host my ...
Wolfram's user avatar
  • 123
-1 votes
1 answer
175 views

Best program or tool for visualising big data sets really fast 10-20 GB data if the users don't know squat about GIS? [closed]

I have between 10-20 GB various GIS-data that there is a need to set up och and visualise for various users who want to be able to pan, zoom and switch between various layers and groups of layers ...
Regert's user avatar
  • 7
1 vote
1 answer
171 views

Save millions of points in the server side

I have asked a similar question about millions of points a few days ago:Display millions of points in the browser. It focused on how to display these data in the client(browser), however I think it ...
giser's user avatar
  • 861