Questions tagged [big-data]

A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

Filter by
Sorted by
Tagged with
0 votes
0 answers
34 views

QGIS count points in polygon by category

This question is following to this answer point count in polygon by category array_length(overlay_intersects('pointlayername',$id,filter:="fieldcontainingcategory"='nameofcategory')) This ...
  • 381
1 vote
1 answer
85 views

Finding rasters overlapping polygon shapefile

I have 10,000 rasters but I only need 1000 of them but do not know which 1000. The 1,000 I need to utilize overlap a vector polygon shapefile and another single raster I have (well the value of 1 in ...
3 votes
1 answer
117 views

Exporting large txt file as point shapefile in QGIS

I have a large txt file with 10 million rows and 16 columns and I already have imported the table as a delimited text file in QGIS. In row 7 and row 8, the coordinates are stored (I also allocated x ...
3 votes
0 answers
204 views

Which Big Data analytics platform has most extensive GIS support? [closed]

I've recently had to deal with datasets that are simply too large to be efficiently processed using PostgreSQL and so I've turned to Google's BigQuery for those tasks. BigQuery, while offering ...
  • 2,799
1 vote
2 answers
1k views

Getting "Non-successful status code 502" error when reverse geocoding of large data in GeoPy

I have a data size of 1.87GB containing 14962000 rows and 9 columns. And I want to apply GeoPy but I got the errors. My code: import pandas as pd from geopy.geocoders import Nominatim from geopy....
3 votes
1 answer
316 views

What is the most efficient way to create millions of lines using Python from coordinates? [closed]

I am trying to create a flow map and I have 150 million rows of data. I have tried to use QGIS's Virtual Layer (described in Creating virtual layers using Python and export them without using QGIS UI) ...
  • 571
0 votes
1 answer
159 views

Load big wfs layer in R

i´m trying to load the layer Mroad_Lden_17 into R with the code below: proj_folder <- "C:/Users/... " setwd(proj_folder) # helper function for querying uba load_laerm <- function(what)...
  • 9
0 votes
0 answers
113 views

Big data and spatial regression: how to make spml more efficient in R?

I am running a spatial panel regression in R using spml from package splm. When I conduct this on about 1/200th of my data, the program executes. But when I try to do so on a third of my data, R uses ...
  • 353
1 vote
1 answer
719 views

Google Earth Engine: Error exporting a large Feature Collection to CSV

On GEE, I want to export daily weather variables for 18 years for about 30,000 rectangular polygons. That would be exporting a FeatureCollection to a csv file with ~200 million rows (30000 places * ...
  • 111
2 votes
0 answers
104 views

Spatio-temporal clustering Algorithm

I'm working with a dataset with latitude, longitude and date-time. I have millions of points per day and I want to group them using a clustering algorithm. I read this question Clustering geographic ...
7 votes
1 answer
1k views

Multi-threaded compression in rasterio?

For writing large raster files (e.g. satellite image GeoTIFFs) to disk, the throughput rate tends to be limited by the compression algorithm rather than by the raw disk IO. (Benchmarks: 1, 2.) This is ...
  • 466
5 votes
2 answers
742 views

Display millions of locations very, very fast

We're building a mapping application for voter data using PostGIS and Leaflet. It has a web interface. Our voter file will have several million entries. In our app you can select a subset of the voter ...
  • 297
1 vote
0 answers
49 views

Storing earth observation derived attributes for polygons [closed]

I want to generate a number of vegetation related indices (NDVI, NDTI, etc., min/max/avg/std (pixel values)) from earth observation data for a large number of polygons during the entire growing season,...
0 votes
1 answer
556 views

Fast road snapping for big data

I hope this question is not too general. I'm dealing with a lot of data from vehicles (1.000.000 GPS points per day) and I want to snap the data to a big city's street network. The timely order is not ...
3 votes
1 answer
345 views

Compute pixel area of a large raster in Google Earth Engine

I want to compute a total area of a large raster dataset using .reduceRegion(), but receive the "Earth Engine capacity exceeded" Error. Is it possible to overcome it? For example create a vector grid ...
  • 3,373
1 vote
5 answers
2k views

Is there anything faster than PostGIS ST_Intersects for seeing where features intersect (or something wrong with my approach)?

Basically have 2 tables. 1 is buildings (1809555 rows called oregon). Another is flood zone (164 rows called oregonslr1). Want to see where they intersect to get number of buildings flooded. I am ...
0 votes
1 answer
128 views

Second-level neighbors of large polygon (neighbors' neighbors)

I have a large-ish set of polygons and am after identifying second-level neighbors of each, that is, the neighbors of the neighbors of each polygon (distinctly, i.e. the 2nd-level neighbors cannot ...
3 votes
1 answer
650 views

Filtering (lasfilter on) huge LiDAR dataset

My dataset is about 8.3 GB after loaded into R environment by using lidR package. Some points which have return number greater than number of returns need to be removed. With a small dataset, I ...
  • 153
5 votes
1 answer
2k views

Using Google BigQuery GIS table as a data source to QGIS

Google BigQuery recently added GIS functionality (still in beta), see here. I have a dataset in BigQuery that I would like to use as a data source in QGIS - much the same way as I would do with a ...
1 vote
1 answer
1k views

Extracting mean values in Earth Engine

I am trying to extract the dates and mean NDVI values of a processed Google Earth Engine image collection thru Python. All goes well until I try to use getInfo(), where I always encounter this error: ...
-1 votes
1 answer
59 views

Merging/combining huge point (layer) data using ArcMap?

i have 2 point feature layer (file GDB), each layer have a lot of point feature (records with depth value), around 80.000.000-100.000.000 records/point. there are overlapping area between those layer (...
6 votes
1 answer
2k views

Iterating an Earth Engine calculation (reducer) over every country in the world: a simple example

I'm an EE newbie, and would like to create a detailed 'resource' for the following challenge. The problem: What is the best way to iterate a simple calculation (e.g. a Reducer, or a simple user-...
  • 138
0 votes
1 answer
1k views

Creating climate maps in QGIS? [closed]

I have millions of points having temperature and humidity information covering the Middle East and North Africa (MENA). How can I create animated heatmaps/hotspot maps of change over time? The data ...
  • 1,095
3 votes
4 answers
746 views

Calculate length of huge vector lines dataset

I have a MultiLineString vector layer from which I would like to calculate the length. I have tried to calculate the length with the Field Calculator by putting in the expression $length. The ...
  • 373
0 votes
1 answer
425 views

Processing Problems with the Difference (QGIS) and Erase (ArcGIS) Tool with big data

I am struggling with processing a huge data set. I downloaded it from a German geodata site (https://download.geofabrik.de/europe/germany.html). I downloaded the ZIP of Bavaria and clipped the eight ...
2 votes
2 answers
1k views

Dice a polygon with over 21 million vertices in ArcMap?

Based on posts like this I suspect I'm bumping up against issues when trying to work with a feature class that contains features with a single very complicated polygon. These errors include 000426, ...
  • 329
0 votes
0 answers
89 views

Why does myRaster[1] <- 5 takes a very long time for a large raster?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
  • 676
1 vote
0 answers
375 views

How to getValues for raster with many values?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
  • 676
2 votes
0 answers
229 views

create neighborhood list in R with large dataset

I want to create a weight matrix based on distance. My code for the moment looks as follows and functions for a smaller sample of the data. However, with the large dataset (569424 individuals in 24077 ...
  • 21
1 vote
0 answers
162 views

How can I limit a Hive spatial binning query to a specific map extent?

I'm trying to do a HQL query based off a map extent. I cant get this to work, what am I doing wrong? I'm using the taxi_demo in the ESRI example. SELECT ST_Bin(0.001, ST_Point(dropoff_longitude,...
1 vote
0 answers
90 views

Point data grouping based on location + huge dataset in QGIS

I need community help to solve my puzzle. I have one huge dataset of points that represent places where a set of trucks stopped. Points go to the database if the reported speed of a truck was 0. The ...
  • 111
0 votes
0 answers
61 views

Transferring data from file geodatabase (.gdb) to cassandra?

I want to store my spatial data in cassandra. I want to import data from an existing file geodatabase(.gdb) which has a table containing boundaries of polygons and a table containing point data. I ...
3 votes
1 answer
821 views

Exporting Multiple Rasters to a Table: working with large dataset

I have 20 rasters that cover the same geographic extent, are in the same projection, have the same cell size etc. Instead of a stack of rasters, I want a table where the columns are the different ...
user avatar
3 votes
1 answer
1k views

read and write geotiffs bigger than memory?

The geotiff format is a versatile and easy to understand file format. The only limitation is when I try to do raster calculations I prefer to use GDAL (over ArcPy) which comes with the limitation that ...
  • 2,995
2 votes
2 answers
1k views

Converting coordinate and raster values of each pixel to csv of large raster using Python?

There should be a simple way to convert a large raster to a csv or dataframe by reading the raster in as chunks, converting raster chunk to table of just three columns (x, y, and values), and ...
  • 329
2 votes
1 answer
219 views

Feature class to feature class on huge data - iterate loop in ArcGIS pro?

I have a very large dataset containing a very large area. I have two files; one that is area specific shapefile and a standalone table that I have joined to the shapefile. Earlier, I have run feature ...
  • 61
0 votes
0 answers
436 views

Finding nearest points to POI with big data at Yelp?

Businesses often need to find nearest points to Points of Interest (POI). For example, use Yelp to find restaurants within 10 mile distance. As a normal GIS practice, we solve this problem by using ...
  • 291
4 votes
0 answers
1k views

How to store large time series of raster images

I've got a large time-series of rasters, derived from satellite images. Each raster is about 2000 x 2000 pixels, and I've got about 10,000 individual images. I'm trying to work out what the best way ...
  • 3,906
1 vote
2 answers
420 views

How to split a large point vector layer in QGIS?

I need to split my shapefiles into much smaller pieces so that I can upload them onto the Postgis server. Shapefiles have DBF files with 14 Go (1,000,000 rows) and I need them to be under 2 Go. I've ...
  • 2,220
3 votes
0 answers
581 views

Doing spatial regression using big data when Spatial weight matrix is too large for R?

I am currently doing an analysis on a quite huge data set - 250,000 polygons. I have figured out the general Moran's I for a linear model, but when I was trying to do a spatial lag or spatial error ...
1 vote
2 answers
2k views

Displaying big data set on Google Maps? [closed]

I have a pretty large polygon shapefile (almost 1.5 million buildings) that I want to display in Google Maps. As the KML would be far too big, I have to create tiles, right? I don't want to host my ...
  • 123
-1 votes
1 answer
152 views

Best program or tool for visualising big data sets really fast 10-20 GB data if the users don't know squat about GIS? [closed]

I have between 10-20 GB various GIS-data that there is a need to set up och and visualise for various users who want to be able to pan, zoom and switch between various layers and groups of layers ...
  • 7
1 vote
1 answer
154 views

Save millions of points in the server side

I have asked a similar question about millions of points a few days ago:Display millions of points in the browser. It focused on how to display these data in the client(browser), however I think it ...
  • 1,007
2 votes
1 answer
13k views

Any tool to view large tiff images?

I'm looking for some application to view some large TIFF images and large mosaics consist of tiff images. At the moment I have VLIV for viewing large tiff files. And tools based on HD view: HD View ...
  • 601
7 votes
2 answers
201 views

Remote geoprocessing solutions?

I have a relatively old laptop (i5 8Gb) but it is fine for most of my GIS needs. Infrequently I would like to run processes which requires larger amounts of RAM (16G+) but I cannot justify forking out ...
  • 505
1 vote
0 answers
42 views

Sending commands into FGDB

I have a large .gdb of all blocks in the United States, available from Census. I would like to create a point shape file of all census tract centroids in the US. I have QGIS with FileGDB enabled, as ...
25 votes
2 answers
4k views

What is the best hack for importing large datasets into PostGIS?

I have to import large Shapefiles (> 1 million records) into PostGIS, and I have been wondering about the best way to do it. In my question I used the word "hack", instead of tool, on purpose because ...
5 votes
2 answers
1k views

Moving PostGIS database for Social Media to Cassandra? [closed]

I currently have a large and growing database of geo-social media. I'm using Postgres with postgis, and so far it's been quite good. My use case thus far has been to query the database for all posts ...
1 vote
0 answers
321 views

Big data rastering using ArcGIS Platform?

I have the daunting task doing raster analysis on a record set that runs into quite a few millions, and worse still is that the data is in Hive. So far I have managed to use record set of 2 million ...
2 votes
2 answers
336 views

What simple solutions are available for sharing big data with end users?

I have approximately 1700 raster tiles showing classified land cover with a total size of approximately 300GB. I am looking for a simple solution to privately serve these data to non-technical end ...
  • 50.5k