Questions tagged [big-data]

A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

Filter by
Sorted by
Tagged with
1
vote
0answers
19 views

Storing earth observation derived attributes for polygons [closed]

I want to generate a number of vegetation related indices (NDVI, NDTI, etc., min/max/avg/std (pixel values)) from earth observation data for a large number of polygons during the entire growing season,...
0
votes
1answer
50 views

Fast road snapping for big data

I hope this question is not too general. I'm dealing with a lot of data from vehicles (1.000.000 GPS points per day) and I want to snap the data to a big city's street network. The timely order is not ...
3
votes
1answer
63 views

Compute pixel area of a large raster in Google Earth Engine

I want to compute a total area of a large raster dataset using .reduceRegion(), but receive the "Earth Engine capacity exceeded" Error. Is it possible to overcome it? For example create a vector grid ...
1
vote
4answers
163 views

Is there anything faster than PostGIS ST_Intersects for seeing where features intersect (or something wrong with my approach)?

Basically have 2 tables. 1 is buildings (1809555 rows called oregon). Another is flood zone (164 rows called oregonslr1). Want to see where they intersect to get number of buildings flooded. I am ...
0
votes
1answer
29 views

Second-level neighbors of large polygon (neighbors' neighbors)

I have a large-ish set of polygons and am after identifying second-level neighbors of each, that is, the neighbors of the neighbors of each polygon (distinctly, i.e. the 2nd-level neighbors cannot ...
3
votes
1answer
158 views

Filtering (lasfilter on) huge LiDAR dataset

My dataset is about 8.3 GB after loaded into R environment by using lidR package. Some points which have return number greater than number of returns need to be removed. With a small dataset, I ...
3
votes
1answer
440 views

Using Google BigQuery GIS table as a data source to QGIS

Google BigQuery recently added GIS functionality (still in beta), see here. I have a dataset in BigQuery that I would like to use as a data source in QGIS - much the same way as I would do with a ...
1
vote
1answer
452 views

Extracting mean values in Earth Engine

I am trying to extract the dates and mean NDVI values of a processed Google Earth Engine image collection thru Python. All goes well until I try to use getInfo(), where I always encounter this error: ...
-1
votes
1answer
42 views

Merging/combining huge point (layer) data using ArcMap?

i have 2 point feature layer (file GDB), each layer have a lot of point feature (records with depth value), around 80.000.000-100.000.000 records/point. there are overlapping area between those layer (...
2
votes
1answer
597 views

Iterating an Earth Engine calculation (reducer) over every country in the world: a simple example

I'm an EE newbie, and would like to create a detailed 'resource' for the following challenge. The problem: What is the best way to iterate a simple calculation (e.g. a Reducer, or a simple user-...
0
votes
1answer
214 views

Creating climate maps in QGIS? [closed]

I have millions of points having temperature and humidity information covering the Middle East and North Africa (MENA). How can I create animated heatmaps/hotspot maps of change over time? The data ...
3
votes
4answers
342 views

Calculate length of huge vector lines dataset

I have a MultiLineString vector layer from which I would like to calculate the length. I have tried to calculate the length with the Field Calculator by putting in the expression $length. The ...
0
votes
1answer
137 views

Processing Problems with the Difference (QGIS) and Erase (ArcGIS) Tool with big data

I am struggling with processing a huge data set. I downloaded it from a German geodata site (https://download.geofabrik.de/europe/germany.html). I downloaded the ZIP of Bavaria and clipped the eight ...
2
votes
2answers
463 views

Dice a polygon with over 21 million vertices in ArcMap?

Based on posts like this I suspect I'm bumping up against issues when trying to work with a feature class that contains features with a single very complicated polygon. These errors include 000426, ...
0
votes
0answers
52 views

Why does myRaster[1] <- 5 takes a very long time for a large raster?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
1
vote
0answers
158 views

How to getValues for raster with many values?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
2
votes
0answers
102 views

create neighborhood list in R with large dataset

I want to create a weight matrix based on distance. My code for the moment looks as follows and functions for a smaller sample of the data. However, with the large dataset (569424 individuals in 24077 ...
1
vote
0answers
119 views

How can I limit a Hive spatial binning query to a specific map extent?

I'm trying to do a HQL query based off a map extent. I cant get this to work, what am I doing wrong? I'm using the taxi_demo in the ESRI example. SELECT ST_Bin(0.001, ST_Point(dropoff_longitude,...
1
vote
0answers
62 views

Point data grouping based on location + huge dataset in QGIS

I need community help to solve my puzzle. I have one huge dataset of points that represent places where a set of trucks stopped. Points go to the database if the reported speed of a truck was 0. The ...
0
votes
0answers
46 views

Transferring data from file geodatabase (.gdb) to cassandra?

I want to store my spatial data in cassandra. I want to import data from an existing file geodatabase(.gdb) which has a table containing boundaries of polygons and a table containing point data. I ...
3
votes
1answer
409 views

Exporting Multiple Rasters to a Table: working with large dataset

I have 20 rasters that cover the same geographic extent, are in the same projection, have the same cell size etc. Instead of a stack of rasters, I want a table where the columns are the different ...
2
votes
1answer
470 views

read and write geotiffs bigger than memory?

The geotiff format is a versatile and easy to understand file format. The only limitation is when I try to do raster calculations I prefer to use GDAL (over ArcPy) which comes with the limitation that ...
1
vote
2answers
562 views

Converting coordinate and raster values of each pixel to csv of large raster using Python?

There should be a simple way to convert a large raster to a csv or dataframe by reading the raster in as chunks, converting raster chunk to table of just three columns (x, y, and values), and ...
2
votes
1answer
133 views

Feature class to feature class on huge data - iterate loop in ArcGIS pro?

I have a very large dataset containing a very large area. I have two files; one that is area specific shapefile and a standalone table that I have joined to the shapefile. Earlier, I have run feature ...
1
vote
0answers
284 views

Finding nearest points to POI with big data at Yelp?

Businesses often need to find nearest points to Points of Interest (POI). For example, use Yelp to find restaurants within 10 mile distance. As a normal GIS practice, we solve this problem by using ...
2
votes
0answers
709 views

How to store large time series of raster images

I've got a large time-series of rasters, derived from satellite images. Each raster is about 2000 x 2000 pixels, and I've got about 10,000 individual images. I'm trying to work out what the best way ...
1
vote
2answers
272 views

How to split a large point vector layer in QGIS?

I need to split my shapefiles into much smaller pieces so that I can upload them onto the Postgis server. Shapefiles have DBF files with 14 Go (1,000,000 rows) and I need them to be under 2 Go. I've ...
3
votes
0answers
426 views

Doing spatial regression using big data when Spatial weight matrix is too large for R?

I am currently doing an analysis on a quite huge data set - 250,000 polygons. I have figured out the general Moran's I for a linear model, but when I was trying to do a spatial lag or spatial error ...
1
vote
2answers
1k views

Displaying big data set on Google Maps? [closed]

I have a pretty large polygon shapefile (almost 1.5 million buildings) that I want to display in Google Maps. As the KML would be far too big, I have to create tiles, right? I don't want to host my ...
-1
votes
1answer
132 views

Best program or tool for visualising big data sets really fast 10-20 GB data if the users don't know squat about GIS? [closed]

I have between 10-20 GB various GIS-data that there is a need to set up och and visualise for various users who want to be able to pan, zoom and switch between various layers and groups of layers ...
1
vote
1answer
106 views

Save millions of points in the server side

I have asked a similar question about millions of points a few days ago:Display millions of points in the browser. It focused on how to display these data in the client(browser), however I think it ...
2
votes
1answer
8k views

Any tool to view large tiff images?

I'm looking for some application to view some large TIFF images and large mosaics consist of tiff images. At the moment I have VLIV for viewing large tiff files. And tools based on HD view: HD View ...
7
votes
2answers
189 views

Remote geoprocessing solutions?

I have a relatively old laptop (i5 8Gb) but it is fine for most of my GIS needs. Infrequently I would like to run processes which requires larger amounts of RAM (16G+) but I cannot justify forking out ...
1
vote
0answers
39 views

Sending commands into FGDB

I have a large .gdb of all blocks in the United States, available from Census. I would like to create a point shape file of all census tract centroids in the US. I have QGIS with FileGDB enabled, as ...
20
votes
2answers
2k views

What is the best hack for importing large datasets into PostGIS?

I have to import large Shapefiles (> 1 million records) into PostGIS, and I have been wondering about the best way to do it. In my question I used the word "hack", instead of tool, on purpose because ...
5
votes
2answers
936 views

Moving PostGIS database for Social Media to Cassandra? [closed]

I currently have a large and growing database of geo-social media. I'm using Postgres with postgis, and so far it's been quite good. My use case thus far has been to query the database for all posts ...
1
vote
0answers
311 views

Big data rastering using ArcGIS Platform?

I have the daunting task doing raster analysis on a record set that runs into quite a few millions, and worse still is that the data is in Hive. So far I have managed to use record set of 2 million ...
2
votes
2answers
308 views

What simple solutions are available for sharing big data with end users?

I have approximately 1700 raster tiles showing classified land cover with a total size of approximately 300GB. I am looking for a simple solution to privately serve these data to non-technical end ...
1
vote
1answer
215 views

Arcpy scripts fail on big datasets - anyone has experienced similar behavior?

Here a question that is not extremely specific but maybe someone has had similar experiences... I have a few Arcpy scripts that work perfectly when used with small datasets but that silently fail ...
4
votes
1answer
257 views

Limits to torque and cartodb

Bit of a theoretical one this, but I was wondering if anyone had tried to use CartoDB and Torque with any big(ish) data. What I am talking about is having the location of 75,000 people at 1 minute ...
4
votes
1answer
364 views

Should slow refresh rate be expected while viewing 1000 points out of 100 million using ArcSDE/Oracle?

Is it normal to experience very slow refresh rate in ArcMap with 1000 points shown out of 100 million using ArcSDE/Oracle? I’am testing a setup with ArcSDE/Oracle database to contain geographical ...
9
votes
3answers
7k views

Best way to clip a large raster ECW file ?

I am trying to clip a large ECW (details below) but the raster file is too big to be processed entirely. Some details of the ECW below Driver: ECW/ERDAS Compressed Wavelets (SDK 5.0) file size: ...
2
votes
2answers
2k views

What is faster for geoprocessing (intersect) big shapefiles?

I have 200 Mb shapefiles of polylines downloaded from OSM. I've been working with QGIS 2.0 in Ubuntu 64 Bit 8 Gb RAM to make some geoprocessing tasks and it's taking 2 days for processing. Once I've ...
3
votes
2answers
628 views

How to fix the “Corrupt Shape file '<shapefile>' encountered — record size 218972988 is invalid” error in FME Workbench?

I have inherited a project where part of the workflow is the execution of numerous FME Workbench files. It has worked flawlessly for several hundred runs, but it gets tripped up on a very large ...
2
votes
2answers
4k views

Using NetCDF for point time-series observations?

I have a large growing dataset of water and climate observations for several years in regular hourly time-step. The number of locations is cca. 1000 locations with latitude, longitude coordinates. At ...
1
vote
1answer
190 views

Prism mobile phone metadata: does it include location? [closed]

I've read that Prism collects metadata, but no details of what that metadata includes. Does anyone know if location is part of the metadata?
16
votes
1answer
2k views

How can I utilize NumPy arrays to optimize big data geoprocessing?

I'm interested in learning how to utilize NumPy arrays to optimize geoprocessing. Much of my work involves "big data", where geoprocessing often takes days to accomplish certain tasks. Needless to ...
1
vote
1answer
3k views

Mosaic large dataset with R

I need to merge together 20 32 bit tiff. For that I tried ArcGis but it keeps crashing (the individual tiff are about 1GB). I need a finale Tiff in order to run some analysis and I figured that having ...
6
votes
2answers
7k views

How to find and delete identical points?

I recently performed Integrate on a merged point feature class in ArcGIS 10.1. I specified "1 meter" XY tolerance for the Integrate command. The resulting point feature appears to be what I want ...
34
votes
2answers
2k views

Streamlining Python Code for Big Data

I have Python code which is designed to take point shapefiles through the following workflow: Merge points Integrate points, such that any points within 1 m of each other become one point Create ...