Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

0
votes
1answer
50 views

Creating climate maps in QGIS? [closed]

I have millions of points having temperature and humidity information covering the Middle East and North Africa (MENA). How can I create animated heatmaps/hotspot maps of change over time? The data ...
3
votes
4answers
154 views

Calculate length of huge vector lines dataset

I have a MultiLineString vector layer from which I would like to calculate the length. I have tried to calculate the length with the Field Calculator by putting in the expression $length. The ...
0
votes
1answer
60 views

Processing Problems with the Difference (QGIS) and Erase (ArcGIS) Tool with big data

I am struggling with processing a huge data set. I downloaded it from a German geodata site (https://download.geofabrik.de/europe/germany.html). I downloaded the ZIP of Bavaria and clipped the eight ...
2
votes
2answers
159 views

Dice a polygon with over 21 million vertices in ArcMap?

Based on posts like this I suspect I'm bumping up against issues when trying to work with a feature class that contains features with a single very complicated polygon. These errors include 000426, ...
0
votes
0answers
32 views

Why does myRaster[1] <- 5 takes a very long time for a large raster?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
1
vote
0answers
53 views

How to getValues for raster with many values?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
1
vote
0answers
59 views

create neighborhood list in R with large dataset

I want to create a weight matrix based on distance. My code for the moment looks as follows and functions for a smaller sample of the data. However, with the large dataset (569424 individuals in 24077 ...
1
vote
0answers
76 views

How can I limit a Hive spatial binning query to a specific map extent?

I'm trying to do a HQL query based off a map extent. I cant get this to work, what am I doing wrong? I'm using the taxi_demo in the ESRI example. SELECT ST_Bin(0.001, ST_Point(dropoff_longitude,...
1
vote
0answers
42 views

Point data grouping based on location + huge dataset in QGIS

I need community help to solve my puzzle. I have one huge dataset of points that represent places where a set of trucks stopped. Points goes to the datase if the reported speed of a truck was 0. ...
0
votes
0answers
150 views

Extracting DEM from data files using QGIS?

I have three large files holding similar data. ***_g_gdal.grd GRD file 470 GB ***_g_int XYZ file 980 GB ***_g_geotiff TIF file 450 GB. I want to extract DEM file from any one of them using QGIS ...
0
votes
0answers
34 views

Transferring data from file geodatabase (.gdb) to cassandra?

I want to store my spatial data in cassandra. I want to import data from an existing file geodatabase(.gdb) which has a table containing boundaries of polygons and a table containing point data. I ...
3
votes
1answer
221 views

Exporting Multiple Rasters to a Table: working with large dataset

I have 20 rasters that cover the same geographic extent, are in the same projection, have the same cell size etc. Instead of a stack of rasters, I want a table where the columns are the different ...
2
votes
1answer
299 views

read and write geotiffs bigger than memory?

The geotiff format is a versatile and easy to understand file format. The only limitation is when I try to do raster calculations I prefer to use GDAL (over ArcPy) which comes with the limitation that ...
1
vote
2answers
363 views

Converting coordinate and raster values of each pixel to csv of large raster using Python?

There should be a simple way to convert a large raster to a csv or dataframe by reading the raster in as chunks, converting raster chunk to table of just three columns (x, y, and values), and ...
1
vote
1answer
93 views

Feature class to feature class on huge data - iterate loop in ArcGIS pro?

I have a very large dataset containing a very large area. I have two files; one that is area specific shapefile and a standalone table that I have joined to the shapefile. Earlier, I have run feature ...
1
vote
0answers
190 views

Finding nearest points to POI with big data at Yelp?

Businesses often need to find nearest points to Points of Interest (POI). For example, use Yelp to find restaurants within 10 mile distance. As a normal GIS practice, we solve this problem by using ...
2
votes
0answers
493 views

How to store large time series of raster images

I've got a large time-series of rasters, derived from satellite images. Each raster is about 2000 x 2000 pixels, and I've got about 10,000 individual images. I'm trying to work out what the best way ...
1
vote
2answers
222 views

How to split a large point vector layer in QGIS?

I need to split my shapefiles into much smaller pieces so that I can upload them onto the Postgis server. Shapefiles have DBF files with 14 Go (1,000,000 rows) and I need them to be under 2 Go. I've ...
3
votes
0answers
358 views

Doing spatial regression using big data when Spatial weight matrix is too large for R?

I am currently doing an analysis on a quite huge data set - 250,000 polygons. I have figured out the general Moran's I for a linear model, but when I was trying to do a spatial lag or spatial error ...
0
votes
0answers
152 views

Intersect using Open Source GIS [duplicate]

I have a CSV containing user locations (has lat, long, userid, timestamp). I have a countries shapefile (has country name, geometry - got this from thematicmapping.org). I am trying to append a ...
1
vote
2answers
1k views

Display big data set on Google Maps

I have a pretty large polygon shapefile (almost 1.5 million buildings) that I want to display in Google Maps. As the KML would be far too big, I have to create tiles, right? I don't want to host my ...
-1
votes
1answer
128 views

Best program or tool for visualising big data sets really fast 10-20 GB data if the users don't know squat about GIS? [closed]

I have between 10-20 GB various GIS-data that there is a need to set up och and visualise for various users who want to be able to pan, zoom and switch between various layers and groups of layers ...
1
vote
1answer
102 views

Save millions of points in the server side

I have asked a similar question about millions of points a few days ago:Display millions of points in the browser. It focused on how to display these data in the client(browser), however I think it ...
1
vote
1answer
6k views

Any tool to view large tiff images?

I'm looking for some application to view some large TIFF images and large mosaics consist of tiff images. At the moment I have VLIV for viewing large tiff files. And tools based on HD view: HD View ...
7
votes
2answers
183 views

Remote geoprocessing solutions?

I have a relatively old laptop (i5 8Gb) but it is fine for most of my GIS needs. Infrequently I would like to run processes which requires larger amounts of RAM (16G+) but I cannot justify forking out ...
1
vote
0answers
38 views

Sending commands into FGDB

I have a large .gdb of all blocks in the United States, available from Census. I would like to create a point shape file of all census tract centroids in the US. I have QGIS with FileGDB enabled, as ...
20
votes
2answers
1k views

What is the best hack for importing large datasets into PostGIS?

I have to import large Shapefiles (> 1 million records) into PostGIS, and I have been wondering about the best way to do it. In my question I used the word "hack", instead of tool, on purpose because ...
6
votes
2answers
807 views

Growth of a PostGIS database for Social Media - Move to Cassandra?

I currently have a large and growing database of geo-social media. I'm using Postgres with postgis, and so far it's been quite good. My use case thus far has been to query the database for all posts ...
1
vote
0answers
301 views

Big data rastering using ArcGIS Platform?

I have the daunting task doing raster analysis on a record set that runs into quite a few millions, and worse still is that the data is in Hive. So far I have managed to use record set of 2 million ...
2
votes
2answers
300 views

What simple solutions are available for sharing big data with end users?

I have approximately 1700 raster tiles showing classified land cover with a total size of approximately 300GB. I am looking for a simple solution to privately serve these data to non-technical end ...
1
vote
1answer
214 views

Arcpy scripts fail on big datasets - anyone has experienced similar behavior?

Here a question that is not extremely specific but maybe someone has had similar experiences... I have a few Arcpy scripts that work perfectly when used with small datasets but that silently fail ...
4
votes
1answer
254 views

Limits to torque and cartodb

Bit of a theoretical one this, but I was wondering if anyone had tried to use CartoDB and Torque with any big(ish) data. What I am talking about is having the location of 75,000 people at 1 minute ...
4
votes
1answer
355 views

Should slow refresh rate be expected while viewing 1000 points out of 100 million using ArcSDE/Oracle?

Is it normal to experience very slow refresh rate in ArcMap with 1000 points shown out of 100 million using ArcSDE/Oracle? I’am testing a setup with ArcSDE/Oracle database to contain geographical ...
9
votes
3answers
6k views

Best way to clip a large raster ECW file ?

I am trying to clip a large ECW (details below) but the raster file is too big to be processed entirely. Some details of the ECW below Driver: ECW/ERDAS Compressed Wavelets (SDK 5.0) file size: ...
2
votes
2answers
1k views

What is faster for geoprocessing (intersect) big shapefiles?

I have 200 Mb shapefiles of polylines downloaded from OSM. I've been working with QGIS 2.0 in Ubuntu 64 Bit 8 Gb RAM to make some geoprocessing tasks and it's taking 2 days for processing. Once I've ...
3
votes
2answers
554 views

How to fix the “Corrupt Shape file '<shapefile>' encountered — record size 218972988 is invalid” error in FME Workbench?

I have inherited a project where part of the workflow is the execution of numerous FME Workbench files. It has worked flawlessly for several hundred runs, but it gets tripped up on a very large ...
2
votes
2answers
3k views

Using NetCDF for point time-series observations?

I have a large growing dataset of water and climate observations for several years in regular hourly time-step. The number of locations is cca. 1000 locations with latitude, longitude coordinates. At ...
1
vote
1answer
184 views

Prism mobile phone metadata: does it include location? [closed]

I've read that Prism collects metadata, but no details of what that metadata includes. Does anyone know if location is part of the metadata?
15
votes
1answer
2k views

How can I utilize NumPy arrays to optimize big data geoprocessing?

I'm interested in learning how to utilize NumPy arrays to optimize geoprocessing. Much of my work involves "big data", where geoprocessing often takes days to accomplish certain tasks. Needless to ...
1
vote
1answer
3k views

Mosaic large dataset with R

I need to merge together 20 32 bit tiff. For that I tried ArcGis but it keeps crashing (the individual tiff are about 1GB). I need a finale Tiff in order to run some analysis and I figured that having ...
7
votes
2answers
6k views

How to find and delete identical points?

I recently performed Integrate on a merged point feature class in ArcGIS 10.1. I specified "1 meter" XY tolerance for the Integrate command. The resulting point feature appears to be what I want ...
34
votes
2answers
2k views

Streamlining Python Code for Big Data

I have Python code which is designed to take point shapefiles through the following workflow: Merge points Integrate points, such that any points within 1 m of each other become one point Create ...
12
votes
2answers
3k views

Methods for optimizing multcore processing in ArcGIS

I am interested in learning methods to utilize the full extent of multicore processing power available on a desktop computer. Arc states that background geoprocessing allows the user to utilize ...
2
votes
0answers
801 views

Large point to Polygon by buffer-join-buffer-dissolve issues

I have a few large datasets (about 10 million points) which I am buffering, then joining, dowgrading and finally buffering and dissolving. This is all in a model which works fine with sample smaller ...