A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

learn more… | top users | synonyms

1
vote
1answer
33 views

Save millions of points in the server side

I have asked a similar question about millions of points a few days ago:Display millions of points in the browser. It focused on how to display these data in the client(browser), however I think it ...
2
votes
1answer
132 views

Any tool to view large tiff images?

I'm looking for some application to view some large TIFF images and large mosaics consist of tiff images. At the moment I have VLIV for viewing large tiff files. And tools based on HD view: HD View ...
6
votes
2answers
116 views

Remote geoprocessing solutions?

I have a relatively old laptop (i5 8Gb) but it is fine for most of my GIS needs. Infrequently I would like to run processes which requires larger amounts of RAM (16G+) but I cannot justify forking out ...
1
vote
0answers
29 views

Sending commands into FGDB

I have a large .gdb of all blocks in the United States, available from Census. I would like to create a point shape file of all census tract centroids in the US. I have QGIS with FileGDB enabled, as ...
10
votes
2answers
305 views

What is the best hack for importing large datasets into PostGIS?

I have to import large Shapefiles (> 1 million records) into PostGIS, and I have been wondering about the best way to do it. In my question I used the word "hack", instead of tool, on purpose ...
6
votes
2answers
177 views

Growth of a PostGIS database for Social Media - Move to Cassandra?

I currently have a large and growing database of geo-social media. I'm using Postgres with postgis, and so far it's been quite good. My use case thus far has been to query the database for all posts ...
1
vote
0answers
67 views

Big data rastering using ArcGIS Platform?

I have the daunting task doing raster analysis on a record set that runs into quite a few millions, and worse still is that the data is in Hive. So far I have managed to use record set of 2 million ...
2
votes
2answers
115 views

What simple solutions are available for sharing big data with end users?

I have approximately 1700 raster tiles showing classified land cover with a total size of approximately 300GB. I am looking for a simple solution to privately serve these data to non-technical end ...
1
vote
1answer
94 views

Arcpy scripts fail on big datasets - anyone has experienced similar behavior?

Here a question that is not extremely specific but maybe someone has had similar experiences... I have a few Arcpy scripts that work perfectly when used with small datasets but that silently fail ...
3
votes
1answer
122 views

Limits to torque and cartodb

Bit of a theoretical one this, but I was wondering if anyone had tried to use CartoDB and Torque with any big(ish) data. What I am talking about is having the location of 75,000 people at 1 minute ...
4
votes
1answer
122 views

Should slow refresh rate be expected while viewing 1000 points out of 100 million using ArcSDE/Oracle?

Is it normal to experience very slow refresh rate in ArcMap with 1000 points shown out of 100 million using ArcSDE/Oracle? I’am testing a setup with ArcSDE/Oracle database to contain geographical ...
6
votes
2answers
1k views

Best way to clip a large raster ECW file ?

I am trying to clip a large ECW (details below) but the raster file is too big to be processed entirely. Some details of the ECW below Driver: ECW/ERDAS Compressed Wavelets (SDK 5.0) file size: ...
2
votes
2answers
351 views

What is faster for geoprocessing (intersect) big shapefiles?

I have 200 Mb shapefiles of polylines downloaded from OSM. I've been working with QGIS 2.0 in Ubuntu 64 Bit 8 Gb RAM to make some geoprocessing tasks and it's taking 2 days for processing. Once I've ...
3
votes
2answers
227 views

How to fix the “Corrupt Shape file '<shapefile>' encountered — record size 218972988 is invalid” error in FME Workbench?

I have inherited a project where part of the workflow is the execution of numerous FME Workbench files. It has worked flawlessly for several hundred runs, but it gets tripped up on a very large ...
0
votes
2answers
774 views

Using NetCDF for point time-series observations?

I have a large growing dataset of water and climate observations for several years in regular hourly time-step. The number of locations is cca. 1000 locations with latitude, longitude coordinates. At ...
1
vote
1answer
160 views

Prism mobile phone metadata: does it include location? [closed]

I've read that Prism collects metadata, but no details of what that metadata includes. Does anyone know if location is part of the metadata?
13
votes
1answer
813 views

How can I utilize NumPy arrays to optimize big data geoprocessing?

I'm interested in learning how to utilize NumPy arrays to optimize geoprocessing. Much of my work involves "big data", where geoprocessing often takes days to accomplish certain tasks. Needless to ...
1
vote
1answer
752 views

Mosaic large dataset with R

I need to merge together 20 32 bit tiff. For that I tried ArcGis but it keeps crashing (the individual tiff are about 1GB). I need a finale Tiff in order to run some analysis and I figured that having ...
4
votes
1answer
3k views

How to find and delete identical points?

I recently performed Integrate on a merged point feature class in ArcGIS 10.1. I specified "1 meter" XY tolerance for the Integrate command. The resulting point feature appears to be what I want ...
29
votes
2answers
2k views

Streamlining Python Code for Big Data

I have Python code which is designed to take point shapefiles through the following workflow: Merge points Integrate points, such that any points within 1 m of each other become one point Create ...
10
votes
2answers
1k views

Methods for optimizing multcore processing in ArcGIS

I am interested in learning methods to utilize the full extent of multicore processing power available on a desktop computer. Arc states that background geoprocessing allows the user to utilize ...
1
vote
0answers
501 views

Large point to Polygon by buffer-join-buffer-dissolve issues

I have a few large datasets (about 10 million points) which I am buffering, then joining, dowgrading and finally buffering and dissolving. This is all in a model which works fine with sample smaller ...