difficult to process and manage because the size is usually bigger than the limits software can normally deal with.
14
votes
9answers
3k views
Bulk Geocode 20 million records
Are there any free or reasonably priced databases for the US which can be searched and return latitude and longitude information?
13
votes
2answers
3k views
Looking for fastest solution for Point in Polygon analysis of 200 million points
Apologies for asking a question that has been asked before -- I've read through answers, and I'm left wondering how to implement the recommendations. I've never used SQL, Python, C, and have only ...
11
votes
4answers
344 views
How to generalize coastlines?
I have a large set of data points that essentially represent the coastline of the world. I don't know the exact resolution but I'd say it's in the neighbourhood of 5-10m.
Imagine a set of points ...
9
votes
1answer
299 views
Efficient raster sampling of billions of polygons (bounding boxes)
How can a raster be computed efficiently (in Python), given a set consisting of billions of bounding boxes (read sequentially from a file), and given that the raster values for each cell should give ...
8
votes
4answers
923 views
Queries returning very big datasets in PostGIS
I have a PostGIS query that will return several million rows:
SELECT
t1.id AS id1,
t2.id AS id2,
ABS(t1.mean_h - t2.mean_h) AS h_diff,
ST_Distance(t1.the_geom, t2.the_geom) AS dist
FROM ...
8
votes
2answers
614 views
Is there a difference in performance between Merge and Append in ArcGIS?
Often I find myself in a situation where I don't care whether or not my tool produces a new feature class, but I do care how long it takes to combine all of my large datasets. Does it take longer to ...
8
votes
1answer
95 views
Cluster processing - what resources to ask for 50 Gb DEM processing
For "renting purposes", I have to specify the computing resources I need from the computing center at my University. I have very little idea of what to ask for.
Considering the information below may ...
7
votes
9answers
1k views
WMS with Access Control (ACL) for different users? Or solution for thousands of points on web map?
I have a set of Sites in the thousands. Not all users see same the same Sites on the map - thus ACL. Data is stored in a non-GIS database (mssql, ora, etc) and right now is retrieved with a standard ...
6
votes
2answers
1k views
Is there an effective limit to the geoprocessing task Geocode Addresses?
The issue I face has the following background:
We are geocoding an immense directory of records. It is something in excess of 100 million addresses. I have split the addresses up into as small a ...
5
votes
2answers
509 views
Big Satellite Image Processing
Im trying to run Mort Canty's http://mcanty.homepage.t-online.de/ Python iMAD implementation on bitemporal RapidEye Multispectral images. Which basically calculates the canonical correlation for the ...
4
votes
4answers
203 views
Is city-supplied GIS data sufficient for transit operators or do they need their own set of GIS data?
City GIS data is usually built in a format where streets/avenues/boulevards/etc are modeled as lines that are geographically located in the middle of the streets/avenues/boulevards/etc. Is that ...
4
votes
2answers
851 views
Adding data from a SQL Server Spatial database to ArcMap or ArcEngine application with ArcObjects for better performance
I have an ArcSDE geodatabase stored in MS SQL Server 2008 R2. The datasets in this geodatabase range from 10 record to 7 million records and there is a mixture of tables and spatial datasets (SQL ...
4
votes
2answers
228 views
memory issue when trying to buffer/union large dataset using postgis
I am trying to generated a dissolved buffer of a large point dataset (Ideally 29 million points - Address data in Great-Britain, but I receive the data by chunks of 1 million points, so these could be ...
4
votes
4answers
518 views
Need a location based social network data set
I'm looking for a data-set of Geosocial network consisting of Gps trajectories, chek-in,geotag contetnts and stuff like these available for research purposes.
I'd really appreciate it if you can ...
4
votes
3answers
231 views
Free good quality small scale (1:70,000) global coastline datasets?
I've been searching for a few days for the most accurate world coastline datasets.. Does anyone know where l can get coastlines at scales of 1:250 000, 1:2 000 000 and 1: 70 000. I've tried using the ...
4
votes
1answer
133 views
Where can I find a national MLS boundary dataset for the US?
Every 6 months or so I continue to revisit a topic I have never found an answer for. What I am looking for is some type (ANY TYPE) of national map which outlines the boundaries of each of the 1000+ ...
3
votes
0answers
109 views
Python coding question, looping through feature classes, AddField, CalculateField, then errors (might be schema lock)
This is not a SDE question. File GDB.
Hi, I have a general question on looping through feature classes in a file geodatabase. I have (attached) a python script that adds a ton of fields and ...
3
votes
1answer
315 views
Online/cloud storage options for GIS file-based data [closed]
I am currently looking into online/cloud-based storage solutions in general and also for my GIS data which consists of shapefiles, rasters, file geodatabases, and the like (no relational databases). ...
2
votes
1answer
178 views
Are there any efficient methods for processing ArcGIS-based database tables outside of the ArcEtc program library?
Let us say, for the sake of argument, that you are working with a table that is over 2 GB in size (so .dbf is automatically ruled out... right?). To even create such a document, ArcGIS requires that ...
2
votes
1answer
548 views
XY to Line tool crashing
The problem I'm encountering is simple. I have a shape file of points (about 8 million). They have three XY pairs that I am attempting to create XY lines from for purposes of measuring distance. ...
2
votes
1answer
118 views
GeoServer: Show (Random?) Subset of Features
Background
We have a GeoServer layer backed by a PostGIS table with 22 million smallish features covering the continental US. We're using GeoServer to serve the features as a WMS layer to a client ...
1
vote
1answer
124 views
Large dataset processing via cloud computing
I need to process a large dataset. Aside from processing locally on my computer, can I install, for instance, arcgis10 on a cloud service like amazon. I don't have esri server, only arcgis10.
I'm ...
1
vote
0answers
24 views
How to randomize colors for large datasets in QGIS new symbology?
Someone has already asked how to randomize colors by class in QGIS new symbology, but the answer there does not work with datasets containing more than 100 unique values since the Random option only ...
0
votes
0answers
38 views
Use Overviews to Select Area of Interest in Very Large GIS Rasters/Images (e.g., GeoTiff's)
I am working with large images in Python and Gdal on a Windows machine and wondering if it is possible to directly access the specific smaller overviews I built into them using gdladdo. I am viewing ...
