difficult to process and manage because the size is usually bigger than the limits software can normally deal with.

learn more… | top users | synonyms

25
votes
2answers
9k views

Looking for fastest solution for Point in Polygon analysis of 200 million points

Apologies for asking a question that has been asked before -- I've read through answers, and I'm left wondering how to implement the recommendations. I've never used SQL, Python, C, and have only ...
23
votes
12answers
24k views

Bulk Geocode 20 million records

Are there any free or reasonably priced databases for the US which can be searched and return latitude and longitude information?
11
votes
4answers
492 views

How to generalize coastlines?

I have a large set of data points that essentially represent the coastline of the world. I don't know the exact resolution but I'd say it's in the neighbourhood of 5-10m. Imagine a set of points ...
9
votes
2answers
8k views

Is there a difference in performance between Merge and Append in ArcGIS?

Often I find myself in a situation where I don't care whether or not my tool produces a new feature class, but I do care how long it takes to combine all of my large datasets. Does it take longer to ...
9
votes
1answer
445 views

Efficient raster sampling of billions of polygons (bounding boxes)

How can a raster be computed efficiently (in Python), given a set consisting of billions of bounding boxes (read sequentially from a file), and given that the raster values for each cell should give ...
8
votes
4answers
2k views

Queries returning very big datasets in PostGIS

I have a PostGIS query that will return several million rows: SELECT t1.id AS id1, t2.id AS id2, ABS(t1.mean_h - t2.mean_h) AS h_diff, ST_Distance(t1.the_geom, t2.the_geom) AS dist FROM ...
8
votes
1answer
152 views

Cluster processing - what resources to ask for 50 Gb DEM processing

For "renting purposes", I have to specify the computing resources I need from the computing center at my University. I have very little idea of what to ask for. Considering the information below may ...
8
votes
2answers
3k views

Is there an effective limit to the geoprocessing task Geocode Addresses?

The issue I face has the following background: We are geocoding an immense directory of records. It is something in excess of 100 million addresses. I have split the addresses up into as small a ...
7
votes
3answers
578 views

Thinning large LiDAR point cloud?

I have a large point cloud (LAS file) and I am trying to thin this using a complex technique. I know it is possible to thin a point cloud using simple techniques such as extracting every nth point. ...
7
votes
9answers
2k views

WMS with Access Control (ACL) for different users? Or solution for thousands of points on web map?

I have a set of Sites in the thousands. Not all users see same the same Sites on the map - thus ACL. Data is stored in a non-GIS database (mssql, ora, etc) and right now is retrieved with a standard ...
6
votes
2answers
2k views

Big Satellite Image Processing

Im trying to run Mort Canty's http://mcanty.homepage.t-online.de/ Python iMAD implementation on bitemporal RapidEye Multispectral images. Which basically calculates the canonical correlation for the ...
6
votes
2answers
1k views

memory issue when trying to buffer/union large dataset using postgis

I am trying to generated a dissolved buffer of a large point dataset (Ideally 29 million points - Address data in Great-Britain, but I receive the data by chunks of 1 million points, so these could be ...
5
votes
4answers
1k views

Need a location based social network data set

I'm looking for a data-set of Geosocial network consisting of Gps trajectories, chek-in,geotag contetnts and stuff like these available for research purposes. I'd really appreciate it if you can ...
4
votes
4answers
241 views

Is city-supplied GIS data sufficient for transit operators or do they need their own set of GIS data?

City GIS data is usually built in a format where streets/avenues/boulevards/etc are modeled as lines that are geographically located in the middle of the streets/avenues/boulevards/etc. Is that ...
4
votes
2answers
2k views

Adding data from a SQL Server Spatial database to ArcMap or ArcEngine application with ArcObjects for better performance

I have an ArcSDE geodatabase stored in MS SQL Server 2008 R2. The datasets in this geodatabase range from 10 record to 7 million records and there is a mixture of tables and spatial datasets (SQL ...
4
votes
2answers
197 views

Alternative to Spatial Query on large data

I have around 110000 points data spread around the world. the shapefile size is 116 Mb. I want to select points which are inside asia. I have been using qgis but this time qgis is not being able to ...
4
votes
3answers
386 views

Free good quality small scale (1:70,000) global coastline datasets?

I've been searching for a few days for the most accurate world coastline datasets.. Does anyone know where l can get coastlines at scales of 1:250 000, 1:2 000 000 and 1: 70 000. I've tried using the ...
4
votes
1answer
249 views

smallest, lossless, GDAL ready image format

I would like to know what the smallest file format is I can use with GDAL that fulfills the following criteria: lossless compression (relatively) easy to read/write with GDAL Python bindings least ...
4
votes
1answer
425 views

Where can I find a national MLS boundary dataset for the US?

Every 6 months or so I continue to revisit a topic I have never found an answer for. What I am looking for is some type (ANY TYPE) of national map which outlines the boundaries of each of the 1000+ ...
3
votes
3answers
259 views

How to find accurate latitudes and longitudes data for cities? [closed]

I want to put the coordinates of cities in a spatial db and then find cities from a given city at some given distance. I am hoping these functions would be readily available in a spatial db like ...
3
votes
1answer
89 views

Unexpected vertices from the ouput of ArcGIS union tool

I have found some unexpected behavior of ArcGIS union tool. My assumption was that this tool will never add any new vertex to the output layer, but this was not the case. I have run the ArcGIS union ...
3
votes
2answers
404 views

Create smaller OSM with only basic zoom levels data from larger OSM file

From a large .osm/.o5m file (for India), I only want to extract the data that would actually show up when one is looking from zoom levels 1/0 through 8 (which is quite zoomed out.. only major cities, ...
3
votes
2answers
601 views

Is there a way to speed up gIntersections for large spatial objects?

I'm attempting to perform a gIntersection of a big grid on a shapefile for country boundaries, so that I can calculate the % area of each grid cell that lies within the country. It takes a few seconds ...
3
votes
2answers
448 views

Stratified Random Sampling with large dataset in ArcGIS?

I want to create a Stratified Random Sample with a land cover classification map. The purpose for the Stratrified Random Sample is to produce an Accuracy Assessment. I am using ArcGIS 10.2 with access ...
3
votes
1answer
920 views

XY to Line tool crashing

The problem I'm encountering is simple. I have a shape file of points (about 8 million). They have three XY pairs that I am attempting to create XY lines from for purposes of measuring distance. ...
3
votes
1answer
55 views

Importing large .asc files into QGIS

I've been given two .asc files containing a 1m raster data from a LiDAR survey. I only have access to QGIS.2.14.1 (which I'am using for the 1st time only having previous experience with ArcGIS) and I ...
3
votes
1answer
779 views

Python coding question, looping through feature classes, AddField, CalculateField, then errors (might be schema lock)

This is not a SDE question. File GDB. Hi, I have a general question on looping through feature classes in a file geodatabase. I have (attached) a python script that adds a ton of fields and ...
3
votes
0answers
201 views

Memory issue inserting a large dataset in GDB with ArcObjects

I am using the ArcObjects API for Java to load a very large dataset in a File Geodatabase in one feature class. The whole dataset I have to load is composed of 200 datasets of 30k polygons, and I ...
3
votes
0answers
255 views

Performance issues Tilemill with PostgreSQL and large datasets

I have a Tilemill project the size of europe. I developed my style-sheet by using a smaller dataset (size auf Austria) and everything was perfect. But now I did apply the same styles to my big europe ...
3
votes
1answer
661 views

Online/cloud storage options for GIS file-based data [closed]

I am currently looking into online/cloud-based storage solutions in general and also for my GIS data which consists of shapefiles, rasters, file geodatabases, and the like (no relational databases). ...
2
votes
2answers
109 views

Splitting vector file by date (year) but column containing dates in Day/Month/Year format?

I am using national accident data in QGIS and quite frankly the dataset is huge as it contains every recorded accident between 2005 and 2014. I've split datasets by field for a number of vector layers....
2
votes
1answer
249 views

Are there any efficient methods for processing ArcGIS-based database tables outside of the ArcEtc program library?

Let us say, for the sake of argument, that you are working with a table that is over 2 GB in size (so .dbf is automatically ruled out... right?). To even create such a document, ArcGIS requires that ...
2
votes
1answer
1k views

ArcMap many-to-many join

I'm working in ArcMap 10 and can move around in R, though am not an expert. I have a shp file and an xlsx file. The shp file is 20,221 rows and includes a field GID (grid ID) which is not unique ...
2
votes
1answer
299 views

GeoServer: Show (Random?) Subset of Features

Background We have a GeoServer layer backed by a PostGIS table with 22 million smallish features covering the continental US. We're using GeoServer to serve the features as a WMS layer to a client ...
2
votes
0answers
468 views

Minidump Error Messages With Large Dataset

I'm using QGIS 2.8 on a system running Windows 8.1 (x64) with 12GB of RAM. I am attempting to run analyses of a very large database (.gdb format, 7.17GB). In attempting to run the "Join attributes by ...
2
votes
0answers
225 views

Arcpy script for summing pixel values in loop can't handle large number of rasters

I want to sum the pixel values of large numbers of rasters of different extents (ArcMap v10.2). 16GB RAM. I have the following code (with help from this site) which solves the different extent ...
2
votes
1answer
319 views

Can ArcGIS Online navigate and identify large GIS datasets? [closed]

For my upcoming project, I want to see if ArcGIS Online can provide a viable solution. Users will need to have access to webmap on their desktops with basic navigation and identify. This will be an ...
2
votes
1answer
125 views

Split data for OpenLayers

(I gave up. I already spend 10 hours reading internet and I am not any closer to get the job done.) If I start at the beginning: I have country map divided into 12 regions. I have country map ...
1
vote
1answer
2k views

Any tool to view large tiff images?

I'm looking for some application to view some large TIFF images and large mosaics consist of tiff images. At the moment I have VLIV for viewing large tiff files. And tools based on HD view: HD View ...
1
vote
3answers
127 views

Windows software to convert GeoJSON (> 6Mb) to Excel/CSV?

I am trying to convert OpenStreetmap data GeoJSON to Excel (or .csv files). Few online converters are there. However, when the file is big (example: 6Mb Geojson), the browser is becoming ...
1
vote
2answers
575 views

Using very large image formats with GDAL

I would like to use gdalwarp and gdal_translate with some very large raster images (313,040 x 313,040 px) and I'm wondering what the best format or method is to do this, if it's even possible. I have ...
1
vote
1answer
74 views

Working w/ Shapefiles Containing 50,000 to 300,000 Features

I work daily with shapefiles that are quite large in number of features using ArcGIS. Sometimes I am able to perform operations (joins, selections, exports, etc) on these shapefiles (though they do ...
1
vote
3answers
329 views

Is there an OSM High Level Dataset (pbf) available?

I'm currently trying to setup an offline Nominatim geocoder VM and I only need to have access high level data (countries, states and cities). Is there an OSM high level dataset (.pdf package) ...
1
vote
1answer
304 views

Large dataset processing via cloud computing

I need to process a large dataset. Aside from processing locally on my computer, can I install, for instance, arcgis10 on a cloud service like amazon. I don't have esri server, only arcgis10. I'm ...
1
vote
0answers
21 views

Creating a contiguous raster stack from different sources (vectors and DTM bands)

This is a conceptual question I've been trying to implement for a while, and not being very good at raster/data processing or python I'd like to hear best practices in dealing with an extent such as ...
1
vote
0answers
175 views

Speeding up ArcGIS arcpy.Merge_management for MANY files? [closed]

I'm currently running a script that is using the arcpy.Merge_management function to merge roughly ~1500 shapefiles together. When I tried this with much smaller test runs, it ran perfectly. Now ...
1
vote
0answers
179 views

Using gdal_merge to merge a large amount of files (in pairs)

I'm new to python and how it can be applied to gis. I have a large amount of MODIS files to go through and I've already extracted, reprojected and converted to GTiff the bands that I need from each ...
1
vote
0answers
70 views

Managing Very Large Vector File Geodatabases

I am currently looking for ways to more effectively work with an extremely large 10,000,000 (+) vector features contained in a file geodatabase. Its just very cumbersome to work with in ArcGIS (v 10....
1
vote
0answers
37 views

Printing large size with OpenLayers plugin basemaps?

trying to print A0, with one of the maplayers from the openlayers plugin, the resulting print misses tiles and is corrupted. any scale i choose there's no way except (i need >= 1.50.000) I tried to ...
1
vote
0answers
68 views

How to Buffer with Dissolve ALL on very large polyline dataset? [duplicate]

So I have a fairly large polyline network covering a decent chunk of North America, containing just shy of 900,000 table records. I want to do some very simple "GIS 101" kind of processing to it -- I ...