difficult to process and manage because the size is usually bigger than the limits software can normally deal with.

learn more… | top users | synonyms

2
votes
1answer
107 views

Any tool to view large tiff images?

I'm looking for some application to view some large TIFF images and large mosaics consist of tiff images. At the moment I have VLIV for viewing large tiff files. And tools based on HD view: HD View ...
1
vote
0answers
31 views

Performance issues Tilemill with PostgreSQL and large datasets

I have a Tilemill project the size of europe. I developed my style-sheet by using a smaller dataset (size auf Austria) and everything was perfect. But now I did apply the same styles to my big europe ...
1
vote
2answers
59 views

Using very large image formats with GDAL

I would like to use gdalwarp and gdal_translate with some very large raster images (313,040 x 313,040 px) and I'm wondering what the best format or method is to do this, if it's even possible. I have ...
0
votes
0answers
69 views

How to work with shapefile with 110,000 points?

I have a very large shapefile. It has 110,000 parcels, represented as points. I'm not trying to do anything too advanced with it. Here's what I want to do: Join it with an Excel file, export it as a ...
1
vote
0answers
19 views

Printing large size with OpenLayers plugin basemaps?

trying to print A0, with one of the maplayers from the openlayers plugin, the resulting print misses tiles and is corrupted. any scale i choose there's no way except (i need >= 1.50.000) I tried to ...
1
vote
3answers
76 views

Is there an OSM High Level Dataset (pbf) available?

I'm currently trying to setup an offline Nominatim geocoder VM and I only need to have access high level data (countries, states and cities). Is there an OSM high level dataset (.pdf package) ...
5
votes
2answers
102 views

Alternative to Spatial Query on large data

I have around 110000 points data spread around the world. the shapefile size is 116 Mb. I want to select points which are inside asia. I have been using qgis but this time qgis is not being able to ...
0
votes
0answers
30 views

Things to consider while and after merging DEM-Tiles?

In some tutorials it is said that after merging you should stretch the “histogram to full dataset” (also mentioned in the QGIS Training Manual). Is there anything else you would suggest doing after ...
3
votes
1answer
173 views

Can ArcGIS Online navigate and identify large GIS datasets?

For my upcoming project, I want to see if ArcGIS Online can provide a viable solution. Users will need to have access to webmap on their desktops with basic navigation and identify. This will be an ...
1
vote
0answers
56 views

How to Buffer with Dissolve ALL on very large polyline dataset? [duplicate]

So I have a fairly large polyline network covering a decent chunk of North America, containing just shy of 900,000 table records. I want to do some very simple "GIS 101" kind of processing to it -- I ...
2
votes
1answer
64 views

Split data for OpenLayers

(I gave up. I already spend 10 hours reading internet and I am not any closer to get the job done.) If I start at the beginning: I have country map divided into 12 regions. I have country map ...
3
votes
2answers
147 views

Is there a way to speed up gIntersections for large spatial objects?

I'm attempting to perform a gIntersection of a big grid on a shapefile for country boundaries, so that I can calculate the % area of each grid cell that lies within the country. It takes a few seconds ...
1
vote
0answers
86 views

How to randomize colors for large datasets in QGIS new symbology?

Someone has already asked how to randomize colors by class in QGIS new symbology, but the answer there does not work with datasets containing more than 100 unique values since the Random option only ...
3
votes
1answer
525 views

Python coding question, looping through feature classes, AddField, CalculateField, then errors (might be schema lock)

This is not a SDE question. File GDB. Hi, I have a general question on looping through feature classes in a file geodatabase. I have (attached) a python script that adds a ton of fields and ...
2
votes
1answer
237 views

GeoServer: Show (Random?) Subset of Features

Background We have a GeoServer layer backed by a PostGIS table with 22 million smallish features covering the continental US. We're using GeoServer to serve the features as a WMS layer to a client ...
8
votes
1answer
132 views

Cluster processing - what resources to ask for 50 Gb DEM processing

For "renting purposes", I have to specify the computing resources I need from the computing center at my University. I have very little idea of what to ask for. Considering the information below may ...
5
votes
2answers
601 views

memory issue when trying to buffer/union large dataset using postgis

I am trying to generated a dissolved buffer of a large point dataset (Ideally 29 million points - Address data in Great-Britain, but I receive the data by chunks of 1 million points, so these could be ...
9
votes
2answers
3k views

Is there a difference in performance between Merge and Append in ArcGIS?

Often I find myself in a situation where I don't care whether or not my tool produces a new feature class, but I do care how long it takes to combine all of my large datasets. Does it take longer to ...
1
vote
1answer
184 views

Large dataset processing via cloud computing

I need to process a large dataset. Aside from processing locally on my computer, can I install, for instance, arcgis10 on a cloud service like amazon. I don't have esri server, only arcgis10. I'm ...
6
votes
2answers
987 views

Big Satellite Image Processing

Im trying to run Mort Canty's http://mcanty.homepage.t-online.de/ Python iMAD implementation on bitemporal RapidEye Multispectral images. Which basically calculates the canonical correlation for the ...
20
votes
2answers
5k views

Looking for fastest solution for Point in Polygon analysis of 200 million points

Apologies for asking a question that has been asked before -- I've read through answers, and I'm left wondering how to implement the recommendations. I've never used SQL, Python, C, and have only ...
18
votes
9answers
10k views

Bulk Geocode 20 million records

Are there any free or reasonably priced databases for the US which can be searched and return latitude and longitude information?
9
votes
1answer
362 views

Efficient raster sampling of billions of polygons (bounding boxes)

How can a raster be computed efficiently (in Python), given a set consisting of billions of bounding boxes (read sequentially from a file), and given that the raster values for each cell should give ...
4
votes
4answers
225 views

Is city-supplied GIS data sufficient for transit operators or do they need their own set of GIS data?

City GIS data is usually built in a format where streets/avenues/boulevards/etc are modeled as lines that are geographically located in the middle of the streets/avenues/boulevards/etc. Is that ...
8
votes
2answers
2k views

Is there an effective limit to the geoprocessing task Geocode Addresses?

The issue I face has the following background: We are geocoding an immense directory of records. It is something in excess of 100 million addresses. I have split the addresses up into as small a ...
4
votes
2answers
1k views

Adding data from a SQL Server Spatial database to ArcMap or ArcEngine application with ArcObjects for better performance

I have an ArcSDE geodatabase stored in MS SQL Server 2008 R2. The datasets in this geodatabase range from 10 record to 7 million records and there is a mixture of tables and spatial datasets (SQL ...
4
votes
1answer
225 views

Where can I find a national MLS boundary dataset for the US?

Every 6 months or so I continue to revisit a topic I have never found an answer for. What I am looking for is some type (ANY TYPE) of national map which outlines the boundaries of each of the 1000+ ...
4
votes
4answers
791 views

Need a location based social network data set

I'm looking for a data-set of Geosocial network consisting of Gps trajectories, chek-in,geotag contetnts and stuff like these available for research purposes. I'd really appreciate it if you can ...
11
votes
4answers
416 views

How to generalize coastlines?

I have a large set of data points that essentially represent the coastline of the world. I don't know the exact resolution but I'd say it's in the neighbourhood of 5-10m. Imagine a set of points ...
2
votes
1answer
789 views

XY to Line tool crashing

The problem I'm encountering is simple. I have a shape file of points (about 8 million). They have three XY pairs that I am attempting to create XY lines from for purposes of measuring distance. ...
3
votes
1answer
429 views

Online/cloud storage options for GIS file-based data [closed]

I am currently looking into online/cloud-based storage solutions in general and also for my GIS data which consists of shapefiles, rasters, file geodatabases, and the like (no relational databases). ...
4
votes
3answers
293 views

Free good quality small scale (1:70,000) global coastline datasets?

I've been searching for a few days for the most accurate world coastline datasets.. Does anyone know where l can get coastlines at scales of 1:250 000, 1:2 000 000 and 1: 70 000. I've tried using the ...
2
votes
1answer
214 views

Are there any efficient methods for processing ArcGIS-based database tables outside of the ArcEtc program library?

Let us say, for the sake of argument, that you are working with a table that is over 2 GB in size (so .dbf is automatically ruled out... right?). To even create such a document, ArcGIS requires that ...
7
votes
9answers
1k views

WMS with Access Control (ACL) for different users? Or solution for thousands of points on web map?

I have a set of Sites in the thousands. Not all users see same the same Sites on the map - thus ACL. Data is stored in a non-GIS database (mssql, ora, etc) and right now is retrieved with a standard ...
8
votes
4answers
1k views

Queries returning very big datasets in PostGIS

I have a PostGIS query that will return several million rows: SELECT t1.id AS id1, t2.id AS id2, ABS(t1.mean_h - t2.mean_h) AS h_diff, ST_Distance(t1.the_geom, t2.the_geom) AS dist FROM ...