difficult to process and manage because the size is usually bigger than the limits software can normally deal with.

learn more… | top users | synonyms

0
votes
0answers
5 views

Loading a large osm file in osmar : why does loading time scale quadratically with file size?

I'm trying to load huge .osm files into R in order to run advanced analysis. I did not expect the loading part to be complex, but it proved difficult. The issue here is that the time it takes to ...
0
votes
2answers
34 views

How to efficiently extract time series from a very large multi-band GeoTiff in R?

First some context: I want to run a time-series analysis function that is only available in R for a continental sized multi-band time-series raster, which is stored in disk in GeoTiff format, that was ...
1
vote
3answers
616 views

Using very large image formats with GDAL

I would like to use gdalwarp and gdal_translate with some very large raster images (313,040 x 313,040 px) and I'm wondering what the best format or method is to do this, if it's even possible. I have ...
3
votes
1answer
59 views

Importing large .asc files into QGIS

I've been given two .asc files containing a 1m raster data from a LiDAR survey. I only have access to QGIS.2.14.1 (which I'am using for the 1st time only having previous experience with ArcGIS) and I ...
3
votes
3answers
270 views

How to find accurate latitudes and longitudes data for cities? [closed]

I want to put the coordinates of cities in a spatial db and then find cities from a given city at some given distance. I am hoping these functions would be readily available in a spatial db like ...
23
votes
12answers
25k views

Bulk Geocode 20 million records

Are there any free or reasonably priced databases for the US which can be searched and return latitude and longitude information?
0
votes
1answer
26 views

Identifying the closest polygon to a point dataset

I have two large data sets one point and one polygon. What I was looking to do was identify which polygon is closest to each point. Hopefully I was looking for a automated process as the point data ...
2
votes
2answers
118 views

Splitting vector file by date (year) but column containing dates in Day/Month/Year format?

I am using national accident data in QGIS and quite frankly the dataset is huge as it contains every recorded accident between 2005 and 2014. I've split datasets by field for a number of vector layers....
0
votes
0answers
33 views

Has anyone used the “Iterativly Load Parcel Topology into Parcel Fabric” tool recently?

I am trying to migrate our data (165,221 polygons and 677,338 lines) to a Local Government Information Model enabled parcel fabric. All the data and the empty LGIM fabric are hosted on the same SQL ...
1
vote
1answer
2k views

Any tool to view large tiff images?

I'm looking for some application to view some large TIFF images and large mosaics consist of tiff images. At the moment I have VLIV for viewing large tiff files. And tools based on HD view: HD View ...
0
votes
3answers
135 views

Rendering large amount of data in ArcMap? [closed]

Is there a way to load 17 big cadastar plans of urban area in ArcMap and make it "invisible" and use data only when printing to PDF? Total amount of data is too big to work with and I use it only as ...
1
vote
3answers
144 views

Windows software to convert GeoJSON (> 6Mb) to Excel/CSV?

I am trying to convert OpenStreetmap data GeoJSON to Excel (or .csv files). Few online converters are there. However, when the file is big (example: 6Mb Geojson), the browser is becoming ...
3
votes
2answers
480 views

Stratified Random Sampling with large dataset in ArcGIS?

I want to create a Stratified Random Sample with a land cover classification map. The purpose for the Stratrified Random Sample is to produce an Accuracy Assessment. I am using ArcGIS 10.2 with access ...
7
votes
3answers
628 views

Thinning large LiDAR point cloud?

I have a large point cloud (LAS file) and I am trying to thin this using a complex technique. I know it is possible to thin a point cloud using simple techniques such as extracting every nth point. ...
1
vote
0answers
22 views

Creating a contiguous raster stack from different sources (vectors and DTM bands)

This is a conceptual question I've been trying to implement for a while, and not being very good at raster/data processing or python I'd like to hear best practices in dealing with an extent such as ...
1
vote
0answers
187 views

Speeding up ArcGIS arcpy.Merge_management for MANY files? [closed]

I'm currently running a script that is using the arcpy.Merge_management function to merge roughly ~1500 shapefiles together. When I tried this with much smaller test runs, it ran perfectly. Now ...
2
votes
0answers
496 views

Minidump Error Messages With Large Dataset

I'm using QGIS 2.8 on a system running Windows 8.1 (x64) with 12GB of RAM. I am attempting to run analyses of a very large database (.gdb format, 7.17GB). In attempting to run the "Join attributes by ...
1
vote
0answers
182 views

Using gdal_merge to merge a large amount of files (in pairs)

I'm new to python and how it can be applied to gis. I have a large amount of MODIS files to go through and I've already extracted, reprojected and converted to GTiff the bands that I need from each ...
4
votes
1answer
257 views

smallest, lossless, GDAL ready image format

I would like to know what the smallest file format is I can use with GDAL that fulfills the following criteria: lossless compression (relatively) easy to read/write with GDAL Python bindings least ...
1
vote
1answer
74 views

Working w/ Shapefiles Containing 50,000 to 300,000 Features

I work daily with shapefiles that are quite large in number of features using ArcGIS. Sometimes I am able to perform operations (joins, selections, exports, etc) on these shapefiles (though they do ...
0
votes
2answers
391 views

How to reduce large dataset for processing by Tippecanoe and upload to Mapbox

I'm a masters student at the University of Washington without any background in geo, so if this question seems naive or unclear, I apologize. Please let me know if I can add any information for ...
3
votes
0answers
204 views

Memory issue inserting a large dataset in GDB with ArcObjects

I am using the ArcObjects API for Java to load a very large dataset in a File Geodatabase in one feature class. The whole dataset I have to load is composed of 200 datasets of 30k polygons, and I ...
2
votes
1answer
324 views

Can ArcGIS Online navigate and identify large GIS datasets? [closed]

For my upcoming project, I want to see if ArcGIS Online can provide a viable solution. Users will need to have access to webmap on their desktops with basic navigation and identify. This will be an ...
2
votes
1answer
1k views

ArcMap many-to-many join

I'm working in ArcMap 10 and can move around in R, though am not an expert. I have a shp file and an xlsx file. The shp file is 20,221 rows and includes a field GID (grid ID) which is not unique ...
1
vote
0answers
73 views

Managing Very Large Vector File Geodatabases

I am currently looking for ways to more effectively work with an extremely large 10,000,000 (+) vector features contained in a file geodatabase. Its just very cumbersome to work with in ArcGIS (v 10....
0
votes
2answers
162 views

Neighbors within distance from every row

I have a research project of 180,000 entries and I need to calculate the number of neighbors (points from the same table) that exist within a 300 meter radius for each point. My method is very slow, ...
2
votes
0answers
234 views

Arcpy script for summing pixel values in loop can't handle large number of rasters

I want to sum the pixel values of large numbers of rasters of different extents (ArcMap v10.2). 16GB RAM. I have the following code (with help from this site) which solves the different extent ...
3
votes
2answers
429 views

Create smaller OSM with only basic zoom levels data from larger OSM file

From a large .osm/.o5m file (for India), I only want to extract the data that would actually show up when one is looking from zoom levels 1/0 through 8 (which is quite zoomed out.. only major cities, ...
3
votes
1answer
89 views

Unexpected vertices from the ouput of ArcGIS union tool

I have found some unexpected behavior of ArcGIS union tool. My assumption was that this tool will never add any new vertex to the output layer, but this was not the case. I have run the ArcGIS union ...
3
votes
0answers
260 views

Performance issues Tilemill with PostgreSQL and large datasets

I have a Tilemill project the size of europe. I developed my style-sheet by using a smaller dataset (size auf Austria) and everything was perfect. But now I did apply the same styles to my big europe ...
0
votes
0answers
73 views

How to work with shapefile with 110,000 points?

I have a very large shapefile. It has 110,000 parcels, represented as points. I'm not trying to do anything too advanced with it. Here's what I want to do: Join it with an Excel file, export it as a ...
1
vote
0answers
39 views

Printing large size with OpenLayers plugin basemaps?

trying to print A0, with one of the maplayers from the openlayers plugin, the resulting print misses tiles and is corrupted. any scale i choose there's no way except (i need >= 1.50.000) I tried to ...
1
vote
3answers
335 views

Is there an OSM High Level Dataset (pbf) available?

I'm currently trying to setup an offline Nominatim geocoder VM and I only need to have access high level data (countries, states and cities). Is there an OSM high level dataset (.pdf package) ...
4
votes
2answers
202 views

Alternative to Spatial Query on large data

I have around 110000 points data spread around the world. the shapefile size is 116 Mb. I want to select points which are inside asia. I have been using qgis but this time qgis is not being able to ...
1
vote
0answers
69 views

How to Buffer with Dissolve ALL on very large polyline dataset? [duplicate]

So I have a fairly large polyline network covering a decent chunk of North America, containing just shy of 900,000 table records. I want to do some very simple "GIS 101" kind of processing to it -- I ...
2
votes
1answer
131 views

Split data for OpenLayers

(I gave up. I already spend 10 hours reading internet and I am not any closer to get the job done.) If I start at the beginning: I have country map divided into 12 regions. I have country map ...
3
votes
2answers
633 views

Is there a way to speed up gIntersections for large spatial objects?

I'm attempting to perform a gIntersection of a big grid on a shapefile for country boundaries, so that I can calculate the % area of each grid cell that lies within the country. It takes a few seconds ...
8
votes
2answers
3k views

Is there an effective limit to the geoprocessing task Geocode Addresses?

The issue I face has the following background: We are geocoding an immense directory of records. It is something in excess of 100 million addresses. I have split the addresses up into as small a ...
3
votes
1answer
786 views

Python coding question, looping through feature classes, AddField, CalculateField, then errors (might be schema lock)

This is not a SDE question. File GDB. Hi, I have a general question on looping through feature classes in a file geodatabase. I have (attached) a python script that adds a ton of fields and ...
1
vote
0answers
210 views

How to randomize colors for large datasets in QGIS new symbology?

Someone has already asked how to randomize colors by class in QGIS new symbology, but the answer there does not work with datasets containing more than 100 unique values since the Random option only ...
8
votes
1answer
154 views

Cluster processing - what resources to ask for 50 Gb DEM processing

For "renting purposes", I have to specify the computing resources I need from the computing center at my University. I have very little idea of what to ask for. Considering the information below may ...
2
votes
1answer
302 views

GeoServer: Show (Random?) Subset of Features

Background We have a GeoServer layer backed by a PostGIS table with 22 million smallish features covering the continental US. We're using GeoServer to serve the features as a WMS layer to a client ...
9
votes
1answer
451 views

Efficient raster sampling of billions of polygons (bounding boxes)

How can a raster be computed efficiently (in Python), given a set consisting of billions of bounding boxes (read sequentially from a file), and given that the raster values for each cell should give ...
6
votes
2answers
1k views

memory issue when trying to buffer/union large dataset using postgis

I am trying to generated a dissolved buffer of a large point dataset (Ideally 29 million points - Address data in Great-Britain, but I receive the data by chunks of 1 million points, so these could be ...
27
votes
2answers
9k views

Looking for fastest solution for Point in Polygon analysis of 200 million points

Apologies for asking a question that has been asked before -- I've read through answers, and I'm left wondering how to implement the recommendations. I've never used SQL, Python, C, and have only ...
9
votes
2answers
8k views

Is there a difference in performance between Merge and Append in ArcGIS?

Often I find myself in a situation where I don't care whether or not my tool produces a new feature class, but I do care how long it takes to combine all of my large datasets. Does it take longer to ...
1
vote
1answer
307 views

Large dataset processing via cloud computing

I need to process a large dataset. Aside from processing locally on my computer, can I install, for instance, arcgis10 on a cloud service like amazon. I don't have esri server, only arcgis10. I'm ...
6
votes
2answers
2k views

Big Satellite Image Processing

Im trying to run Mort Canty's http://mcanty.homepage.t-online.de/ Python iMAD implementation on bitemporal RapidEye Multispectral images. Which basically calculates the canonical correlation for the ...
4
votes
4answers
242 views

Is city-supplied GIS data sufficient for transit operators or do they need their own set of GIS data?

City GIS data is usually built in a format where streets/avenues/boulevards/etc are modeled as lines that are geographically located in the middle of the streets/avenues/boulevards/etc. Is that ...
4
votes
2answers
2k views

Adding data from a SQL Server Spatial database to ArcMap or ArcEngine application with ArcObjects for better performance

I have an ArcSDE geodatabase stored in MS SQL Server 2008 R2. The datasets in this geodatabase range from 10 record to 7 million records and there is a mixture of tables and spatial datasets (SQL ...