Questions tagged [large-datasets]

difficult to process and manage because the size is usually bigger than the limits software can normally deal with.

Filter by
Sorted by
Tagged with
1 vote
1 answer
2k views

Saving raster to BigTIFF in QGIS?

Using QGIS 3.4, I am trying to merge multiple DEMs downloaded from the National Map Viewer that are in ArcGRID format (Arc/Info Binary Grid). The output will be a large file so I can't use any of the ...
  • 450
8 votes
3 answers
8k views

Converting large GeoJSON to shapefile?

I have recently downloaded a building footprint GeoJSON file from Microsoft Bing Maps. The file I downloaded was for the entirety of Texas, and is of approximately 2.2 GB file size. They do not ...
1 vote
1 answer
278 views

Using "Difference" geoprocessing tool on Big Data taking a very long time

I am using QGIS Bonn 3.2.3. I've recently been using the Difference function to calculate a list of points and it is taking a terribly long time. I was wondering if you could point out any obvious ...
1 vote
0 answers
521 views

GeoServer GetMap with geopackage very slow (with high zoom levels)

I have a GeoServer instance and have imported multiple high res tiff files as layers and created a layergroup from the layers. When I call the WMS endpoint with a specified bbox and the geopackage ...
0 votes
0 answers
214 views

Improving performance of Eliminate tool in ArcGIS Pro?

My data is very large (polygon feature class with 884187 features), thats why Python geoprocessing tools (which use geoprocesses like Intersect, Buffer etc.) don't work due to performance issues. So ...
0 votes
0 answers
592 views

Merging multiple rasters (Very large dataset)

I am working on project to create a DSM raster for whole Kentucky. I will have over 46000 tiff tiles and each one is about 10 Mb and one band. Normally, I use Mosaic to New Raster Tool but this time ...
  • 493
3 votes
2 answers
2k views

Calculating time efficient seasonal means from raster bricks?

I'm starting from a Rasterbrick like that called r.brick: class : RasterBrick dimensions : 17, 19, 323, 21915 (nrow, ncol, ncell, nlayers) resolution : 0.11, 0.11 (x, y) extent : 8....
  • 91
1 vote
1 answer
114 views

Are there any continental-scale flow accumulation products besides Hydrosheds?

I am trying to compute branching angles of drainage networks for roughly the western half of the U.S. Because many of these streams are non-perennial (don't have water all year), some of the standard ...
  • 2,854
0 votes
1 answer
2k views

Extract values from large rasterstack, R

I have a large rasterstack of about 500 objects. I want to sample easch object with the use of a SpatialPointDataframe and then export the results as file.csv. I have tried the following but within ...
  • 594
2 votes
1 answer
242 views

Working with Sentinel-2 data and aggrigation in R

So I was wandering if there is a way to simplify a couple of Sentinel's rasters so I can apply raster analysis on them. Problem is that whenever I try to aggregate the data my RAM, CPU and storage ...
  • 447
2 votes
0 answers
339 views

Dealing with Very Large Satellite Imagery

I'm curious if anyone has suggestions for viewing and processing really large imagery. I'm pretty experienced with desktop apps like ENVI and am comfortable with python and GDAL but I'm analyzing ...
  • 379
3 votes
1 answer
839 views

Displaying LiDAR data with too many points in ArcMap?

I have imported LiDAR data into ArcMap in the shape of a Las DataSet with twelve .las files (contains over 100 million data points). I believe the file itself is working properly. However, no points ...
  • 153
2 votes
2 answers
1k views

Dice a polygon with over 21 million vertices in ArcMap?

Based on posts like this I suspect I'm bumping up against issues when trying to work with a feature class that contains features with a single very complicated polygon. These errors include 000426, ...
  • 331
0 votes
0 answers
94 views

Why does myRaster[1] <- 5 takes a very long time for a large raster?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
  • 696
1 vote
0 answers
359 views

Unable to create Cache for Large Vector

I have uploaded a large vector data (20 million records) to Postgres; published the data on Geoserver. But, when I try to create cache for the same it throws GC overhead limit exceeded error. I have ...
1 vote
1 answer
254 views

Calculate nearest town for 2900 lot centroids

I have 16 town point shapefiles. I have a layer of 2900 centroids for lots. I need to determine the nearest town for each of the centroids. The option I came up with is to Split by Attribute by ...
7 votes
1 answer
1k views

How do you speed up creating new field in attribute table for huge amount of data in QGIS?

In my attribute table for a shapefile, I am converting one field (Julian day) to another field (date) using the field calculator but I'm working with about 7.7 million rows. QGIS has been running for ...
5 votes
1 answer
439 views

Calculating distance between multiple associated points in QGIS

I have two large lists of points. List A contains the location of a farmers farmstead and list B contains locations of pastures. Both lists have a column that will identify the farmer that the pasture ...
2 votes
2 answers
232 views

How do apps like zillow serve up data?

I'm a total newbie to GIS, but attempting to build an app similar to zillows that quickly serves up data based on user input. Currently, i'm collecting the data via shp files which are being store ...
  • 23
3 votes
2 answers
471 views

How can I perform a Join by Location for a large vector and very large point file?

I have no programming skills, but am thinking I may need to at least figure out how to run this one script from inside QGIS. I have a vector file of global quarter degree grid cells and a point file ...
0 votes
2 answers
781 views

Processing a large number of raster files in QGIS

I have a large number (100,000+) of raster files I need to process in QGIS (exporting to tiff, clipping to polygon layers, etc), for a total of approx 500GB. Can anyone advise if there's some way of ...
  • 886
0 votes
1 answer
883 views

Joining CSV-Generated Datapoints with Shapefiles in ArcGIS Desktop?

I have a question about joining data points (with a large amount of information) with shapefiles in ArcGIS Desktop: I have imported a CSV file into GIS. This CSV File is quite large: each row ...
  • 61
2 votes
1 answer
400 views

Split millions of points in shp with 100'000 points each [closed]

I havie over 8 million points in a shapefile. Now I would like to split these 8 mio points into parts of 100'000 points each, in order to facilitate further calculations with these points. In the end ...
  • 71
0 votes
1 answer
846 views

Bing maps and large datasets

We were using Bing maps to place pins on a map when a user loads a page. The number of pins has grown significantly so we started to cluster the pins to reduce the load on the client. Now even ...
  • 101
2 votes
1 answer
243 views

Problems intersecting large polygon shapefiles in QGIS (V2.18.2)

I am using the BirdLife International .gdb file (polygons) of bird species distributions to try and make a species richness map across the Brazilian Amazon. To do this I have clipped the global ...
  • 21
2 votes
2 answers
2k views

Showing huge GIS data on map using Leaflet?

I have a huge amount of GIS data in a database. I am fetching this data and converting it into GeoJSON before viewing the map. I am fetching all of the data to meet the requirements, which is taking a ...
1 vote
0 answers
817 views

QGIS: Raster to vector too large

I am working with a DTM of 5x5m for a large area (already clipped it). I calculated the slopes and want to maintain only the slopes above 2 degrees. I already used the raster calculator to set all ...
2 votes
1 answer
618 views

Loading a large osm file in osmar : why does loading time scale quadratically with file size?

I'm trying to load huge .osm files into R in order to run advanced analysis. I did not expect the loading part to be complex, but it proved difficult. The issue here is that the time it takes to ...
1 vote
2 answers
2k views

How to efficiently extract time series from a very large multi-band GeoTIFF in R?

First some context: I want to run a time-series analysis function that is only available in R for a continental sized multi-band time-series raster, which is stored in disk in GeoTIFF format, that was ...
3 votes
1 answer
762 views

Importing large .asc files into QGIS

I've been given two .asc files containing a 1m raster data from a LiDAR survey. I only have access to QGIS.2.14.1 (which I'am using for the 1st time only having previous experience with ArcGIS) and I ...
  • 31
0 votes
1 answer
229 views

Identifying the closest polygon to a point dataset

I have two large data sets one point and one polygon. What I was looking to do was identify which polygon is closest to each point. Hopefully I was looking for a automated process as the point data ...
0 votes
3 answers
426 views

Rendering large amount of data in ArcMap? [closed]

Is there a way to load 17 big cadastar plans of urban area in ArcMap and make it "invisible" and use data only when printing to PDF? Total amount of data is too big to work with and I use it only as ...
  • 181
1 vote
3 answers
3k views

Windows software to convert GeoJSON (> 6Mb) to Excel/CSV?

I am trying to convert OpenStreetmap data GeoJSON to Excel (or .csv files). Few online converters are there. However, when the file is big (example: 6Mb Geojson), the browser is becoming ...
  • 31
9 votes
4 answers
6k views

Thinning large LiDAR point cloud?

I have a large point cloud (LAS file) and I am trying to thin this using a complex technique. I know it is possible to thin a point cloud using simple techniques such as extracting every nth point. ...
  • 149
1 vote
0 answers
95 views

Creating a contiguous raster stack from different sources (vectors and DTM bands)

This is a conceptual question I've been trying to implement for a while, and not being very good at raster/data processing or python I'd like to hear best practices in dealing with an extent such as ...
5 votes
2 answers
2k views

Splitting vector file by date (year) but column containing dates in Day/Month/Year format?

I am using national accident data in QGIS and quite frankly the dataset is huge as it contains every recorded accident between 2005 and 2014. I've split datasets by field for a number of vector layers....
3 votes
3 answers
1k views

How to find accurate latitudes and longitudes data for cities? [closed]

I want to put the coordinates of cities in a spatial db and then find cities from a given city at some given distance. I am hoping these functions would be readily available in a spatial db like ...
1 vote
0 answers
629 views

Speeding up ArcGIS arcpy.Merge_management for MANY files? [closed]

I'm currently running a script that is using the arcpy.Merge_management function to merge roughly ~1500 shapefiles together. When I tried this with much smaller test runs, it ran perfectly. Now ...
  • 51
3 votes
0 answers
886 views

Minidump Error Messages With Large Dataset

I'm using QGIS 2.8 on a system running Windows 8.1 (x64) with 12GB of RAM. I am attempting to run analyses of a very large database (.gdb format, 7.17GB). In attempting to run the "Join attributes by ...
1 vote
0 answers
521 views

Using gdal_merge to merge a large amount of files (in pairs)

I'm new to python and how it can be applied to gis. I have a large amount of MODIS files to go through and I've already extracted, reprojected and converted to GTiff the bands that I need from each ...
1 vote
1 answer
90 views

Working w/ Shapefiles Containing 50,000 to 300,000 Features

I work daily with shapefiles that are quite large in number of features using ArcGIS. Sometimes I am able to perform operations (joins, selections, exports, etc) on these shapefiles (though they do ...
2 votes
2 answers
2k views

How to reduce large dataset for processing by Tippecanoe and upload to Mapbox

I'm a masters student at the University of Washington without any background in geo, so if this question seems naive or unclear, I apologize. Please let me know if I can add any information for ...
  • 21
4 votes
2 answers
2k views

Stratified Random Sampling with large dataset in ArcGIS?

I want to create a Stratified Random Sample with a land cover classification map. The purpose for the Stratrified Random Sample is to produce an Accuracy Assessment. I am using ArcGIS 10.2 with access ...
3 votes
0 answers
342 views

Memory issue inserting a large dataset in GDB with ArcObjects

I am using the ArcObjects API for Java to load a very large dataset in a File Geodatabase in one feature class. The whole dataset I have to load is composed of 200 datasets of 30k polygons, and I ...
1 vote
0 answers
188 views

Managing Very Large Vector File Geodatabases

I am currently looking for ways to more effectively work with an extremely large 10,000,000 (+) vector features contained in a file geodatabase. Its just very cumbersome to work with in ArcGIS (v 10....
  • 399
2 votes
2 answers
1k views

Neighbors within distance from every row

I have a research project of 180,000 entries and I need to calculate the number of neighbors (points from the same table) that exist within a 300 meter radius for each point. My method is very slow, ...
  • 127
5 votes
1 answer
976 views

smallest, lossless, GDAL ready image format

I would like to know what the smallest file format is I can use with GDAL that fulfills the following criteria: lossless compression (relatively) easy to read/write with GDAL Python bindings least ...
  • 9,675
5 votes
2 answers
2k views

Creating smaller OSM with only basic zoom levels data from larger OSM file?

From a large .osm/.o5m file (for India), I only want to extract the data that would actually show up when one is looking from zoom levels 1/0 through 8 (which is quite zoomed out.. only major cities, ...
  • 956
2 votes
0 answers
449 views

Arcpy script for summing pixel values in loop can't handle large number of rasters

I want to sum the pixel values of large numbers of rasters of different extents (ArcMap v10.2). 16GB RAM. I have the following code (with help from this site) which solves the different extent ...
  • 21
3 votes
1 answer
113 views

Unexpected vertices from the ouput of ArcGIS union tool

I have found some unexpected behavior of ArcGIS union tool. My assumption was that this tool will never add any new vertex to the output layer, but this was not the case. I have run the ArcGIS union ...
  • 1,906