Questions tagged [large-datasets]

difficult to process and manage because the size is usually bigger than the limits software can normally deal with.

Filter by
Sorted by
Tagged with
2
votes
2answers
68 views

Publishing a large and complex map to the web

I have created a complex map of my area and I wish to publish it as an interactive map on the web. The dataset appears to be too large for QGIS2Web to work - the script completes eventually but in ...
1
vote
0answers
33 views

Heatmap of a large polyline dataset in QGIS

To summarize: I'm having difficulties with creating a heatmap from a large (90021-item) polyline dataset in QGIS 3.10.1. First, it is worth noting that this question is somewhat similar to Doing ...
1
vote
0answers
39 views

Merging and snapping shapefiles

I have several large shapefiles from various sources. All cover various sections of oceans and many overlaps, as such the coastline is repeated across the several shapefiles but to varying levels of ...
0
votes
1answer
68 views

Serving large (1.7Tb) dataset of aerial images with GeoServer

I have a large dataset of aerial images consisting of 8000 TIFFs where single images are no larger than 0.5G and all have have .ovr pyramid files with them. Previously I have served this dataset as ...
2
votes
3answers
140 views

Convert large OSM files to GeoJSON

I have been trying to convert a large OSM file (5G) to the GeoJSON format. All the tools I have been using, use extensive memory and throw an out of memory exception. What is the best way to convert ...
1
vote
0answers
41 views

Leaflet: Load GeoJSON point data in polygon, when selecting polygon

I've got a point dataset (~ 160.000 points) which has a filesize of around 30 MB. Because that is too large I decided to slit it into 12 separate GeoJSON files. I've got a map with 12 polygons. When ...
0
votes
1answer
23 views

Creating a layer with selected values (divided by commas) from another layer - query or model needed

I have a very large dataset in ArcMap 10.6 and I would like to create a subset that selects only rows that have specific attributes (some of them with special characters) from a field that in my case ...
1
vote
0answers
38 views

Extracting or masking of very large raster to reduce file size on disk?

I have a very large 8-band raster (339806 x 61704) that I cannot load to memory to manipulate (R, raster:canProcessInMemory indicates I need 4999.16 GB of ram to load this file). However, I only need ...
0
votes
1answer
39 views

Counting points contained in polygon for large number of points in ArcMap?

I'm trying to determine the number of points within a set of polygons for a large number of points. There are 2GB of lat-long coordinates in a CSV file, and I want to see how many are contained in ...
4
votes
3answers
4k views

Converting large GeoJSON to shapefile?

I have recently downloaded a building footprint GeoJSON file from Microsoft Bing Maps. The file I downloaded was for the entirety of Texas, and is of approximately 2.2 GB file size. They do not ...
0
votes
1answer
99 views

Reduce processing time for large GeoTIFF image

I am trying to process large GeoTIFF images (128 MB) using GeoTIFF reader provided by Geotools library. However the processing time is very high, can anyone suggest how to reduce this time. Normal ...
1
vote
1answer
36 views

Why does geodataframe never complete?

I had a feature class made up of 1,700,000 polygons. I used Geopandas to create a geodataframe: state = "MD" state_gdb = r"C:\Projects\Pop_Alloc\{}_Data.gdb".format(state) join_feat = "{}...
12
votes
3answers
15k views

Measuring performance difference between Merge and Append in ArcGIS Desktop?

Often I find myself in a situation where I don't care whether or not my tool produces a new feature class, but I do care how long it takes to combine all of my large datasets. Does it take longer to ...
5
votes
2answers
1k views

Creating smaller OSM with only basic zoom levels data from larger OSM file?

From a large .osm/.o5m file (for India), I only want to extract the data that would actually show up when one is looking from zoom levels 1/0 through 8 (which is quite zoomed out.. only major cities, ...
2
votes
2answers
121 views

How do apps like zillow serve up data?

I'm a total newbie to GIS, but attempting to build an app similar to zillows that quickly serves up data based on user input. Currently, i'm collecting the data via shp files which are being store ...
0
votes
0answers
56 views

dotspatial Load in the map from large shapefile with query

Can I load only a little set of data from a large shapefile with a query in dotspatial c#? I think something like "Definition query" of arcmap.
0
votes
0answers
17 views

Prevent slowing of story map from large data?

I have a building layer that has about 9k polygons and it's slowing my story map. I've tried simplifying the polygon as much as possible, but no luck. Has anyone figured out a method to work with ...
1
vote
0answers
69 views

R process: Zonal Stat Raster has No Data Values vs Arc output

I am trying to re-create my ArcGIS Zonal Stat model with R All my files will be 30m pixels; 18000 x 18000. I have used R two different ways to create the output and both create NoData values in the ...
0
votes
0answers
27 views

Union very large layer in FGDB and only keep one polygon that is overlapping?

I'm trying to union an exceptionally large layer (45k features) of footprints with many of them largely overlapping. I'd like to union this layer but I'm having trouble due to the size of it. The end ...
2
votes
1answer
425 views

How to snap to a geojson-vt layer (Vector Tile Layer)?

I'm using geojson-vt to render a large geojson (originated from OSM) file. This file contains all road from a state (extracted with overpass-turbo way[highway="*"]). I'm buinding a domain-specific ...
3
votes
2answers
123 views

Script I made for pyQgis runs slow but can't find another way around it

registry = QgsProject.instance() first_layer = registry.mapLayersByName('buffered')[0] second_layer = registry.mapLayersByName('buffered')[0] areas = [] instrument_number = "tin_ttl_in" proprietor =...
8
votes
4answers
4k views

Thinning large LiDAR point cloud?

I have a large point cloud (LAS file) and I am trying to thin this using a complex technique. I know it is possible to thin a point cloud using simple techniques such as extracting every nth point. ...
0
votes
0answers
56 views

ArcMap fails to convert all records from CSV file using Table to Table geoprocessing tool

I am using the table to table geoprocessing tool to convert a CSV file with nearly 5 million records into an GDB table in a file geodatabase. The tool succeeds each time, but only 603 records are ...
1
vote
1answer
510 views

Saving raster to BigTIFF in QGIS?

Using QGIS 3.4, I am trying to merge multiple DEMs downloaded from the National Map Viewer that are in ArcGRID format (Arc/Info Binary Grid). The output will be a large file so I can't use any of the ...
1
vote
1answer
39 views

Using “Difference” geoprocessing tool on Big Data taking a very long time

I am using QGIS Bonn 3.2.3. I've recently been using the Difference function to calculate a list of points and it is taking a terribly long time. I was wondering if you could point out any obvious ...
0
votes
0answers
95 views

Improving performance of Eliminate tool in ArcGIS Pro?

My data is very large (polygon feature class with 884187 features), thats why Python geoprocessing tools (which use geoprocesses like Intersect, Buffer etc.) don't work due to performance issues. So ...
1
vote
0answers
220 views

GeoServer GetMap with geopackage very slow (with high zoom levels)

I have a GeoServer instance and have imported multiple high res tiff files as layers and created a layergroup from the layers. When I call the WMS endpoint with a specified bbox and the geopackage ...
2
votes
2answers
855 views

Calculating time efficient seasonal means from raster bricks?

I'm starting from a Rasterbrick like that called r.brick: class : RasterBrick dimensions : 17, 19, 323, 21915 (nrow, ncol, ncell, nlayers) resolution : 0.11, 0.11 (x, y) extent : 8....
0
votes
0answers
215 views

Merging multiple rasters (Very large dataset)

I am working on project to create a DSM raster for whole Kentucky. I will have over 46000 tiff tiles and each one is about 10 Mb and one band. Normally, I use Mosaic to New Raster Tool but this time ...
2
votes
2answers
1k views

Showing huge GIS data on map using Leaflet?

I have a huge amount of GIS data in a database. I am fetching this data and converting it into GeoJSON before viewing the map. I am fetching all of the data to meet the requirements, which is taking a ...
1
vote
1answer
56 views

Are there any continental-scale flow accumulation products besides Hydrosheds?

I am trying to compute branching angles of drainage networks for roughly the western half of the U.S. Because many of these streams are non-perennial (don't have water all year), some of the standard ...
0
votes
1answer
674 views

Extract values from large rasterstack, R

I have a large rasterstack of about 500 objects. I want to sample easch object with the use of a SpatialPointDataframe and then export the results as file.csv. I have tried the following but within ...
2
votes
1answer
127 views

Working with Sentinel-2 data and aggrigation in R

So I was wandering if there is a way to simplify a couple of Sentinel's rasters so I can apply raster analysis on them. Problem is that whenever I try to aggregate the data my RAM, CPU and storage ...
2
votes
1answer
185 views

Problems intersecting large polygon shapefiles in QGIS (V2.18.2)

I am using the BirdLife International .gdb file (polygons) of bird species distributions to try and make a species richness map across the Brazilian Amazon. To do this I have clipped the global ...
0
votes
1answer
669 views

Bing maps and large datasets

We were using Bing maps to place pins on a map when a user loads a page. The number of pins has grown significantly so we started to cluster the pins to reduce the load on the client. Now even ...
2
votes
0answers
314 views

Dealing with Very Large Satellite Imagery

I'm curious if anyone has suggestions for viewing and processing really large imagery. I'm pretty experienced with desktop apps like ENVI and am comfortable with python and GDAL but I'm analyzing ...
35
votes
2answers
15k views

Looking for fastest solution for Point in Polygon analysis of 200 million points [closed]

I have a CSV containing 200 million observations with the following format: id,x1,y1,x2,y2,day,color 1,"-105.4652334","39.2586939","-105.4321296","39.2236632","Monday","Black" 2,"-105.3224523","39....
3
votes
1answer
400 views

Displaying LiDAR data with too many points in ArcMap?

I have imported LiDAR data into ArcMap in the shape of a Las DataSet with twelve .las files (contains over 100 million data points). I believe the file itself is working properly. However, no points ...
3
votes
4answers
270 views

Is city-supplied GIS data sufficient for transit operators or is own set of GIS data needed? [closed]

City GIS data is usually built in a format where streets/avenues/boulevards/etc are modeled as lines that are geographically located in the middle of the streets/avenues/boulevards/etc. Is that ...
2
votes
1answer
472 views

Loading a large osm file in osmar : why does loading time scale quadratically with file size?

I'm trying to load huge .osm files into R in order to run advanced analysis. I did not expect the loading part to be complex, but it proved difficult. The issue here is that the time it takes to ...
2
votes
2answers
609 views

Dice a polygon with over 21 million vertices in ArcMap?

Based on posts like this I suspect I'm bumping up against issues when trying to work with a feature class that contains features with a single very complicated polygon. These errors include 000426, ...
0
votes
1answer
561 views

Joining CSV-Generated Datapoints with Shapefiles in ArcGIS Desktop?

I have a question about joining data points (with a large amount of information) with shapefiles in ArcGIS Desktop: I have imported a CSV file into GIS. This CSV File is quite large: each row ...
0
votes
0answers
62 views

Why does myRaster[1] <- 5 takes a very long time for a large raster?

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
6
votes
4answers
2k views

Need a location based social network data set

I'm looking for a data-set of Geosocial network consisting of GPS trajectories, check-in, geotag contents and stuff like these available for research purposes. I'd really appreciate it if you can ...
6
votes
1answer
1k views

How do you speed up creating new field in attribute table for huge amount of data in QGIS?

In my attribute table for a shapefile, I am converting one field (Julian day) to another field (date) using the field calculator but I'm working with about 7.7 million rows. QGIS has been running for ...
1
vote
0answers
226 views

Unable to create Cache for Large Vector

I have uploaded a large vector data (20 million records) to Postgres; published the data on Geoserver. But, when I try to create cache for the same it throws GC overhead limit exceeded error. I have ...
1
vote
1answer
232 views

Calculate nearest town for 2900 lot centroids

I have 16 town point shapefiles. I have a layer of 2900 centroids for lots. I need to determine the nearest town for each of the centroids. The option I came up with is to Split by Attribute by ...
3
votes
1answer
189 views

How do I calculate the distance between multiple associated points in QGIS?

I have two large lists of points. List A contains the location of a farmers farmstead and list B contains locations of pastures. Both lists have a column that will identify the farmer that the pasture ...
3
votes
2answers
258 views

How can I perform a Join by Location for a large vector and very large point file?

I have no programming skills, but am thinking I may need to at least figure out how to run this one script from inside QGIS. I have a vector file of global quarter degree grid cells and a point file ...
0
votes
2answers
474 views

Processing a large number of raster files in QGIS

I have a large number (100,000+) of raster files I need to process in QGIS (exporting to tiff, clipping to polygon layers, etc), for a total of approx 500GB. Can anyone advise if there's some way of ...