Questions tagged [large-datasets]
difficult to process and manage because the size is usually bigger than the limits software can normally deal with.
114
questions
1
vote
0answers
25 views
What is the canonical way to perform raster calculations using R stars?
I have a data cube that I am handling in the R stars package. It has 3 dimensions (latitude, longitude and time) and 2 attributes (northwards and eastwards components of wind). A dummy equivalent ...
1
vote
0answers
23 views
GEE large export of pixels using shapefile polygons: Is it possible to automate a division of the polygons asset?
I need to export pixels from a processed image of Sentinel-2 using GEE Javascript with a 10 pixel size. The export is for pixels within 120,000 land parcel polygons.
I am way above the limit for pixel ...
0
votes
1answer
44 views
Leaflet - remove data from layer?
I have a quite complex webapp with a lot of data on a map. Except the basemap, the data of each layers are coming from AJAX queries, witch's responses are formatted as GeoJSON object -- up to this ...
0
votes
0answers
19 views
How to calculate and store (ArcMap) slope on huge mosaic dataset?
I have a huge raster dataset stored in a geoDB (DHM with 2m resolution for all of Switzerland), uncompressed size ~75 GB. I want to use the ArcMap Slope tool (or another tool would also work) to ...
0
votes
1answer
41 views
How to normalize a large point cloud using lasheight and a .tif DTM?
I am normalizing a large point cloud (>16 GB), for which I'm using a preprocessed DTM that I store as .tif.
For smaller amounts of data, I would use an R script that lets me do this:
library(lidR)
...
0
votes
0answers
111 views
Export large area data in Google Earth Engine
I am trying to average an image collection from 2019-02-01" to "2019-02-28 on Earth Engine platform. My goal is to export the data in Google Drive that I can use it in QGIS for other ...
0
votes
0answers
33 views
Intersecting large shapefiles onto a grid in QGIS
I have 21 separate shapefiles containing data for Ireland as various
polygons on QGIS 3.10. I'm trying to intersect each shapefile onto a
250m2 grid of Ireland so that I'll have 21 separate shapefiles ...
0
votes
0answers
55 views
Calculate and plot the feature importance of the input dataset of a random forest classification within Google Earth Engine
I run a random forest classification for agricultural land use and other land cover classes (12 classes). My dataset for 2019 consists of a Sentinel 1 and Sentinel 2 monthly time series, statistical ...
2
votes
2answers
86 views
Zonal statistics USFS tree cover by census block
I want average tree cover for every census block. I have a file of census blocks, and I have a USFS tree cover file from USFS Tree Canopy Cover Datasets that looks like this:
I'm trying to use zonal ...
0
votes
1answer
74 views
gdal_calc raster calculation fails on 28 GB GeoTiff
I am trying to do a raster calculations on a 28 GB GeoTiff using gdal_calc.py (3.0.4) but the process fails
gdal_calc.py -A DEM_0.tif --outfile=DEM_1.tif --calc="A+0.33" --format=GTiff --...
0
votes
1answer
67 views
Large-scale cloud masking, grouping and mosaic in Descartes Labs platform
I am looking to create a stack of cloud-free sentinel-2 imagery by day for a large portion of California for all of 2020 in the Descartes Lab platform. I realize that this is a very large amount of ...
1
vote
1answer
277 views
Google Earth Engine: Error exporting a large Feature Collection to CSV
On GEE, I want to export daily weather variables for 18 years for about 30,000 rectangular polygons. That would be exporting a FeatureCollection to a csv file with ~200 million rows (30000 places * ...
1
vote
0answers
119 views
Generate synthetic trajectory dataset that has global coverage
I want to generate a spatial trajectory dataset which basically contains tuples - (lat,lon,time), grouped together by an identifier so they form trajectories. In addition, I want this dataset to have ...
0
votes
0answers
48 views
Georeferencing larger GeoTIFF's in QGIS
I have a GeoTiff (>100GB) and I want to georeference it in QGIS using the georeferencer option. I did manage to create the GCP's (>15) but when the georeferencing is set to run, it crashed mid-way. ...
3
votes
1answer
283 views
Calculating max shardSize and fileDimensions to avoid tiling large raster exports in Earth Engine
I am exporting a very large raster in Google Earth Engine to my Google Drive. I am trying to keep my exported raster to 1 file to avoid additional processing to merge the tiles (e.g. GDAL merge in ...
-1
votes
1answer
352 views
Exporting large rasters in Google Earth Engine
When exporting large raster data in Google Earth Engine to your Google Drive, is there a function that prevents the export from automatically tiling into smaller files, or even decreases the number of ...
3
votes
2answers
396 views
Publishing a large and complex map to the web
I have created a complex map of my area and I wish to publish it as an interactive map on the web. The dataset appears to be too large for QGIS2Web to work - the script completes eventually but in ...
1
vote
0answers
94 views
Heatmap of a large polyline dataset in QGIS
To summarize: I'm having difficulties with creating a heatmap from a large (90021-item) polyline dataset in QGIS 3.10.1.
First, it is worth noting that this question is somewhat similar to Doing ...
2
votes
1answer
65 views
Merging and snapping shapefiles
I have several large shapefiles from various sources. All cover various sections of oceans and many overlaps, as such the coastline is repeated across the several shapefiles but to varying levels of ...
0
votes
1answer
175 views
Serving large (1.7Tb) dataset of aerial images with GeoServer
I have a large dataset of aerial images consisting of 8000 TIFFs where single images are no larger than 0.5G and all have have .ovr pyramid files with them.
Previously I have served this dataset as ...
2
votes
3answers
714 views
Convert large OSM files to GeoJSON
I have been trying to convert a large OSM file (5G) to the GeoJSON format. All the tools I have been using, use extensive memory and throw an out of memory exception.
What is the best way to convert ...
1
vote
0answers
45 views
Leaflet: Load GeoJSON point data in polygon, when selecting polygon
I've got a point dataset (~ 160.000 points) which has a filesize of around 30 MB. Because that is too large I decided to slit it into 12 separate GeoJSON files. I've got a map with 12 polygons. When ...
0
votes
1answer
26 views
Creating a layer with selected values (divided by commas) from another layer - query or model needed
I have a very large dataset in ArcMap 10.6 and I would like to create a subset that selects only rows that have specific attributes (some of them with special characters) from a field that in my case ...
1
vote
0answers
64 views
Extracting or masking of very large raster to reduce file size on disk?
I have a very large 8-band raster (339806 x 61704) that I cannot load to memory to manipulate (R, raster:canProcessInMemory indicates I need 4999.16 GB of ram to load this file). However, I only need ...
0
votes
1answer
54 views
Counting points contained in polygon for large number of points in ArcMap?
I'm trying to determine the number of points within a set of polygons for a large number of points. There are 2GB of lat-long coordinates in a CSV file, and I want to see how many are contained in ...
0
votes
1answer
184 views
Reduce processing time for large GeoTIFF image
I am trying to process large GeoTIFF images (128 MB) using GeoTIFF reader provided by Geotools library. However the processing time is very high, can anyone suggest how to reduce this time.
Normal ...
1
vote
1answer
54 views
Why does geodataframe never complete?
I had a feature class made up of 1,700,000 polygons. I used Geopandas to create a geodataframe:
state = "MD"
state_gdb = r"C:\Projects\Pop_Alloc\{}_Data.gdb".format(state)
join_feat = "{}...
1
vote
0answers
85 views
R process: Zonal Stat Raster has No Data Values vs Arc output
I am trying to re-create my ArcGIS Zonal Stat model with R
All my files will be 30m pixels; 18000 x 18000.
I have used R two different ways to create the output and both create NoData values in the ...
0
votes
0answers
28 views
Union very large layer in FGDB and only keep one polygon that is overlapping?
I'm trying to union an exceptionally large layer (45k features) of footprints with many of them largely overlapping. I'd like to union this layer but I'm having trouble due to the size of it. The end ...
2
votes
1answer
743 views
How to snap to a geojson-vt layer (Vector Tile Layer)?
I'm using geojson-vt to render a large geojson (originated from OSM) file. This file contains all road from a state (extracted with overpass-turbo way[highway="*"]). I'm buinding a domain-specific ...
3
votes
2answers
169 views
Script I made for pyQgis runs slow but can't find another way around it
registry = QgsProject.instance()
first_layer = registry.mapLayersByName('buffered')[0]
second_layer = registry.mapLayersByName('buffered')[0]
areas = []
instrument_number = "tin_ttl_in"
proprietor =...
0
votes
0answers
71 views
ArcMap fails to convert all records from CSV file using Table to Table geoprocessing tool
I am using the table to table geoprocessing tool to convert a CSV file with nearly 5 million records into an GDB table in a file geodatabase. The tool succeeds each time, but only 603 records are ...
1
vote
1answer
1k views
Saving raster to BigTIFF in QGIS?
Using QGIS 3.4, I am trying to merge multiple DEMs downloaded from the National Map Viewer that are in ArcGRID format (Arc/Info Binary Grid). The output will be a large file so I can't use any of the ...
6
votes
3answers
5k views
Converting large GeoJSON to shapefile?
I have recently downloaded a building footprint GeoJSON file from Microsoft Bing Maps. The file I downloaded was for the entirety of Texas, and is of approximately 2.2 GB file size. They do not ...
1
vote
1answer
89 views
Using “Difference” geoprocessing tool on Big Data taking a very long time
I am using QGIS Bonn 3.2.3.
I've recently been using the Difference function to calculate a list of points and it is taking a terribly long time. I was wondering if you could point out any obvious ...
1
vote
0answers
333 views
GeoServer GetMap with geopackage very slow (with high zoom levels)
I have a GeoServer instance and have imported multiple high res tiff files as layers and created a layergroup from the layers. When I call the WMS endpoint with a specified bbox and the geopackage ...
0
votes
0answers
124 views
Improving performance of Eliminate tool in ArcGIS Pro?
My data is very large (polygon feature class with 884187 features), thats why Python geoprocessing tools (which use geoprocesses like Intersect, Buffer etc.) don't work due to performance issues.
So ...
0
votes
0answers
302 views
Merging multiple rasters (Very large dataset)
I am working on project to create a DSM raster for whole Kentucky. I will have over 46000 tiff tiles and each one is about 10 Mb and one band. Normally, I use Mosaic to New Raster Tool but this time ...
2
votes
2answers
1k views
Calculating time efficient seasonal means from raster bricks?
I'm starting from a Rasterbrick like that called r.brick:
class : RasterBrick
dimensions : 17, 19, 323, 21915 (nrow, ncol, ncell, nlayers)
resolution : 0.11, 0.11 (x, y)
extent : 8....
1
vote
1answer
79 views
Are there any continental-scale flow accumulation products besides Hydrosheds?
I am trying to compute branching angles of drainage networks for roughly the western half of the U.S. Because many of these streams are non-perennial (don't have water all year), some of the standard ...
0
votes
1answer
1k views
Extract values from large rasterstack, R
I have a large rasterstack of about 500 objects. I want to sample easch object with the use of a SpatialPointDataframe and then export the results as file.csv. I have tried the following but within ...
2
votes
1answer
185 views
Working with Sentinel-2 data and aggrigation in R
So I was wandering if there is a way to simplify a couple of Sentinel's rasters so I can apply raster analysis on them. Problem is that whenever I try to aggregate the data my RAM, CPU and storage ...
2
votes
0answers
330 views
Dealing with Very Large Satellite Imagery
I'm curious if anyone has suggestions for viewing and processing really large imagery. I'm pretty experienced with desktop apps like ENVI and am comfortable with python and GDAL but I'm analyzing ...
3
votes
1answer
555 views
Displaying LiDAR data with too many points in ArcMap?
I have imported LiDAR data into ArcMap in the shape of a Las DataSet with twelve .las files (contains over 100 million data points).
I believe the file itself is working properly. However, no points ...
2
votes
2answers
883 views
Dice a polygon with over 21 million vertices in ArcMap?
Based on posts like this I suspect I'm bumping up against issues when trying to work with a feature class that contains features with a single very complicated polygon. These errors include 000426, ...
0
votes
0answers
77 views
Why does myRaster[1] <- 5 takes a very long time for a large raster?
I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of ...
1
vote
0answers
272 views
Unable to create Cache for Large Vector
I have uploaded a large vector data (20 million records) to Postgres; published the data on Geoserver. But, when I try to create cache for the same it throws GC overhead limit exceeded error.
I have ...
1
vote
1answer
242 views
Calculate nearest town for 2900 lot centroids
I have 16 town point shapefiles.
I have a layer of 2900 centroids for lots.
I need to determine the nearest town for each of the centroids.
The option I came up with is to Split by Attribute by ...
7
votes
1answer
1k views
How do you speed up creating new field in attribute table for huge amount of data in QGIS?
In my attribute table for a shapefile, I am converting one field (Julian day) to another field (date) using the field calculator but I'm working with about 7.7 million rows. QGIS has been running for ...
3
votes
1answer
212 views
How do I calculate the distance between multiple associated points in QGIS?
I have two large lists of points. List A contains the location of a farmers farmstead and list B contains locations of pastures. Both lists have a column that will identify the farmer that the pasture ...
