Skip to main content

Questions tagged [large-datasets]

difficult to process and manage because the size is usually bigger than the limits software can normally deal with.

Filter by
Sorted by
Tagged with
3 votes
1 answer
103 views

Merging PostGIS tables with billions of rows

I'm trying to create a unified high population density dataset using the Facebook "data for good" dataset. Currently their dataset is available as a raster or CSV per catagory (men, women, ...
RedM's user avatar
  • 1,565
0 votes
0 answers
13 views

Trying to export a time series of EVI values for a large asset in Google Earth Engine

I'm trying to export a time series of EVI data for a large asset containing 40,000 points. I am using the MODIS data. The code that I've written works when I extract data for 5 points, but it's not ...
Lauren's user avatar
  • 1
1 vote
0 answers
20 views

rasterio._err.CPLE_OutOfMemoryError: /io/gdal-3.6.4/frmts/mem/memdataset.cpp when trying to create a new large rasterfile

I am trying to step by step save a lot of data in a single GeoTiff file but rasterio seems to fail right away when I try to open the file I want to write my data to due to my available RAM not being ...
Sven's user avatar
  • 11
2 votes
1 answer
97 views

Get the outline of badly optimized polygons (too many vertices?)

I'm trying to get the area of some flooding data. Overall, the dataset is huge and hardly to handle in QGIS. The dataset comes in several shapefiles ranging from 60 Mb to 4 Gb and include several ...
Waynenation's user avatar
0 votes
0 answers
36 views

Issues Opening Large DTM File in QGIS - ‘Bad Allocation’, 'Error occurred while parsing element at line 1 column 1' Error

I’m trying to open a large DTM file in QGIS but I’m encountering errors. The data is a DTM 5x5 m map of the Lombardia region. The .ige file is 7.7 GB and the .img file is only 522 KB. I’ve tried ...
Zea Mays's user avatar
0 votes
0 answers
220 views

Speed up loading large GeoJSON layers to Leaflet map?

I have some GeoJSON data layers which I fetch their features from phpMyAdmin to display on a Leaflet map. But loading the features takes almost 10 seconds to be loaded. Have you any solution to speed ...
Solmaz's user avatar
  • 149
4 votes
2 answers
1k views

Data too huge to access: Bing Australia Building Footprints

I am relatively inexperienced with QGIS. I seeking to access building footprints in a 20km radius from the centre of Melbourne city, Australia. The dataset I am hoping to use is Bing's 2020 Australia ...
J.Watt's user avatar
  • 51
0 votes
1 answer
680 views

Using sf package for R to calculate distances for large dataset of location pairs

Working with data set of about 84k location pairs and trying to run in same manner as example code below. Getting results where distances are returned in an object of n x n dimension so when run on ...
lhilsbos's user avatar
2 votes
0 answers
530 views

Exporting large attribute table from QGIS to Excel

I am working on a project with an attribute table of approx. 5,564,098 rows. I've tried exporting it to Excel using the "save as" CSV or XLS but the data keeps getting cut off. I would ...
Peyton's user avatar
  • 21
1 vote
1 answer
394 views

Google Earth Engine: how to export large image as one image not tiles

I want to download sentinel5 data for no2 for Saudi Arabia the whole country. This is my code: var collection = ee.ImageCollection('COPERNICUS/S5P/OFFL/L3_NO2') .select('NO2_column_number_density') ...
Randa Al-Harbi's user avatar
0 votes
1 answer
188 views

Tessellation on huge dataset

I have a set of CSV files with lat/long for points spaced every 500 meters apart. There are several files that collectively span a huge area across the Nevada/Utah border. I need to create a ...
a11's user avatar
  • 950
-1 votes
2 answers
94 views

Interpolating using ArcGIS Pro [closed]

I have 2 datasets. They have exactly the same columns and the data for the second dataset was collected the day after first dataset. When I perform Diffusion interpolation with barriers on the first ...
Wayfarer's user avatar
  • 121
0 votes
1 answer
157 views

How to Map.addLayer() rainfall featureCollection over globe using Google Earth Engine?

I am trying to print the precipitation imageCollection allover the whole world but I allways get only a part of the image. as you see in the image, the north of europe is empty and event when I use ...
Ward Bahri's user avatar
0 votes
1 answer
479 views

sf: Session aborts when reading large geopackage into R

I want to read a geopackage into R that is roughly ~ 600 mb in size. Every time I want to read it my R-Studio session aborts when doing the data = read_sf(path_geo) Is there any "trick" ...
Lenn's user avatar
  • 1,297
2 votes
1 answer
411 views

Polygonize large raster to create mask gives memory error

I have a very large land cover/land use raster for South America with dimensions (y: 162565, x: 188387) at the Landsat 30m resolution. I want to use it to mask another raster for the same area at 270m ...
Caio Reis's user avatar
0 votes
0 answers
525 views

Computing area of very large feature collection in GEE?

I'm trying to calculate the total area of a very large Feature Collection (over 1 million features) of burned areas over the Amazon. I want to find the total area (sum), and also the area grouped by ...
KatieGeo's user avatar
2 votes
0 answers
137 views

What DEM Properties Affect Depression Filling Performance and Results?

In preparing to work with large DEM files, I've been trying to understand the differences between burn, fill, and breach pre-processing methods in terms of the resulting accuracy of stream networks, ...
deekay's user avatar
  • 21
2 votes
1 answer
1k views

How to view large data in Google Earth Pro?

I have a dataset with 32,000 location points and I need to view all the points in Google Earth Pro (v. 7.3.4.8248; 64-bit) at the same time. I have tried exporting my data (from R) as a KML and as a ...
cgxytf's user avatar
  • 235
0 votes
1 answer
31 views

Interpreting an image taken over a one-year period

I found this dataset: Global Aboveground and Belowground Biomass Carbon Density Maps that's supposed to contain data for all 2010, but it actually just contains one image (size is =1). Should I ...
Sebastian Nuñez's user avatar
4 votes
1 answer
194 views

Efficient analysis of large amount of CSV files in QGIS

I have hundreds of CSV files with geometry values in the same folder, and I'm trying to import all into QGIS, and use 'count point in grid' function to see how many of each data on each grid. I've ...
Cookie's user avatar
  • 355
2 votes
0 answers
227 views

How to build raster file from large folder tiles (256x256), which has 70.000 file tiles

First, I'd like to mention that I'm new to working with GIS, so please forgive my inexperience. I have map folder with structure: zoom level contains multi tile image files (256x256),ex: folder zoom ...
Hong Phat's user avatar
0 votes
1 answer
712 views

How to handle layer that contains millions of vertices in QGIS

I have a multi polygon layer with 25 features, each containing millions of vertices, holes, etc. It is a statewide floodplain. Right now, it takes forever to draw even when zoomed in, plus the ...
ar-siddiqui's user avatar
  • 1,768
1 vote
0 answers
101 views

What is a good way to store a large (tiled) geo raster?

I have a large geotiff raster map (after compression 88 GB) as a series of 919 geotiff-tiles. How can I turn these into a practical file solution (e.g. one file) that allows me to efficiently work ...
Honeybear's user avatar
  • 2,504
2 votes
1 answer
70 views

Creating random points using ArcMap

I want to create 100 random datasets in ArcMap for a particular area. Each dataset should contain 10,000 random points. I am using the "Create Random Points" tool in ArcMap. Can anyone ...
Madusha's user avatar
  • 115
-1 votes
1 answer
143 views

Calculate NDVI in a large feature [closed]

I'm having some trouble with my GEE script. I have to calculate NDVI of each feature of my shapefile, but it is too large. Someone could help me with this code? I think it's king of simple, but I'm ...
Rogério Goncalves's user avatar
1 vote
3 answers
114 views

Count how many new points are captured when a polygon buffers by x miles

I have a polygon and a layer of many points. How many more points would be captured if I expanded the polygon by x meters? I want to end up with values for x = 1 mile, 5 miles, 10 miles, and maybe a ...
Benny Henny's user avatar
4 votes
1 answer
1k views

Using PostGIS to find the overall difference between two (large) polygon datasets

I need to get the Difference of an input layer with ~30k polygons and an overlay layer with ~500k polygons (of which ~200k intersect the input layer). I am trying to learn PostGIS to do this, as it ...
MDB's user avatar
  • 175
2 votes
0 answers
307 views

PostGIS: Using ST_CoveredBy (or similar) to speed up Difference on large datasets?

I need to get the Difference of a single large-area polygon (input layer) and a very numerous set of polygons (overlay layer). The overlay layer has ~500k polygons, of which ~200k intersect the input ...
MDB's user avatar
  • 175
3 votes
2 answers
560 views

What is the canonical way to perform raster calculations using R stars?

I have a data cube that I am handling in the R stars package. It has 3 dimensions (latitude, longitude and time) and 2 attributes (northwards and eastwards components of wind). A dummy equivalent ...
TJC's user avatar
  • 151
1 vote
0 answers
133 views

GEE large export of pixels using shapefile polygons: Is it possible to automate a division of the polygons asset?

I need to export pixels from a processed image of Sentinel-2 using GEE Javascript with a 10 pixel size. The export is for pixels within 120,000 land parcel polygons. I am way above the limit for pixel ...
ie-con's user avatar
  • 23
2 votes
1 answer
871 views

Leaflet - remove data from layer?

I have a quite complex webapp with a lot of data on a map. Except the basemap, the data of each layers are coming from AJAX queries, witch's responses are formatted as GeoJSON object -- up to this ...
Pál Rudan's user avatar
0 votes
0 answers
75 views

How to calculate and store (ArcMap) slope on huge mosaic dataset?

I have a huge raster dataset stored in a geoDB (DHM with 2m resolution for all of Switzerland), uncompressed size ~75 GB. I want to use the ArcMap Slope tool (or another tool would also work) to ...
Honeybear's user avatar
  • 2,504
0 votes
1 answer
477 views

How to normalize a large point cloud using lasheight and a .tif DTM?

I am normalizing a large point cloud (>16 GB), for which I'm using a preprocessed DTM that I store as .tif. For smaller amounts of data, I would use an R script that lets me do this: library(lidR) ...
Honeybear's user avatar
  • 2,504
-1 votes
1 answer
2k views

Export large area data in Google Earth Engine

I am trying to average an image collection from 2019-02-01" to "2019-02-28 on Earth Engine platform. My goal is to export the data in Google Drive that I can use it in QGIS for other ...
goldy_mit's user avatar
3 votes
1 answer
572 views

Intersecting large shapefiles onto grid in QGIS

I have 21 separate shapefiles containing data for Ireland as various polygons on QGIS 3.10. I'm trying to intersect each shapefile onto a 250m² grid of Ireland so that I'll have 21 separate shapefiles ...
Shaun Doolin's user avatar
2 votes
2 answers
235 views

Zonal statistics USFS tree cover by census block

I want average tree cover for every census block. I have a file of census blocks, and I have a USFS tree cover file from USFS Tree Canopy Cover Datasets that looks like this: I'm trying to use zonal ...
Ben Hendel's user avatar
3 votes
2 answers
453 views

gdal_calc raster calculation fails on 28 GB GeoTiff

I am trying to do a raster calculations on a 28 GB GeoTiff using gdal_calc.py (3.0.4) but the process fails gdal_calc.py -A DEM_0.tif --outfile=DEM_1.tif --calc="A+0.33" --format=GTiff --...
EOF's user avatar
  • 71
0 votes
1 answer
132 views

Large-scale cloud masking, grouping and mosaic in Descartes Labs platform

I am looking to create a stack of cloud-free sentinel-2 imagery by day for a large portion of California for all of 2020 in the Descartes Lab platform. I realize that this is a very large amount of ...
Madeline Lisaius's user avatar
1 vote
1 answer
1k views

Google Earth Engine: Error exporting a large Feature Collection to CSV

On GEE, I want to export daily weather variables for 18 years for about 30,000 rectangular polygons. That would be exporting a FeatureCollection to a csv file with ~200 million rows (30000 places * ...
vpk's user avatar
  • 111
1 vote
0 answers
342 views

Generate synthetic trajectory dataset that has global coverage

I want to generate a spatial trajectory dataset which basically contains tuples - (lat,lon,time), grouped together by an identifier so they form trajectories. In addition, I want this dataset to have ...
Jio's user avatar
  • 652
4 votes
2 answers
2k views

Calculating max shardSize and fileDimensions to avoid tiling large raster exports in Earth Engine

I am exporting a very large raster in Google Earth Engine to my Google Drive. I am trying to keep my exported raster to 1 file to avoid additional processing to merge the tiles (e.g. GDAL merge in ...
rachell's user avatar
  • 107
0 votes
2 answers
2k views

Exporting large rasters in Google Earth Engine

When exporting large raster data in Google Earth Engine to your Google Drive, is there a function that prevents the export from automatically tiling into smaller files, or even decreases the number of ...
rachell's user avatar
  • 107
3 votes
2 answers
1k views

Publishing a large and complex map to the web

I have created a complex map of my area and I wish to publish it as an interactive map on the web. The dataset appears to be too large for QGIS2Web to work - the script completes eventually but in ...
WestCorkPalaeo's user avatar
1 vote
2 answers
571 views

Heatmap of a large polyline dataset in QGIS

To summarize: I'm having difficulties with creating a heatmap from a large (90021-item) polyline dataset in QGIS 3.10.1. First, it is worth noting that this question is somewhat similar to Doing ...
LRitzdorf's user avatar
  • 146
3 votes
1 answer
153 views

Merging and snapping shapefiles

I have several large shapefiles from various sources. All cover various sections of oceans and many overlaps, as such the coastline is repeated across the several shapefiles but to varying levels of ...
Tom Shelley's user avatar
0 votes
1 answer
770 views

Serving large (1.7Tb) dataset of aerial images with GeoServer

I have a large dataset of aerial images consisting of 8000 TIFFs where single images are no larger than 0.5G and all have have .ovr pyramid files with them. Previously I have served this dataset as ...
jfp's user avatar
  • 51
7 votes
4 answers
7k views

Convert large OSM files to GeoJSON

I have been trying to convert a large OSM file (5G) to the GeoJSON format. All the tools I have been using, use extensive memory and throw an out of memory exception. What is the best way to convert ...
ANN's user avatar
  • 113
1 vote
0 answers
257 views

Leaflet: Load GeoJSON point data in polygon, when selecting polygon

I've got a point dataset (~ 160.000 points) which has a filesize of around 30 MB. Because that is too large I decided to slit it into 12 separate GeoJSON files. I've got a map with 12 polygons. When ...
Jones's user avatar
  • 21
0 votes
1 answer
34 views

Creating a layer with selected values (divided by commas) from another layer - query or model needed

I have a very large dataset in ArcMap 10.6 and I would like to create a subset that selects only rows that have specific attributes (some of them with special characters) from a field that in my case ...
Geo_explorer's user avatar
1 vote
0 answers
275 views

Extracting or masking of very large raster to reduce file size on disk?

I have a very large 8-band raster (339806 x 61704) that I cannot load to memory to manipulate (R, raster:canProcessInMemory indicates I need 4999.16 GB of ram to load this file). However, I only need ...
steelhead's user avatar