Questions tagged [large-datasets]
difficult to process and manage because the size is usually bigger than the limits software can normally deal with.
15 questions
4
votes
1
answer
1k
views
Using PostGIS to find the overall difference between two (large) polygon datasets
I need to get the Difference of an input layer with ~30k polygons and an overlay layer with ~500k polygons (of which ~200k intersect the input layer).
I am trying to learn PostGIS to do this, as it ...
- 175
9
votes
4
answers
8k
views
Thinning large LiDAR point cloud?
I have a large point cloud (LAS file) and I am trying to thin this using a complex technique. I know it is possible to thin a point cloud using simple techniques such as extracting every nth point. ...
- 147
36
votes
2
answers
17k
views
Point in Polygon analysis of 200 million points [closed]
I have a CSV containing 200 million observations with the following format:
id,x1,y1,x2,y2,day,color
1,"-105.4652334","39.2586939","-105.4321296","39.2236632",&...
- 371
12
votes
3
answers
20k
views
Measuring performance difference between Merge and Append in ArcGIS Desktop?
Often I find myself in a situation where I don't care whether or not my tool produces a new feature class, but I do care how long it takes to combine all of my large datasets. Does it take longer to ...
- 211
7
votes
1
answer
2k
views
How do you speed up creating new field in attribute table for huge amount of data in QGIS?
In my attribute table for a shapefile, I am converting one field (Julian day) to another field (date) using the field calculator but I'm working with about 7.7 million rows. QGIS has been running for ...
- 217
5
votes
2
answers
3k
views
Creating smaller OSM with only basic zoom levels data from larger OSM file?
From a large .osm/.o5m file (for India), I only want to extract the data that would actually show up when one is looking from zoom levels 1/0 through 8 (which is quite zoomed out.. only major cities, ...
- 1,026
4
votes
2
answers
2k
views
Calculating max shardSize and fileDimensions to avoid tiling large raster exports in Earth Engine
I am exporting a very large raster in Google Earth Engine to my Google Drive. I am trying to keep my exported raster to 1 file to avoid additional processing to merge the tiles (e.g. GDAL merge in ...
- 107
9
votes
5
answers
3k
views
Queries returning very big datasets in PostGIS
I have a PostGIS query that will return several million rows:
SELECT
t1.id AS id1,
t2.id AS id2,
ABS(t1.mean_h - t2.mean_h) AS h_diff,
ST_Distance(t1.the_geom, t2.the_geom) AS dist
FROM ...
- 9,411
8
votes
9
answers
2k
views
WMS with Access Control (ACL) for different users? Or solution for thousands of points on web map?
I have a set of sites in the thousands. Not all users see same the same Sites on the map - thus ACL. Data is stored in a non-GIS database (mssql, ora, etc) and right now is retrieved with a standard ...
- 3,971
7
votes
2
answers
2k
views
memory issue when trying to buffer/union large dataset using postgis
I am trying to generated a dissolved buffer of a large point dataset (Ideally 29 million points - Address data in Great-Britain, but I receive the data by chunks of 1 million points, so these could be ...
- 135
4
votes
2
answers
2k
views
Stratified Random Sampling with large dataset in ArcGIS?
I want to create a Stratified Random Sample with a land cover classification map. The purpose for the Stratrified Random Sample is to produce an Accuracy Assessment. I am using ArcGIS 10.2 with access ...
2
votes
1
answer
737
views
Loading a large osm file in osmar : why does loading time scale quadratically with file size?
I'm trying to load huge .osm files into R in order to run advanced analysis. I did not expect the loading part to be complex, but it proved difficult.
The issue here is that the time it takes to ...
- 113
1
vote
3
answers
3k
views
Windows software to convert GeoJSON (> 6Mb) to Excel/CSV?
I am trying to convert OpenStreetmap data GeoJSON to Excel (or .csv files). Few online converters are there.
However, when the file is big (example: 6Mb Geojson), the browser is becoming ...
- 31
1
vote
1
answer
3k
views
Saving raster to BigTIFF in QGIS?
Using QGIS 3.4, I am trying to merge multiple DEMs downloaded from the National Map Viewer that are in ArcGRID format (Arc/Info Binary Grid). The output will be a large file so I can't use any of the ...
- 450
0
votes
2
answers
2k
views
Exporting large rasters in Google Earth Engine
When exporting large raster data in Google Earth Engine to your Google Drive, is there a function that prevents the export from automatically tiling into smaller files, or even decreases the number of ...
- 107