Questions tagged [memory]

memory is the process in which information is encoded, stored, and retrieved.

Filter by
Sorted by
Tagged with
0 votes
1 answer
25 views

Configuring GeoTools to convert graphic objects to shapefiles without using filesystem?

By default, GeoTools (20.2) tries to convert the graphic objects into shapefiles on the disk. We want to skip this step and have GeoTools keep the file contents in memory (and return it as a byte ...
user avatar
  • 11
2 votes
1 answer
49 views

Polygonize large raster to create mask gives memory error

I have a very large land cover/land use raster for South America with dimensions (y: 162565, x: 188387) at the Landsat 30m resolution. I want to use it to mask another raster for the same area at 270m ...
user avatar
1 vote
0 answers
85 views

QGIS Python script crash when looping many times (memory problem?)

I'm having trouble with a part of a Python code that does many iterations (about 5000). The problem found is that the program crashes, it returns an error (sometimes memory error, other times all QGI ...
user avatar
  • 21
2 votes
0 answers
45 views

Memory efficient way of creating a large PostGIS table

I am trying to add the results of a query to PostGIS DB. My query converts a raster band into polygons. Although the original raster is quite big (~150GB, 12 bands) my below query runs reasonably fast ...
user avatar
  • 405
1 vote
0 answers
37 views

Optimizing memory usage when running r.viewshed in GRASS

Short explanation: I am using r.viewshed as part of a script. The viewshed analysis is the most costly (time) part of the script and I'm trying to optimize how quickly it runs (I need to run the ...
user avatar
0 votes
1 answer
150 views

Trying to process rasters in python and received CPLE_OutOfMemoryError error

I have a script that does some geoprocessing, that makes sense in a test run, but when I actually try to use it with my actual dataset, appears to totally overload my computer with memory needs, and I ...
user avatar
0 votes
0 answers
66 views

gdalwarp releases memory VERY slowly

I’m trying to use gdalwarp to merge and reproject many elevation tif files (DEM). To do so I first create a VRT file from all the tif files and then convert and reproject to one big tif file. This is ...
user avatar
  • 1
0 votes
1 answer
78 views

GEE: reducing CHIRPS-SPI by region and time. user memory limit problem [closed]

I found a script that calculates the SPI index for places and times of your choice from the CHIRPS data. Off the shelf, the code generates a couple of images (and maps) for each place-time combination ...
user avatar
  • 1
2 votes
2 answers
261 views

PyQGIS 3: Cannot Pass Memory OUTPUT To Next Processing Algorithm As INPUT

I am attempting to convert some ArcGIS arcpy command-line scripts to PyQGIS and am running into a problem working with memory layers. The script takes several vector files as input and progresses ...
user avatar
  • 31
0 votes
0 answers
164 views

Rasterio merge to memory file not taking profile

I am merging datasets to a memory file using rasterio. However I can't get the memory file to take on the profile attributes passed in as dst_kwds. I get the desired result when I write the dataset to ...
user avatar
7 votes
2 answers
327 views

Adding temporary output of processing algorithm to the map canvas using PyQGIS

Let's say I run a simple algorithm like this import processing # https://docs.qgis.org/3.16/en/docs/user_manual/processing_algs/qgis/vectoranalysis.html?highlight=count%20points%20polygon#id3 ...
user avatar
  • 3,017
5 votes
2 answers
448 views

Why can R not allocate enough memory even for a small raster file?

I have a raster layer of vegetation cover (range from 1 to 3). The size of the file totals 60 Mb so it is not big at all. I know this is duplicating other questions, but they have not helped me out. I ...
user avatar
  • 772
1 vote
1 answer
44 views

Why do I get memory problems reducing over specific regions?

The code below works for most countries but there are memory problems for some (e.g. Egypt, Ghana, Angola), even though these countries do not have a bigger population or land area than some countries ...
user avatar
  • 51
-1 votes
1 answer
31 views

Memory error when exporting task

I'm trying to get a building density map per square km of Africa. The following code will run ~15 hours and then report a memory error. How can I avoid this? var buildings = ee.FeatureCollection('...
user avatar
  • 51
0 votes
0 answers
48 views

gdaladdo memory issue:

I have a .gpkg raster that holds data (only) in zoom level 15. I then tried to build overviews with gdaladdo for this raster, using the following command: gdaladdo --config GDAL_CACHEMAX 1500 --config ...
user avatar
  • 101
2 votes
1 answer
626 views

Error: Execution failed; out of memory. Google Earth Engine Fails to Export Image to Asset

I am writing one of my first scripts in GEE. I have grabbed various code chucks from stack exchange and have created a memory intensive script. My goal is two take the union of two subsets of global ...
user avatar
1 vote
0 answers
53 views

Large .las file processing in QGIS

I have a .las file that contains 100.000.000 points (2.6 Gb) and I process it by the custom QGIS plugin based on PDAL and GDAL. The problem is in processing large files as described in the previous ...
user avatar
0 votes
0 answers
15 views

Memory error when using LecoS vector overlay

I am currently trying to use the overlay raster metrics of LecoS. But am experiencing some difficulties. I do have the following error code: File "C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib\site-...
user avatar
0 votes
0 answers
32 views

How to use disk files to replace memfile?

I am trying to collect Sentinel images and compute their NDVI which is done quite simply. However, I need to check determine all shapely multipolygons and map a NDVI value for each. I've decided to ...
user avatar
2 votes
1 answer
234 views

Computation timed out when exporting time series for a large number of locations from Google Earth Engine

I am trying to export Sentinel-2 NDVI time series for a huge number of fields (n>80000) to a csv file. However I got a 'computation timed out' error after 12 hours of running. Here is the code and ...
user avatar
  • 21
1 vote
0 answers
32 views

QGIS file went missing?

I was working on a file for a few weeks, saving it on an external drive, in a folder, along with all layers and shapefiles. Today I went to open it and the file is gone. Not in recycle bin or anywhere ...
user avatar
  • 103
0 votes
0 answers
124 views

How much process memory is needed for ogr2ogr to process a GeoJSON file? -

I am trying to evaluate how many resources ogr2ogr needs to process a GeoJSON file; there's a concern that it's using much more memory than we would expect. All we are doing is converting a GeoJSON ...
user avatar
  • 11
2 votes
2 answers
99 views

Error generating chart: User memory limit exceeded, when changing dataset from T1_TOA to T1_SR

So, I got the code to work for T1_TOA band, but then when I switched it over to T1_SR band it gave me this error: User memory limit exceeded. So, I poked around the forum and found the code to limit ...
user avatar
4 votes
1 answer
226 views

Memory issues projecting a large raster mosaic of USGS DEMs in R

I am trying to project a large raster to UTM (meters), but I keep crashing my laptop (16GB) due to a memory shortage. It seems like using projectRaster tends to deplete the machine's memory, and some ...
user avatar
1 vote
1 answer
148 views

QGIS Hot Spot analysis plugin Python memory error

I try to create an Hotspot crime map of Greater London. I created grid, counted points layers. When I try to use Hotspot analysis plugin after a while my memory spikes like crazy (16gigs) and this ...
user avatar
1 vote
0 answers
45 views

QGIS / "create grid" consumes far more memory on linux than on windows

I am using QGIS 3.10 both on windows and on Linux with the same RAM and CPU resources. When running the tool "create grid" QGIS behaves strange on Linux: For a certain extent (~ 10x10km), ...
user avatar
  • 11
0 votes
1 answer
218 views

Why do I have such a high memory usage for this rioxarray code?

For my code using rioxarray I am opening quite an amount of raster files. One after another. I even added del to remove those from memory, even though I thought it would not be necessary, because they ...
user avatar
  • 17
2 votes
0 answers
65 views

Releasing ArcMap memory by removing *.sr.lock files from shapefiles

Is there a method of removing *.sr.lock files from shapefiles to release memory consumption without closing out of an ArcMap application? I am attempting to run code that will cycle through 200+ ...
user avatar
  • 53
1 vote
1 answer
275 views

How to fix the problem of 'out of memory' in PostGIS Raster processes?

I am trying to combine a series of overlapping rasters to find out the most recent data. I think I may have 10,000 layers. When I run this query: CREATE TABLE most_recent AS ( SELECT ST_Union(rast,...
user avatar
  • 525
1 vote
1 answer
3k views

Read and then write rasterio GeoTIFF file without loading all data into memory

I work with large drone image raster files, on the order of 40,000 x 40,000 and above. I have a large uncompressed GeoTIFF file and I want to use rasterio to rewrite the file in compressed format. I ...
user avatar
  • 849
3 votes
0 answers
80 views

Getting the results of a PostGIS query to a file with ogr2ogr with low memory usage

I have a PostGIS query returning geographies and attributes. The result set is deterministic and ordered (so OFFSET and LIMIT work for getting subsets). The result set of my query would exceed my RAM ...
user avatar
  • 729
0 votes
0 answers
353 views

Working around "Unable to allocate memory" error during for-loop in Arcpy

I'm running a very long for-loop in Arcpy (with ArcGIS 10.7 on Win10) over several ten thousand files. It runs smoothly until, everytime after about 3000 iterations (3-4h), it crashes with the error ...
user avatar
0 votes
1 answer
198 views

ArcObjects C# memory management - Out of memory exception

I'm a noob in C# and ArcObjects (started two weeks ago). I'm trying to write a stand-alone console program that will change label placement quality in all data frames for all MXD files in a given ...
user avatar
5 votes
2 answers
79 views

Do Navigation Apps Shrink the Hippocampus?

A study shows London cab drivers' hippocampi grow compared to a control group of similar non-drivers. I presume these cab drivers don't rely on navigation apps. ...acquisition of an internal ...
user avatar
1 vote
1 answer
220 views

Running into memory issues when loading a composer layout

I'm trying to load a specific composer layout model, but it not respond properly, slowing down my computer, consuming all it's memory, to the the point of giving me no option but a Windows hard reset. ...
user avatar
1 vote
0 answers
105 views

pyqgis 2.18.16 - Segmentation fault error at 2nd run

When use pyqgis+python for auto-mapping, I encounter a tricky Segmentation fault (core dumped) error, at the 2nd run of the program. I checked previous similar post but not able to fix the issue. I ...
user avatar
  • 11
1 vote
0 answers
27 views

Memory management in offline routing with n waypoints?

I am trying to build an offline routing system with a map database on a portable Android device. Since I am still doing some research, I am stuck on what I think is a fundamental problem. To find the ...
user avatar
0 votes
0 answers
41 views

ArcGIS Modelbuilder model clogs up memory?

I have a model in which a zonal histogram is computed of each line in a feature class, showing the amount of cells each line intersects in a raster file. After this, this column is joined to a table ...
user avatar
0 votes
1 answer
118 views

How to free the memory of return value of OGRGeometry::Buffer()?

I have in C++ (gdal 2.3.2) OGRPolygon poly; OGRGeometry* geom = poly.Buffer(1e-12, 1); now how do I free the memory allocated for geom after done using it?
user avatar
  • 103
0 votes
1 answer
297 views

Using Hyperion with Google Earth Engine [closed]

I am trying to use Hyperion Data for my project, unfortunately even loading the Image Collection of Hyperion exceeds my maximum 2.5 GB Data Storage. Hyperion has a revisitrate of 200 Days, so a one ...
user avatar
0 votes
1 answer
72 views

Pros and cons of saving data on a memory stick [closed]

I have just about reached the capacity of my laptops internal Solid State Hard Drive (Sandisk SD8SNAT-256G-1006) and I looking for some additional storage without having to buy a new machine. You ...
user avatar
  • 11
0 votes
0 answers
112 views

Exporting Layer to Feature Class is Incomplete without Error

I have a CSV of 1.2 million records that I want export to a GDB as a feature class. I've done this numerous times, but this was before upgrading from 10.5. Steps: Add CSV Display X/Y table Export to ...
user avatar
  • 533
2 votes
0 answers
42 views

Using destructors in PyQGIS?

Related to my Problem described in PyQGIS memory issues, I'd like to try using destructors for QgsGeometry and other classes. I dont get how to use them though. According to https://qgis.org/api/2.18/...
user avatar
  • 309
3 votes
0 answers
273 views

PyQGIS memory issues [closed]

I have some big problems with the memory usage of my standalone PyQgis script based on QGis 2.18 and run on ubuntu/linux (happens also on Windows 10 though). I will document it very precisely using ...
user avatar
  • 309
0 votes
1 answer
358 views

Geoserver - how clear memory after tile-seeding and normal wmts queries

I am currently doing a lot of seeding of tiles in GeoServer and I noticed that after the tiling process is over the memory that is required during the task is not cleared. The same thing happens after ...
user avatar
  • 982
2 votes
1 answer
57 views

Watershed issue

I am attempting to create a catchment of Danube. So far I have used the r.fill.dir to create a depressionless map and flow directions. Thus, I am now looking to use r.watershed to create stream ...
user avatar
0 votes
1 answer
202 views

Topology error in ArcGis

I am trying to intersect a large feature class (118692 elements) with a grid (58000 elements) but I get an error message. Traceback (most recent call last): File "Path/InterJoin.py", line 24, in &...
user avatar
2 votes
1 answer
1k views

How to increase Java heap size on GeoServer on Windows?

I tried setting wrapper.java.maxmemory=1300 parameter in c:\Program Files (x86)\GeoServer 2.11.1\wrapper\wrapper.conf but I can't set more than 1300MB (geoserver service can't start). I tried ...
user avatar
3 votes
1 answer
1k views

Creating permanent vector layer in QGIS

I am currently creating a vector layer like following: outLayer = QgsVectorLayer('Linestring?crs='+ crs, 'graph_with_azimuth_ogr' , 'memory') The outLayer is now stored in memory, which means it only ...
user avatar
0 votes
0 answers
231 views

Mapshaper running out of memory on R

I'm using rmapshaper, the R wrapper of MapShaper, to simplify polygons without slivers and it works well on small shapefiles. However, when I run it on a larger one (~150M, ~5000 shapes), it runs out ...
user avatar
  • 131