10
votes
Replace NA's with 0 for large raster data using R?
"external/test.grd" isn't the best way to test this. Is a very small raster, so results can't be applied in a large raster.
Here I present a comparison with 4 different approaches, the file used is a ...
- 13.8k
5
votes
Accepted
Dissolving feature/field by interval using ArcGIS Desktop?
Create a new field of intervals and assign an interval to each record. Then dissolve by interval field. This can be done with field calculator and many if-elif-elses or you can use pandas module (...
- 77.8k
4
votes
Convert Raster to Numpy Array with only Arcpy and Numpy
One solution to avoid your 32-bit memory woes is to read the raster in chunks on to a memory-mapped array, then do your calculations. Something like this:
# Make a dict of data type references
...
- 246
4
votes
Accepted
NumPy memory error on large rasters
A MemoryError means that you have exhausted the memory available. If your ArcGIS is 32-bit, then that limit is 4GB. You will see this error with a large array with float64:
# Python 2.7.13 (v2.7.13:...
- 42.4k
2
votes
Accepted
Arcpy RasterToNumPyArray does not read floating point raster
Raster pixel size is generally described in bits; "INTEGER" could refer to anything from 1-, 4-, 8- 13-, 16-, 32-, to 64-bit, which makes it difficult to gauge how much RAM the working raster requires....
- 20.3k
2
votes
Replace NA's with 0 for large raster data using R?
You can use terra::subst
Example data
library(terra)
r <- rast(system.file("ex/elev.tif", package="terra"))
Solution
x <- subst(r, NA, 0)
- 11k
2
votes
Accepted
Google Earth Engine memory capacity exceeded
Your issue is most likely this line:
var outputMonthly = byMonth.filter(ee.Filter.listContains('system:band_names', 'constant').not())
.sort('system:time_start').toBands();
I assume ...
- 3,794
2
votes
Dividing/tiling input features in FME, processing independently and fanning out the output by tile
I don't think a loop solution is going to help here. It just doesn't break it up into multiple processes as you would require. But you certainly can break your data into tiles and process it one tile ...
- 13.2k
2
votes
Using Reclass in Arcpy, Error 010005: Unable to Allocate Memory
Esri does NOT use the new amendment to the reclassify code shown in their docs . They use their old former code to run reclass:
remap = "0 6 0;6 12 10;12 18 20;18 24 30;24 30 40;30 36 50;36 42 60;...
- 31
2
votes
How can I solve MemoryError if I have big list in my Arcpy code?
This is a really inefficient way to insert a repeating sequence of values, because you're building up the list with redundant data until it's as large as the number of features. You don't need to ...
- 5,188
2
votes
Google Earth Engine user limit exceeded error when applying a cloud mask on Sentinel-2 imagery
You are reprojecting the images to 20 meters. That prevents any of Earth Engine's clever pyramiding to do it's work. Read up on it here. If you zoom in to about 20m/px scale (use the inspector in the ...
- 14.8k
1
vote
Mask clusters larger than 500 pixels in GEE ends with a memory error
Any time you're using a kernel or a neighborhood, you're going to be scale limited. A kernel implicitly specifies a neighborhood and neighborhoods propagate all the way down to the inputs. So if you ...
- 12.3k
1
vote
Accepted
Memory error: unable to allocate using zonal statistics
rasterstats MemoryError issues are typically because of projection differences and in this case there is definitely a projection difference.
profile.update(driver='GTiff', crs=wcea_crs)
By doing the ...
- 69.6k
1
vote
Resolving memory issues in R spatial data joins (spatial polygons)
I don't have a solution for the whole dataset, but it is possible to run a subset without too much problems as follows:
set memory.limit (it worked after rebooting my machine)
memory.limit(size=...
- 215
1
vote
GEE unsupervised classification (clustering) of Landsat 8 cloud free composite - User memory limit exceeded
If you get "User memory limit exceeded," try increasing tileScale in the sample() call as indicated in this doc.
- 4,369
1
vote
NumPy memory error on large rasters
Slightly off topic, but anyway.
Brute force is not always the best approach, because there will be one that won't fit into memory. This will do the job for any size raster:
import arcpy, os, ...
- 23.3k
1
vote
Arcpy "MemoryError"; Get Classification Breaks on a TIF Raster; Jenks Breaks; Python
A 2400*2337 (assuming single band 32bit) raster is only ~20mb. I'm not sure why you get a MemoryError. I can create a dummy 32bit 2400*2337 ndarray in numpy and run getJenksBreaks on it "fine"* in ...
- 69.6k
1
vote
Replace NA's with 0 for large raster data using R?
I think you can use raster::calc() as perhaps a more memory efficient method. So, guessing that you have a single-layer raster you can do the following to replace NA values (used some test data here):
...
- 373
Only top scored, non community-wiki answers of a minimum length are eligible