Skip to main content
Title clarity
Link
Vince
  • 20.3k
  • 16
  • 48
  • 65

NumPy Memory Lossmemory error on large rasters

added 410 characters in body
Source Link
Sourav
  • 267
  • 1
  • 13

I have multiple large rasters. Their 5 percentile and 95 percentile are needed to be calculated. After loading the rasters to the ArcMap, I am using the following codes -

import numpy
import arcpy

rasters = ["raster1.tif", "raster2.tif", "raster3.tif", "raster4.tif"]

for i in rasters:
    array = arcpy.RasterToNumPyArray(i,nodata_to_value=10000000000)
    marray = numpy.ma.masked_values(array,10000000000)
    del array
    percentile5 = numpy.percentile(marray.compressed(),(5))
    percentile95 = numpy.percentile(marray.compressed(),(95))
    del marray
    print percentile5 , percentile95 

For the first two rasters, it provides the percentile values. For the third raster, it is showing memory errors as provided below.

Traceback (most recent call last):
  File "E:\test\Stat\Inlinear data.py", line 89, in <module>
    marray = numpy.ma.masked_values(array,10000000000)
  File "C:\Python27\ArcGIS10.5\lib\site-packages\numpy\ma\core.py", line 2200, in masked_values
    condition = umath.less_equal(mabs(xnew - value), atol + rtol * mabs(value))
MemoryError

Failed to execute (IntelinAir).

I am using ArcMap 10.5 in desktop with 32 GB Ram.

Is there any way to get rid of this memory issue?

I have multiple large rasters. Their 5 percentile and 95 percentile are needed to be calculated. After loading the rasters to the ArcMap, I am using the following codes -

import numpy
import arcpy

rasters = ["raster1.tif", "raster2.tif", "raster3.tif", "raster4.tif"]

for i in rasters:
    array = arcpy.RasterToNumPyArray(i,nodata_to_value=10000000000)
    marray = numpy.ma.masked_values(array,10000000000)
    del array
    percentile5 = numpy.percentile(marray.compressed(),(5))
    percentile95 = numpy.percentile(marray.compressed(),(95))
    del marray
    print percentile5 , percentile95 

For the first two rasters, it provides the percentile values. For the third raster, it is showing memory errors.

I am using ArcMap 10.5 in desktop with 32 GB Ram.

Is there any way to get rid of this memory issue?

I have multiple large rasters. Their 5 percentile and 95 percentile are needed to be calculated. After loading the rasters to the ArcMap, I am using the following codes -

import numpy
import arcpy

rasters = ["raster1.tif", "raster2.tif", "raster3.tif", "raster4.tif"]

for i in rasters:
    array = arcpy.RasterToNumPyArray(i,nodata_to_value=10000000000)
    marray = numpy.ma.masked_values(array,10000000000)
    del array
    percentile5 = numpy.percentile(marray.compressed(),(5))
    percentile95 = numpy.percentile(marray.compressed(),(95))
    del marray
    print percentile5 , percentile95 

For the first two rasters, it provides the percentile values. For the third raster, it is showing memory errors as provided below.

Traceback (most recent call last):
  File "E:\test\Stat\Inlinear data.py", line 89, in <module>
    marray = numpy.ma.masked_values(array,10000000000)
  File "C:\Python27\ArcGIS10.5\lib\site-packages\numpy\ma\core.py", line 2200, in masked_values
    condition = umath.less_equal(mabs(xnew - value), atol + rtol * mabs(value))
MemoryError

Failed to execute (IntelinAir).

I am using ArcMap 10.5 in desktop with 32 GB Ram.

Is there any way to get rid of this memory issue?

Source Link
Sourav
  • 267
  • 1
  • 13

NumPy Memory Loss

I have multiple large rasters. Their 5 percentile and 95 percentile are needed to be calculated. After loading the rasters to the ArcMap, I am using the following codes -

import numpy
import arcpy

rasters = ["raster1.tif", "raster2.tif", "raster3.tif", "raster4.tif"]

for i in rasters:
    array = arcpy.RasterToNumPyArray(i,nodata_to_value=10000000000)
    marray = numpy.ma.masked_values(array,10000000000)
    del array
    percentile5 = numpy.percentile(marray.compressed(),(5))
    percentile95 = numpy.percentile(marray.compressed(),(95))
    del marray
    print percentile5 , percentile95 

For the first two rasters, it provides the percentile values. For the third raster, it is showing memory errors.

I am using ArcMap 10.5 in desktop with 32 GB Ram.

Is there any way to get rid of this memory issue?