1

I am trying to use this solution to clean up small holes in my rasters. It uses focal statistics on the no data cells to replace them with a local average. The raster have a small number of unique values so using MEAN in focal statistics increases the number of values and gives dodgy results (I want a hole in a section where value = 100 to be 100 and not 98.9). The shouldn't be a problem because I can use MAJORITY in focal statistics.

What is stange though is that this python code takes 9 minutes to run:

filled = sa.Con(sa.IsNull(in_raster),sa.FocalStatistics(in_raster,
                     sa.NbrCircle(50, "CELL"),'MEAN'), in_raster)

still a little slow but manageable.

Then this code took 4 hours and 26 minutes to run (on the exact same in_raster):

filled = sa.Con(sa.IsNull(in_raster),sa.FocalStatistics(in_raster,
                    sa.NbrCircle(50, "CELL"),'MAJORITY'), in_raster

The only difference is mean vs Majority. Could anyone explain to me why there is such a difference in performance and if there is anything I can do about it?

I have a slightly laborious work-around of reclassifying the output to return to my small set of unique values, but I'd like to understand what seems like a simple solution is unworkably slow

  • 1
    Majority involves much more data fiddling, mean requires single pass. I would calc it first and do con after – FelixIP Mar 25 '16 at 19:54
  • You could also experiment with the size of the neighborhood. 50 cells seems excessive to match the surrounding pattern, if that is your goal. And maybe you could round the mean if you want integer results. – RHB Mar 27 '16 at 11:49
2

I tested my idea in comments using this (perhaps not the most Pytonic and efficient) script:

import numpy,time
# create list of integers
N=int(1e6)
aList=[]
for i in range(N):
    v=int(numpy.random.random()*50)
    aList.append(v)
## calculate mean
t0 = time.time()
for repeat in range(100):
    total=0;count=0
    for v in aList:
        total+=v;count+=1
    mean=float(total)/count
print ('it took %i seconds to compute mean = %s' %((time.time()-t0),mean))
## calculate majority
t0 = time.time()
for repeat in range(100):
    aDict=dict();countMax=0
    for v in aList:
        count=aDict.get(v,0)+1
        aDict[v]=count
        if count<countMax:continue
        countMax=count
        majority=v
print ('it took %i seconds to compute majority = %i' %((time.time()-t0),majority))

RESULT:

  • it took 27 seconds to compute mean = 24.510802
  • it took 47 seconds to compute majority = 10

Does not explains massive difference observed.

NOTE:

I found that map algebra on rasters stored in geodatabase is much slower (and often unreliable) compare to algebra on humble grids.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.