0

I have a set of points representing houses (>500). I am calculating a cumulative viewshed for each house and mosaicing them onto a target raster using iterate feature selection.

One of the tools used in the model is Euclidean distance. The problem is after a certain number of iterations the euclidean distance tool doesn't complete, it gets stuck on either the initialisation phase, or the iteration phase. In the normal iterations the tool completes in 90 seconds or so. Then it gets to a certain point feature and never completes.

I then start the model again from the point it never completed, and then when I restart it it gets through it fine on the same point and iterates for another 20 or so and then fails again. I never get an error message, it just never completes. Then I start it again from the problem point and away it goes again until it fails again.

I have monitored the RAM and it never maxes out or anything, and disc space never fills up. I notice the CPU goes right down when it fails. Usually it uses 25% or so of the CPU when it goes through it normally, and then when its not completing it drops down to 2 or 3%.

So the problem isn't the point features, its not RAM or CPU or disk space. I experimented with different parallel processing parameters but that doesn't seem to solve it either.

I have included the model exported to python below.

import arcpy
from arcpy.ia import *
from arcpy.ia import *
def #  NOT  IMPLEMENTED# Function Body not implemented

def Model1():  # Model 1

    # To allow overwriting outputs change overwriteOutput option to True.
    arcpy.env.overwriteOutput = False

    # Check out any necessary licenses.
    arcpy.CheckOutExtension("3D")
    arcpy.CheckOutExtension("spatial")
    arcpy.CheckOutExtension("ImageAnalyst")

    belmullet_500randomhouses = "belmullet_500randomhouses"
    Elevation_belmulletClip = arcpy.Raster("Elevation_belmulletClip")
    constant_test_3_ = arcpy.Raster("C:\\Users\\0118752s\\Documents\\ArcGIS\\Projects\\Visibility Model\\Visibility Model.gdb\\constant_test")

    for Selected_Features, Value in #  NOT  IMPLEMENTED(belmullet_500randomhouses, [], False):

        # Process: Visibility (Visibility) (3d)
        Visibil_Elev2 = "C:\\Users\\0118752s\\Documents\\ArcGIS\\Projects\\Visibility Model\\Visibility Model.gdb\\Visibil_Elev1"
        Ground_t = "C:\\Users\\0118752s\\Documents\\ArcGIS\\Projects\\Visibility Model\\Visibility Model.gdb\\Ground_t"
        arcpy.ddd.Visibility(in_raster=Elevation_belmulletClip, in_observer_features=Selected_Features, out_raster=Visibil_Elev2, out_agl_raster=Ground_t, analysis_type="FREQUENCY", nonvisible_cell_value="ZERO", z_factor=1, curvature_correction="CURVED_EARTH", refractivity_coefficient=0.13, surface_offset="", observer_elevation="", observer_offset="1.8", inner_radius="", outer_radius="", horizontal_start_angle="", horizontal_end_angle="", vertical_upper_angle="", vertical_lower_angle="")
        Visibil_Elev2 = arcpy.Raster(Visibil_Elev2)
        Ground_t = arcpy.Raster(Ground_t)

        # Process: Point to Raster (Point to Raster) (conversion)
        Windfarm_test_PointToRaster1 = "C:\\Users\\0118752s\\Documents\\ArcGIS\\Projects\\Visibility Model\\Visibility Model.gdb\\Windfarm_test_PointToRaster1"
        with arcpy.EnvManager(extent="31715.0917218177 275593.200642369 114480.572911532 364680.998476251", snapRaster="Elevation_belmulletClip"):
            arcpy.conversion.PointToRaster(in_features=Selected_Features, value_field="OBJECTID", out_rasterdataset=Windfarm_test_PointToRaster1, cell_assignment="MOST_FREQUENT", priority_field="NONE", cellsize="C:\\Users\\0118752s\\Documents\\ArcGIS\\Projects\\Visibility Model\\Visibility Model.gdb\\Elevation_belmulletClip", build_rat="BUILD")

        # Process: Euclidean Distance (Euclidean Distance) (sa)
        EucDist_Wind2 = "C:\\Users\\0118752s\\Documents\\ArcGIS\\Projects\\Visibility Model\\Visibility Model.gdb\\EucDist_Wind2"
        Euclidean_Distance = EucDist_Wind2
        Output_direction_raster = ""
        Out_back_direction_raster = ""
        with arcpy.EnvManager(extent="31715.0917218177 275593.200642369 114480.572911532 364680.998476251", snapRaster="Elevation_belmulletClip"):
            EucDist_Wind2 = arcpy.sa.EucDistance(in_source_data=Windfarm_test_PointToRaster1, maximum_distance=None, cell_size="9.9994450318751", out_direction_raster=Output_direction_raster, distance_method="PLANAR", in_barrier_data="", out_back_direction_raster=Out_back_direction_raster)
            EucDist_Wind2.save(Euclidean_Distance)


        # Process: Raster Calculator (Raster Calculator) (ia)
        angle_t = "C:\\Users\\0118752s\\Documents\\ArcGIS\\Projects\\Visibility Model\\visibility model.gdb\\angle_t"
        Raster_Calculator = angle_t
        angle_t = ( ATanH((140-Ground_t)/ EucDist_Wind2))*57.2958
        angle_t.save(Raster_Calculator)


        # Process: Raster Calculator (2) (Raster Calculator) (ia)
        angle_t0 = "C:\\Users\\0118752s\\Documents\\ArcGIS\\Projects\\Visibility Model\\visibility model.gdb\\angle_t0"
        Raster_Calculator_2_ = angle_t0
        angle_t0 =  Con(angle_t<0,0,angle_t)
        angle_t0.save(Raster_Calculator_2_)


        # Process: Mosaic (Mosaic) (management)
        CreateConsta1_2_ = arcpy.management.Mosaic(inputs=[angle_t0], target=constant_test_3_, mosaic_type="SUM", colormap="FIRST", background_value=None, nodata_value=None, onebit_to_eightbit="NONE", mosaicking_tolerance=0, MatchingMethod="NONE")[0]
        CreateConsta1_2_ = arcpy.Raster(CreateConsta1_2_)

if __name__ == '__main__':
    # Global Environment settings
    with arcpy.EnvManager(scratchWorkspace=r"C:\Users\0118752s\Documents\ArcGIS\Projects\Visibility Model\Visibility Model.gdb", workspace=r"C:\Users\0118752s\Documents\ArcGIS\Projects\Visibility Model\Visibility Model.gdb"):
        Model1()
1

Sounds like a memory leak to me, which should be reported to Esri. I've come across these sorts of problems in the past and most recently with the zonal stats tool blowing up catastrophically, in my case no standard best practise error trapping could trap it and it killed the main execution code.

With some sleuthing and of cause excellent support from this site and GeoNet I came up with a solution that was slower but allowed the code to execute to completion. Maybe this approach will work for you.

You have a master script, a simple loop that calls another python script through the subprocess module. Below is the barebones master script, so the bulk of your current script would go into the sub-script that is getting called. In my example I pass in the buffer ID and that is enough for my subscript to start with.

import subprocess
import sys
import arcpy

def main():
    # Open dataset and read in ID's into a dictionary
    print(str(len(dictBuff)) + " rows loaded into dictionary.")

    with open(r"C:\Local\Output\failed.txt",'a') as fo:
        # Main processing loop
        idlist = dictBuff.keys()
        failed = list()
        for x in idlist:
            print("Processing buffer " + str(x))
            ret = subprocess.call([r"C:\Python27\ArcGIS10.7\python.exe" ,r"c:\Temp\Scripts\subscript.py", str(x)])
            if ret != 1:
                failed.append(x)
                fo.write(str(x) + '\n')

        print("FAILED buffers are:")
        print(failed)

if __name__ == '__main__':
    main()

My sub-script would have the basic structure of this:

import sys,os,arcpy

def main():
    bufferID = int(sys.argv[1])

    try:
        # DO STUFF

        sys.exit("success") #(bufferID,"success")
    except:
        sys.exit("failed") #(bufferID,"failed")

if __name__ == '__main__':
    main()

If the code catastrophically failed (which it did some times) the subprocess would return a value that was not 1, so I knew it bombed out, but it did not kill the master script.

2
  • I'm not sure if it is a memory leak as the memory doesn't seem to be affected at all when the tool fails. As I said it never actually fails and I don't get an error message it just gets stuck. There is never an option to send an error report either. So far I have gotten to ObjectID number 20, then 57, 61, 97, 107, 244, 247. – Tom Mar 12 at 15:38
  • In the last two fails I got the following error messages for the different tools: ERROR 000464: Cannot get exclusive schema lock. Either being edited or in use by another application or service. Failed to execute (Point to Raster). ERROR 010240: Could not save raster dataset to angle_t0 with output format FGDBR. Failed to execute (Raster Calculator (2)). Its running again now, will update if and when it fails again. I would hope eventually to run it on a >8000 set of points. Thanks for your response – Tom Mar 12 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.