I have a code written in Python using arcpy where I am looping through all of the features in a feature class, performing multiple operations. It works correctly and runs for a while until it doesn't. I was running in IDLE at first and the code would just stop; no error message, no warnings, nothing. All I would see is the RESTART text in the Shell. This was confusing, but I'd re-run and it would again process away on other features until, again, it didn't (I had a snippet handling features already successfully processed). I moved to PyCharm to see if the same behaviour would be observed and it does the same, but now I get an error message but it doesn't display as a typical PyCharm error; it is just text:


What is going on here? I have thousands of features to process and it is going to take multiple days. I don't want to babysit this thing.... here is the code in full:

arcpy.SelectLayerByAttribute_management(layer1, 'NEW_SELECTION', '"Proccessed" = \'InComplete\'')
total = int(arcpy.GetCount_management(layer1).getOutput(0))

current = 1
print 'Start time = {}'.format(datetime.datetime.fromtimestamp(time.time()).strftime('%b %d, %Y @ %H:%M:%S'))
with arcpy.da.UpdateCursor(points,['SHAPE@', 'wtrshd_ha', 'runoff_ca', 'wtrshd_slp', 'dem','wtrshddist', 'max_elev', 'ids', 'proccessed']) as cursor:
    for point in cursor:
            ids = point[7]
            if point[8] == 'Complete' or point[8] == 'Error':
            print 'Working on point {} of {}'.format(str(current), str(total))
            dsc = arcpy.Describe(dem)
            arcpy.env.extent = dsc.Extent

            #print 'Creating watershed'
            arcpy.gp.Watershed_sa(flow_direction, point[0], 'watersheds')
            arcpy.gp.RasterToPolygon_conversion('watersheds', 'watershed', "NO_SIMPLIFY")
            arcpy.AddField_management('watershed', 'w_id', 'LONG')
            arcpy.CalculateField_management('watershed', 'w_id', '{}'.format(str(ids)))

            dsc = arcpy.Describe('watershed')
            arcpy.env.extent = dsc.Extent

            #print 'Assigning watershed area to point'
            area = 0
            for r in arcpy.SearchCursor('watershed'):
                area += (r.getValue("shape_area")) / float(10000)
            point[1] = area

            #print 'Assigining maximum elevation to point'
            arcpy.gp.ExtractByMask_sa(dem, 'watershed', 'ext_dem1')
            arcpy.gp.ExtractByMask_sa('slope', 'watershed', 'slope1')
            max_elv = (arcpy.GetRasterProperties_management("ext_dem1", "MAXIMUM")).getOutput(0)

            #print 'Setting maximum distance for cost path
            ras = SetNull(Raster("ext_dem1"),Raster("ext_dem1"), '"Value" <> {}'.format(max_elv))
            arcpy.gp.RasterToPoint_conversion(ras, 'to')
            arcpy.PointDistance_analysis(point[0], "to", out_loc + "\\dist1.dbf")
            dist = 0
            for f in arcpy.SearchCursor(out_loc + "\\dist1.dbf"):
                if f.getValue("distance") > dist:
                    dist = f.getValue("distance")

            #print 'Creating cost path to determine length of watershed
            cbl = CostBackLink(point[0], "slope1", float(dist*1000))
            cp = CostPath("to",  "slope1", cbl, 'BEST_SINGLE')
            distance = 0
            for r in arcpy.SearchCursor('path'):
                distance += r.getValue("Shape_Length")

            point[3] = float((float(max_elv) - float(point[4])) / float(distance))                 
            point[5] = distance
            point[6] = max_elv
            del cbl,cp,distance

            #print 'Assigining runoff coefficient to point'
            arcpy.gp.ExtractByMask_sa(runoff, 'watershed', 'ext_runoff')
            ras_count = 0
            for r in arcpy.da.SearchCursor('ext_runoff', ["VALUE", "COUNT"]):
                ras_count += r[1]
            runoff_c = 0
            for r in arcpy.da.SearchCursor('ext_runoff', ["VALUE", "COUNT"]):
                runoff_c += (float(r[0])/float(100)  * float(r[1])/float(ras_count))  
            point[2] = runoff_c
            point[8] = 'Complete'

                print 'Append not working'

            current += 1
            print str(ids) + ' No Worky'
            total -= 1
            point[8] = 'Error'

PyCharm output:

Working on point 54 of 2464
Cost Distance Mapping
Please wait ...
Working on point 55 of 2464
  • Have you tried running it in the python window of ArcCatalog for point 55? I use neither shells as it just adds complexity, you could try running in a CMD window, it would have the least overhead. It could have something to do with your %TMP%, you could try changing your environment variables TEMP and TMP to point to a device with more storage than C drive. If the problem persists add a bunch more AddMessage statements until you can find the exact line that fails and (hopefully) the exact feature that fails. Sorry I can't be more specific but there's not much info on the likely fail point. – Michael Stimson Nov 29 '17 at 4:56
  • Yeah, I was looking at gis.stackexchange.com/questions/15300/… and realize I might be out of luck as I am implementing most of what is said. I will try the kill and restart I think next. My workspaces are defined to a network drive. – D_C Nov 29 '17 at 4:57
  • Can you avoid using network drives? I have found these sort of quirky failures occur with extended transactions and momentary network hiccups. In some cases it can be devastating to have a network hiccup, completely corrupting an entire feature class. This does not apply to enterprise databases which must have failure recovery mechanisms that are superior to OS network dropped packet recovery. But the %TEMP% to which I refer is defaulted to C:\Users\...\AppData\Local\Temp on Windows 7 can have problems if quota management is enabled on your profile, better to make it C:\Temp (or D:\Temp). – Michael Stimson Nov 29 '17 at 5:15
  • I can avoid network drives but the way the IT dept have them set up have the write speed faster to network than to my C:! Chatting with them, they said that the network runs enterprise but that there are so many layers of complexity it is hard to tell how it is linked to my problem (at least that is my interpretation of their gibberish). I am currently in a groove with current processing (up to 350 features run), so I will make changes to the current code once it crashes. Thanks for the input – D_C Nov 29 '17 at 16:27
  • 1
    No worries. Despite what I.T. say, as they will be understandably defensive about their network, in my experience (16 years with ArcGIS) Esri ArcGIS software generally does not recover from dropped packets like other software does, especially during editing. I would copy the feature class points to either a local drive or in_memory as it is being modified, the rest can stay where/as it is, then when successful copy overwriting the input. This should fix the stability issues; Try to avoid editing over the network ever with ArcGIS, I've seen too many corrupt shapefiles and geodatabases. – Michael Stimson Nov 29 '17 at 22:30

I am currently looping fine now that I have wrapped the .py script inside another .py script which calls it whenever it crashes as proposed in python.multiprocessing and "FATAL ERROR (INFADI) MISSING DIRECTORY"

Thanks Micheal as well as you mentioned many things that I should be doing in all further scripts as well.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.