3

I have a distribution of points on a geometric network. I have code to select a point, run an upstream trace on the network, select intersecting polygon features, sum some attributes from those polygons, and write the summation to the selected point.

The problem with the trace geometric network tool is that it returns a group layer. My stand-alone script approach (not shown) would not return this group layer to Python. I can run the script in the Python window of an ArcGIS 10.5 project though.

My script generates the group layer and adds it to the table of contents. I exploit the trace to get a selection and calculate my summation. Then I remove the trace group layer output and continue to the next point. After a few thousand iterations ArcGIS offers up a message box with the text:

TooboxEXT Out of Memory

If I do not rename the group layer output from the trace geometric tool each iteration arcpy returns an error that the group layer already exists even though I can watch it get removed from the table of contents. This suggests that the arcpy mapping module is not actually removing the group layer or that something is getting stuck in memory.

My question: How do I remove or delete the group layer so that I do not end up with an out-of-memory error?

Here is the code.

import sys
import traceback
import numpy
import arcpy

inNodes = r'Z:\GISpublic\GerryG\WaterResources\WRIA1Growth\WRIA1_Growth_Outputs.gdb\flags'
inSSHIAPsegCat = r'Z:\GISpublic\GerryG\WaterResources\WRIA1Growth\OtherSupportingData\joinedandsummed.shp'
inGeometricNetwork = r'Z:\GISpublic\GerryG\WaterResources\WRIA1Growth\WRIA1_Growth_Outputs.gdb\gfd\gfd_Net'
mxd = arcpy.mapping.MapDocument("CURRENT")
fieldList = arcpy.ListFields(inNodes)
fieldList2 = []
for field in fieldList:
    fieldList2.append(field.name)
if "sum_1_4" not in fieldList2:
    arcpy.AddField_management(inNodes, "sum_1_4", "DOUBLE", "", "", "", "", "NULLABLE", "NON_REQUIRED", "")  
if "sum_1_6" not in fieldList2:
    arcpy.AddField_management(inNodes, "sum_1_6", "DOUBLE", "", "", "", "", "NULLABLE", "NON_REQUIRED", "")         
inNodessLayer = arcpy.MakeFeatureLayer_management(inNodes)
inSSHIAPsegCatLayer = arcpy.MakeFeatureLayer_management(inSSHIAPsegCat)
watershedsFields = ['sum1_4', 'sum1_6']
nodesFields = ["SHAPE@", "sum_1_4","sum_1_6"]
counter = 0

with arcpy.da.UpdateCursor(inNodessLayer, nodesFields) as cursor:
    for row in cursor:
        if row[1] == -999 or row[1] >= 0 :
            pass
        else:
            tracelayers = "tracelayers"+str(counter)
            traceselect = tracelayers +"/Flowlines"
            arcpy.TraceGeometricNetwork_management(inGeometricNetwork, tracelayers, row[0], "FIND_UPSTREAM_ACCUMULATION", "", "", "", "", "", "NO_TRACE_ENDS", "NO_TRACE_INDETERMINATE_FLOW", "", "", "AS_IS", "", "", "", "AS_IS")
            arcpy.SelectLayerByLocation_management (inSSHIAPsegCatLayer, "INTERSECT", traceselect)
            field = arcpy.da.TableToNumPyArray (inSSHIAPsegCatLayer, 'sum1_4', skip_nulls=True)
            sum1_4 = field['sum1_4'].sum() 
            field = arcpy.da.TableToNumPyArray (inSSHIAPsegCatLayer, 'sum1_6', skip_nulls=True)
            sum1_6 = field['sum1_6'].sum() 
            row[1]= sum1_4
            row[2]= sum1_6
            cursor.updateRow(row) 
            arcpy.SelectLayerByAttribute_management(inSSHIAPsegCatLayer, "CLEAR_SELECTION")
            print "remove trace now..."
            dfList = arcpy.mapping.ListDataFrames(mxd, "*")
            for df in dfList:  
                for lyr in arcpy.mapping.ListLayers(mxd, "", df):  
                    if lyr.isGroupLayer:  
                        if lyr.name == tracelayers:  
                            arcpy.mapping.RemoveLayer(df, lyr)
        counter +=1
  • 1
    Did you try making a list of grouplayers, and then after the for df loop, delete each grouplayer? Make sure you're not deleting a grouplayer that's a child of an already-deleted grouplayer. Also, did you try to isolate the problem writing a simple script that adds and removes grouplayers (without all the network tracing). If that runs without errors, it might suggest the issue is elsewhere (e.g. Tracing or SelectByLocation). – Kirk Kuykendall Sep 4 '18 at 22:50
  • I also think you need to delete the data, remove might not be enough. You cal also try setting arcpy.env.OverwriteOutput = True, maybe it will overwrite old data instead of creating new data for each row in the cursor – BERA Sep 6 '18 at 7:03
  • Have you installed 64bit Background Processing and enabled it in Geoprocessing Options. This fixes most all out-of-memory errors for me. – klewis Sep 7 '18 at 18:53
2

This error suggests the following workarounds noted on this page:

https://support.esri.com/en/technical-article/000014733

  • chunk out data into smaller datasets and run the script more times
  • disable background processing
  • set scratch geoprocessing geodatabase to a disk location where more disk space is available
1
+50

Consider these options:

  • Turning off geo-processing login
  • Setting the addOutputsToMap to false
  • Pausing the display speeds things up
  • As it appears you are processing hydrological data consider "chunking" it by catchment ID
  • 1
    Setting addOutputsToMap = False, setting overwriteOutputes = False, and turning off geoprocessin logging solved the out of memory problem (although I did not experiment with which combination actually did the trick). Note, the initial runs of the script had the display paused and background processing disabled so those did not have an affect on the memory issue. The initial run of the script has the scratch geoprocessing geodatabase set to a disk location with ample memory available. – GBG Sep 10 '18 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.