Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I have a script written in Python that uses ArcPy to create maps of the kernel densities of thousands of datasets. Each dataset is relatively small, but there about 50,000 sets in total.

When I run my script, it stops after about 10-15 minutes and gives the "python.exe has stopped working" popup. The Windows 7 event viewer tells me it is an Exception Code 0x0000005, which appears to be an access violation. The datasets the code works on are not the same each time, so it is not a specific set that is causing the problem.

Watching the task manager when the script is running, I noticed that python.exe is increasing its memory usage every few seconds, which makes me believe there is a memory leak somewhere. I thought Python had garbage collection? Is there something that arcPy is doing?

EDIT: So I tracked down the part of my code causing the problem, but I don't know HOW it is leaking. I think I am cleaning everything up. Does anyone spot something I missed?

def makeMap(asno, isKdens):
try:
    mxd = arcpy.mapping.MapDocument("F:\\testFiles\\testMap.mxd")   ###Reference to basemap document
    df = arcpy.mapping.ListDataFrames(mxd, "Layers")[0]     ###Get data frame reference in var
    ##++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    if isKdens:
        ###Set up layer file
        lyrFile = arcpy.mapping.Layer(r"F:\\testFiles\\output\\AS_"+asno+"_KDens.lyr")
    ##++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    else:
        ###Set up layer file
        lyrFile = arcpy.mapping.Layer(r"F:\\testFiles\\output\\AS_"+asno+".lyr")
        lyrFile.visible = True
    ###Add layer file
    arcpy.mapping.AddLayer(df, lyrFile)

    ###ref added layer
    lyr = arcpy.mapping.ListLayers(mxd)[0]
    lyr.visible = True

    #If this is a raw map, just select points.
    if isKdens==0:
        arcpy.SelectLayerByAttribute_management ("pointslyr", "NEW_SELECTION", "\"asno\" = "+asno+"")
    ##++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    #If KDens map, apply symbology
    else:
        arcpy.ApplySymbologyFromLayer_management(lyr, templateLayer)
    ##++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

    ###Zoom to selection
    df.zoomToSelectedFeatures()
    lyrExtent = lyr.getSelectedExtent()
    df.extent = lyrExtent

    ###Export to PDF!!!
    ##++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    if isKdens:
        arcpy.mapping.ExportToPDF(mxd, r"F:\\testFiles\\output\\AS_"+asno+"_KDens_Map.pdf")
    ##++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    else:
        arcpy.mapping.ExportToPDF(mxd, r"F:\\testFiles\\output\\AS_"+asno+"_Map.pdf")
except:
    print arcpy.GetMessages()
#####DELETION
del mxd, df, lyrFile, lyrExtent, lyr
print "DELETED MXD AND DFS"

EDIT 2 So, I also tried turning off logging, since I have heard that can cause memory leaks, too. There is still something leaking. I also tried using Delete_management instead of del, but that just messes things up. It almost seems like my code is not running sequentially. A delete after a reference still gives a "references before defined" error...

Anyway, now I am trying to locate the memory leak using memory analyzer packages. Out of all of them that work with Python, I only got objgraph to work. I placed a objgraph.show_growth call before and after the function whose code is above. Here are the results:

Before

tuple                         11959    +11959
function                       3643     +3643
dict                           1632     +1632
wrapper_descriptor             1086     +1086
cell                            927      +927
list                            635      +635
builtin_function_or_method      590      +590
property                        551      +551
method_descriptor               390      +390
weakref                         373      +373

After

weakref                 398       +25
wrapper_descriptor     1105       +19
tuple                 11977       +18
dict                   1644       +12
member_descriptor       172        +4
getset_descriptor       213        +3
method_descriptor       392        +2
instance                 15        +2
list                    636        +1
`_RLock                    2        +1

It doesn't look like any of my objects are leaking, though I am not quite sure what instance means in the second growth dump. Anyone have any insights?

share|improve this question
5  
Maybe you could post some pseudocode? –  Aaron Feb 11 '13 at 23:12
2  
Are you creating lots of MapDocument objects and not cleaning them up as you go by using del? –  PolyGeo Feb 11 '13 at 23:49
1  
Also, where is the output of the kernel density going? –  om_henners Feb 11 '13 at 23:59
2  
I have no experience of ArcPy, but some of the geoprocessing tools in earlier versions of ArcGIS do leak memory. Python does have garbage collection and, in my experience, memory leaks in ArcGIS scripts are rarely due to Python itself; they're either due to programmer error or to leaky geoprocessing tools. Some options for profiling memory consumption in your Python code are described here –  JamesS Feb 12 '13 at 10:13
    
@PolyGeo I have updated my original post to answer your question (as well as om_henners') and to provide a bit more information about my code. –  N-C Feb 13 '13 at 18:56
show 4 more comments

1 Answer

Just use the Delete tool. Call arcpy.management.Delete("rdlayer") after saving the layer file.

share|improve this answer
    
Thanks. I added this after saving the layer file, but python.exe still crashes. There must be some other leak elsewhere, but at least I took care of one of them! –  N-C Feb 13 '13 at 20:27
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.