7

I have a Con() statement that creates two "extra" scratch files during it's processing, but that do not get cleaned up once the operation is over and the output is saved.

    r=arcpy.sa.Con(elv, rCtypeAlpine, rCtype, 'VALUE > 10000')
    r.save('covtypeprms_a')

prior to saving r, I can see that there is also a scratch file for that object. This gets renamed with r.save(). But I'm still left w/two directories (grids?).

I've been finding and deleting them using arcpy.ListRasters('t_t*'), which works, but this is really just a hack. Lock files seem to get left behind with this approach. I've also tried del r to get rid of the handle, but not destroy the copy on disk ('covtypeprms_a').

Why are these appearing and is there more precise way to reference these intermediate objects (so's I can delete them!)?

update on enviroment

product: Desktop 10.1 sp1 OS: Win7 Enterprise sp1

python: Enthought 32-bit in use via Wing IDE

sys.executable yields 'C:\Users\joeblow\AppData\Local\Enthought\Canopy32\User\Scripts\python.exe'

Improve this question
5
  • I should probably add that my arcpy.env.workspace and arcpy.env.scratchWorkspace are both set to the same directory and that all inputs except for the elv raster are local. This is in 10.1, too. Both the rasters have the exact same properties of min, max, mean, std in the values, fwiw.
    – Roland
    Jun 12, 2013 at 15:13
  • I found a bug report in 10.1 that's probably the root of the problem. Nuts. Fact is, because my Con() example produces multiple scratch files, their solution doesn't clean everything up.
    – Roland
    Jun 12, 2013 at 16:02
  • another update: the lock persist and are not delete-able with things like os.remove() even after the scratch grids are destroyed, and even after del arcpy. It's like the python process itself owns the lock. They disappear when I close my IDE (WingIDE).
    – Roland
    Jun 12, 2013 at 17:32
  • 1) What happens when you del r (This will NOT delete the raster, just the object, since you've saved it.) 2) the bug you linked only applies to the ArcGIS python window - under what environment are you working?
    – Curtis Price
    Jun 12, 2013 at 17:58
  • 1) Right. That's not clear about del r in my write-up. Was trying to release the handle and allow python to remove the *.lock file even if the grid persisted. Didn't work. 2) Certainly the bug isn't an exact match, but figured since the same compile of arcpy would be running in both places (my IDE and my ArcDesktop python prompt), it was likely related. Environment details posted to original question.
    – Roland
    Jun 12, 2013 at 19:20

2 Answers 2

Reset to default
2

Figured out the issue. Part of the problem was that I reduced the scope of my real problem too much when I asked the question. Had some good help from Barbara S. @ ESRI on this.

The soundbite is that arcpy.env.mask gets applied before actually executing the function, creating new grids.

Two of the inputs to my Con() are raster objects (elv, rCtype). Because these objects had a larger spatial extent than I needed for the output from the Con(), I had set arcpy.env.mask and arcpy.env.extent to limit the spatial footprint of the output. I thought this would simply control which cells were accessed in the application of the Con(). What apparently is happening is that prior to processing the Con(), a reduced-footprint derivative of each of the grids was being created and used in place of my original raster inputs. These grids are not temporary and persist on the disk. In addition, because of the dynamic way in which these grids are generated, there was no variable or way to reference these new grids. Bit of a bummer since cleaning these up is a hassle (really have to leave the python session altogether).

Seems like one robust solution is to explicitly pre-filter the spatial footprint of the input grids and the delete them when the Con() is done. Actually writing out a new grid, instead of constraining data read into the Con() function, seems pretty wasteful. Maybe it allows arcpy.sa functions to better scale to very large rasters. I'm wondering if this pre-extraction must occur I should run in the "in_memory" workspace and then explicitly save stuff out. I assumed that running local to the input grids would be fastest. I'll run some tests and post to a separate question for that, I guess.

Update

Turns out that the clean up of by-products created from arcpy.env.mask varies depending on what you use for your mask. I haven't tested too many options, but my use case relied on a fgdb featureclass. There was no leftover data when I used a grid for the mask.

Improve this answer
1

I have had a similar problem (ArcGIS 10.1) where temporary rasters created by a Reclassify operation are not being deleted. Iterating over hundreds of such operations in a script, eventually caused a crash, either running out of memory if using 'in_memory' for the scratchWorkspace, or getting a directory full error if using a folder, e.g. c:\temp

I tried the workaround proposed in the related Bug Report (Bug NIM-079976) i.e. 

outReclassify = Reclassify(RasterCalc, "VALUE", remap, "NODATA")
... <use outReclassify.>
del outReclassify

However that did not remove the temporary raster. To force it to be deleted, I needed to explicitly save it, then delete it. i.e.

outReclassify = Reclassify(RasterCalc, "VALUE", remap, "NODATA")
outReclassify.save(os.path.join(arcpy.env.scratchWorkspace,'tmp_Rcls'))
... <use outReclassify.>
arcpy.Delete_management(outReclassify)

Note that if I tried saving to 'in_memory' (for performance gains) the delete operation apparently succeeds but the memory used by the python script continued to increase until it crashed. So for now I'm stuck with saving to disk with the much reduced performance associated.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.