1

As is not possible formulate, or add a solution to question:

Delete temporary files from geoprocessing in ArcGIS 10 and above , because is [closed]

I would like to make this question with same problem, I found that geoprocessing tools like arcpy.gp.SegmentMeanShift makes a lot of heavy temporal files (script is shown below) that keeps locked while the script shell still open, so you can't delete them, in the same Python script, also when tool is repeated in loop many times, script also crash.

I would like to ask for a correct way to manage this problem. I'm using Python 2.7, and ArcPy (ArcMap 10.6) package in Windows 10.

This is a part of script code that repeats many times "arcpy.gp.SegmentMeanShift_sa" on same raster (in this sample), so in this case on image shouldn't be the problem of crashing:

import arcpy
import os

# Check out any necessary licenses
arcpy.CheckOutExtension("spatial")

# segmentation variables
Spectral_Detail = "15"
Spatial_Detail = "18"
Minimun_Segment = "9"

# folders route
work_folder = os.getcwd()+"\\work_folder"
out = work_folder +"\\out"
out_OS = work_folder +"\\out_OS"


#geoprocessing 
order =0
while order < 150:
    order +=1
    print "processing " +str(order)
    id_item = "31165796"
    arcpy.gp.SegmentMeanShift_sa(work_folder +"\\out\\"+id_item+".tif",
                                 out_OS+"\\" + id_item +"_"+Spectral_Detail+"_"+Spatial_Detail+"_"+Minimun_Segment+"_"+str(order)+".img",
                                 Spectral_Detail, Spatial_Detail, Minimun_Segment, "")

When this script is executed, these files are generated in temp directory each iteration (XX = random number):

SMS_Index_XX.idx
SMS_Index_XX.mrf
SMS_Index_XX.mrf.aux.xml
SMS_Index_XX.til --> 32.768 kb
....

After more or less a hundred iterations, Python crash, whith this error: Traceback (most recent call last): File "D:/TALLER_SSD/20190405_SEGMENTACION/python/test/segmentacion_temporal_problem_2.py", line 212, in <module> Spectral_Detail, Spatial_Detail, Minimun_Segment, "") File "C:\Program Files (x86)\ArcGIS\Desktop10.6\ArcPy\arcpy\geoprocessing\_base.py", line 510, in <lambda> return lambda *args: val(*gp_fixargs(args, True)) ExecuteError: ERROR 999998: Unexpected Error. Failed to execute (SegmentMeanShift).

This crash is not the problem, errors can be controled, the problem still in main cuestion, temp files can't be deleted while script is running Python's Shell, filling disk space in many iterations. Even when script finish well and quit, temporary files are not deleted.

Any solution, or something missed?

  • 1
    This appears to be a solution in search of a problem. Under normal circumstances, any temporary files created by a process are needed by that process and it deletes them when no longer needed. This multiprocessing solution is wasteful of system resources, bloats your code, and reduces portability of your scripts. The better solution would be to ignore active management of temp files. – Vince May 22 at 12:13
  • That's the main problem, after finishing this geoprocess, temporal files keeps in folder, even in next iteration (i.e. another geoprocess), this files can't be deleted, because its locked by the running script, when you do a several iterations, geoprocess start to fail (Error 999999) , even when disk it's not full. Doing this solution, (instead one by one , temporal files are erased each 50 iterations) the problem is solved, managing a big amount of iterations, and data. For me change Temporal disk location it's not a solution. – Alx May 22 at 13:04
  • 1
    It's been 25 years since I've last seen intermediate products left in a directory by Arc processes without a catastrophic failure. There's something missing from this question. – Vince May 22 at 13:41
  • did you try to use a geodatabase for output? – Andreas Müller May 23 at 9:19
  • I've tried to save raster output in database with same result. – Alx May 24 at 10:37
0

A solution that works for me it's to divide script into two:
- first script iterate on an item list, pass an "id_item" to the second, and erase temporal files
- second script do geoprocess for each item.

Note: in sample code there is a comented sample geoprocess, it's only for take an idea of what could be.

First script (01_call_process.py):

import subprocess
import os

script = os.getcwd()+"\\02_process.py" # script process script route

item_list = ["1","2","3"]


for item in item_list: #iterate through items
    # run script process, passing item value, without open cmd window
    subprocess.call ("python " + script +" "+item, shell=True)  

    #delete temporal files, by Kevin @ stackoverflow
    tmp = os.environ.get("TMP")
    if tmp:
        for f in os.listdir(tmp):
            name = os.path.join(tmp, f)
            try:
                if os.path.isfile(name):
                    os.remove(name)
                else:
                    os.removedirs(name)
            except OSError as e:
                # Specific exception handling could be done based on the 'errno'.
                pass # skip files that are in use on Windows or otherwise cannot be removed.d.


Second script, placed in the same folder as first(02_process.py) will do the job:


import sys
import os
id_item = str(sys.argv[1]) #get passed data
raster = os.getcwd()+"\\"+id_item+".tif"

#sample code in order to visualice results in log files... 
file = open(os.getcwd()+"\\Sample_file_"+id_item+".txt", "w")
file.write(raster+" Processed") 
file.close() 


# do geoprocess.... 

When second script is finnished, first script keeps running until the end of the loop.

This solution works for me, but make two scripts more complicated to manage workflow. But at less prevent disk to get full and script to crash.

Any other solutión?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.