Clearing cache memory using python?

Question

In this script I use 2 nested 'for' loops to extract a series of variables by month (month = 0 - 11) from 5 netcdf files and then mosaic together each extracted variable. The script runs but after processing about 3 variables, I get the following error statement:

ExecuteError: ERROR 999999: Error executing function. Failed to copy raster dataset Failed to execute (MosaicToNewRaster).

My suspicion is that the cache memory has been exceeded. I tried using arcpy.Delete_management(SCRATCH) but then received the error that the scratch space no longer existed after only one variable was processed. Maybe I am putting that statement in the wrong place? Any other suggestions?

# Import arcpy module
import arcpy
import os

# Set Geoprocessing environments
arcpy.env.scratchWorkspace = "C:\\WORKSPACE\\Process_4"
SCRATCH = arcpy.env.scratchWorkspace


# Script arguments
ENA = arcpy.GetParameterAsText(0)
if ENA == '#' or not ENA:
    ENA = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_ena_eh5_2030-2034_avg_trim_v4.nc" 

NRM = arcpy.GetParameterAsText(1)
if NRM == '#' or not NRM:
    NRM = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_nrm_eh5_2030-2034_avg_trim_v4.nc"

PNW = arcpy.GetParameterAsText(2)
if PNW == '#' or not PNW:
    PNW = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_pnw_eh5_2030-2034_avg_trim_v4.nc" 

PSW = arcpy.GetParameterAsText(3)
if PSW == '#' or not PSW:
    PSW = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_psw_eh5_2030-2034_avg_trim_v4.nc" 

SRM = arcpy.GetParameterAsText(4)
if SRM == '#' or not SRM:
    SRM = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_srm_eh5_2030-2034_avg_trim_v4.nc" 

Output_Location = arcpy.GetParameterAsText(5)
if Output_Location == '#' or not Output_Location:
    Output_Location = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\summary\\" 


#Variable list
for variable in ['SNOW', 'SMR', 'SWI', 'T0',  'T33', 'TA']:


#    SCRATCH = arcpy.env.scratchWorkspace               I attempted to place SCRATCH within the loop thinking that it would be recreated.  Nope.

    #Month Iteration
    for month in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]:
        ENA_Layer = SCRATCH +"\\ENA"+variable+str(month)
        NRM_Layer = SCRATCH +"\\NRM"+variable+str(month)
        PNW_Layer = SCRATCH +"\\PNW"+variable+str(month)
        PSW_Layer = SCRATCH +"\\PSW"+variable+str(month)
        SRM_Layer = SCRATCH +"\\SRM"+variable+str(month)

        # Process: Make NetCDF Raster Layer (169)
        arcpy.MakeNetCDFRasterLayer_md(ENA, variable, "x", "y", ENA_Layer, "", "time "+ str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (157)
        arcpy.MakeNetCDFRasterLayer_md(NRM, variable, "x", "y", NRM_Layer, "", "time "+     str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (145)
        arcpy.MakeNetCDFRasterLayer_md(PNW, variable, "x", "y", PNW_Layer, "", "time "+ str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (133)
        arcpy.MakeNetCDFRasterLayer_md(PSW, variable, "x", "y", PSW_Layer, "", "time "+ str(month), "BY_INDEX") 

        # Process: Make NetCDF Raster Layer (121)
        arcpy.MakeNetCDFRasterLayer_md(SRM, variable, "x", "y", SRM_Layer, "", "time "+ str(month), "BY_INDEX")     


        # Process: Mosaic To New Raster (25)
        arcpy.MosaicToNewRaster_management("\"" + ENA_Layer +";"+ NRM_Layer +";"+ PNW_Layer +";"+ PSW_Layer +";"+ SRM_Layer +"\"", Output_Location, variable+"_"+str(month)+"2032", "PROJCS['WGS_1984_Lambert_Conformal_Conic',GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Lambert_Conformal_Conic'],PARAMETER['false_easting',0.0],PARAMETER['false_northing',0.0],PARAMETER['central_meridian',-102.3000030517578],PARAMETER['standard_parallel_1',30.0],PARAMETER['standard_parallel_2',60.0],PARAMETER['latitude_of_origin',52.0],UNIT['Kilometer',1000.0]]", "", "1", "1", "MEAN", "FIRST")

   arcpy.Delete_management(SCRATCH)

Answer 1

FeatureLayers and RasterLayers are layers created in memory, not in the scratch workspace (in background processing they create a file referencing the memory position but that is all).

In order to remove those layers residing in memory, you need to explicitly delete them one by one using arcpy (they do not fall out of scope until Python exits). Note that deleting the variable holding them in a Pythonic way (del varname) will NOT work.

Your code needs to be revised as follows (changes in the end of the code block):

    for month in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]:
        ENA_Layer = SCRATCH +"\\ENA"+variable+str(month)
        NRM_Layer = SCRATCH +"\\NRM"+variable+str(month)
        PNW_Layer = SCRATCH +"\\PNW"+variable+str(month)
        PSW_Layer = SCRATCH +"\\PSW"+variable+str(month)
        SRM_Layer = SCRATCH +"\\SRM"+variable+str(month)

        # Process: Make NetCDF Raster Layer (169)
        arcpy.MakeNetCDFRasterLayer_md(ENA, variable, "x", "y", ENA_Layer, "", "time "+ str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (157)
        arcpy.MakeNetCDFRasterLayer_md(NRM, variable, "x", "y", NRM_Layer, "", "time "+     str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (145)
        arcpy.MakeNetCDFRasterLayer_md(PNW, variable, "x", "y", PNW_Layer, "", "time "+ str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (133)
        arcpy.MakeNetCDFRasterLayer_md(PSW, variable, "x", "y", PSW_Layer, "", "time "+ str(month), "BY_INDEX") 

        # Process: Make NetCDF Raster Layer (121)
        arcpy.MakeNetCDFRasterLayer_md(SRM, variable, "x", "y", SRM_Layer, "", "time "+ str(month), "BY_INDEX")     


        # Process: Mosaic To New Raster (25)
        arcpy.MosaicToNewRaster_management("\"" + ENA_Layer +";"+ NRM_Layer +";"+ PNW_Layer +";"+ PSW_Layer +";"+ SRM_Layer +"\"", Output_Location, variable+"_"+str(month)+"2032", "PROJCS['WGS_1984_Lambert_Conformal_Conic',GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Lambert_Conformal_Conic'],PARAMETER['false_easting',0.0],PARAMETER['false_northing',0.0],PARAMETER['central_meridian',-102.3000030517578],PARAMETER['standard_parallel_1',30.0],PARAMETER['standard_parallel_2',60.0],PARAMETER['latitude_of_origin',52.0],UNIT['Kilometer',1000.0]]", "", "1", "1", "MEAN", "FIRST")

        ##########################################
        # NEW LINES TO ADD TO DELETE MEMORY LAYERS
        # Delete layers from memory    
        arcpy.Delete_management(ENA_Layer)
        arcpy.Delete_management(NRM_Layer)
        arcpy.Delete_management(PNW_Layer)
        arcpy.Delete_management(PSW_Layer)
        arcpy.Delete_management(SRM_Layer)

This is not explicitly mentioned in the help files from ESRI, but it is necessary if you want to deal with data in memory. If you monitor the memory use from the task manager you will notice the drop as soon as the Delete statements are hit (in my case, from 1.6GB use it drops to 100MB in every loop).

Answer 2

A couple of tips that might help:

• Don't use the ArcGIS numpy module - it's rubbish to say the least. Instead use the open source netCDF4 library to read the data into numpy arrays and process those instead, and then use the NumPyArrayToRaster tool in ArcPy to turn the results back into an ArcGIS compatible Raster.
• Rather than using the ArcPy scratch workspace, use the Python tempfile module to create a temporary directory on your disk to write to (with perhaps something like temp_path = tempfile.mkdtemp() - you can delete the directory afterwards with shutil.rmtree(temp_path)
• As a last resort, temporary variables should drop out of scope at the end of each loop and then be collected by the garbage collecter automatically. If this doesn't work you could try invoking the garbage collector manually;

(I don't really advise this though - not very Pythonic...)

del var_name
gc.collect()

Answer 3

A year later, the better answer is to use the Multidimensional Supplemental Tools, which are described in this Esri blog post entitled Introducing the Multidimension Supplemental tools. The tools ship with a specially compiled version of NetCDF4-Python, and include a number of useful features for interacting with NetCDF data sources.

If you're doing analysis that is more complex with NetCDF objects, also look into the MGET tools which are written with multidimensional data in mind, and integrate nicely into ArcGIS (plus it's a great toolset for a variety of other problems).