Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I have looked at several posts here and on other sites but have not found a solution to the problem I am having with this python script. In this script I use 2 nested 'for' loops to extract a series of variables by month (month = 0 - 11) from 5 netcdf files and then mosaic together each extracted variable. The script runs but after processing about 3 variables, I get the following error statement:

ExecuteError: ERROR 999999: Error executing function. Failed to copy raster dataset Failed to execute (MosaicToNewRaster).

My suspicion is that the cache memory has been exceeded. I tried using arcpy.Delete_management(SCRATCH) but then received the error that the scratch space no longer existed after only one variable was processed. Maybe I am putting that statement in the wrong place? Any other suggestions?

# Import arcpy module
import arcpy
import os

# Set Geoprocessing environments
arcpy.env.scratchWorkspace = "C:\\WORKSPACE\\Process_4"
SCRATCH = arcpy.env.scratchWorkspace


# Script arguments
ENA = arcpy.GetParameterAsText(0)
if ENA == '#' or not ENA:
    ENA = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_ena_eh5_2030-2034_avg_trim_v4.nc" 

NRM = arcpy.GetParameterAsText(1)
if NRM == '#' or not NRM:
    NRM = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_nrm_eh5_2030-2034_avg_trim_v4.nc"

PNW = arcpy.GetParameterAsText(2)
if PNW == '#' or not PNW:
    PNW = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_pnw_eh5_2030-2034_avg_trim_v4.nc" 

PSW = arcpy.GetParameterAsText(3)
if PSW == '#' or not PSW:
    PSW = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_psw_eh5_2030-2034_avg_trim_v4.nc" 

SRM = arcpy.GetParameterAsText(4)
if SRM == '#' or not SRM:
    SRM = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\srf_srm_eh5_2030-2034_avg_trim_v4.nc" 

Output_Location = arcpy.GetParameterAsText(5)
if Output_Location == '#' or not Output_Location:
    Output_Location = "F:\\GISdata\\Projects\\USGSClimateChange\\climateproj\\summary\\" 


#Variable list
for variable in ['SNOW', 'SMR', 'SWI', 'T0',  'T33', 'TA']:


#    SCRATCH = arcpy.env.scratchWorkspace               I attempted to place SCRATCH within the loop thinking that it would be recreated.  Nope.

    #Month Iteration
    for month in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]:
        ENA_Layer = SCRATCH +"\\ENA"+variable+str(month)
        NRM_Layer = SCRATCH +"\\NRM"+variable+str(month)
        PNW_Layer = SCRATCH +"\\PNW"+variable+str(month)
        PSW_Layer = SCRATCH +"\\PSW"+variable+str(month)
        SRM_Layer = SCRATCH +"\\SRM"+variable+str(month)

        # Process: Make NetCDF Raster Layer (169)
        arcpy.MakeNetCDFRasterLayer_md(ENA, variable, "x", "y", ENA_Layer, "", "time "+ str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (157)
        arcpy.MakeNetCDFRasterLayer_md(NRM, variable, "x", "y", NRM_Layer, "", "time "+     str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (145)
        arcpy.MakeNetCDFRasterLayer_md(PNW, variable, "x", "y", PNW_Layer, "", "time "+ str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (133)
        arcpy.MakeNetCDFRasterLayer_md(PSW, variable, "x", "y", PSW_Layer, "", "time "+ str(month), "BY_INDEX") 

        # Process: Make NetCDF Raster Layer (121)
        arcpy.MakeNetCDFRasterLayer_md(SRM, variable, "x", "y", SRM_Layer, "", "time "+ str(month), "BY_INDEX")     


        # Process: Mosaic To New Raster (25)
        arcpy.MosaicToNewRaster_management("\"" + ENA_Layer +";"+ NRM_Layer +";"+ PNW_Layer +";"+ PSW_Layer +";"+ SRM_Layer +"\"", Output_Location, variable+"_"+str(month)+"2032", "PROJCS['WGS_1984_Lambert_Conformal_Conic',GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Lambert_Conformal_Conic'],PARAMETER['false_easting',0.0],PARAMETER['false_northing',0.0],PARAMETER['central_meridian',-102.3000030517578],PARAMETER['standard_parallel_1',30.0],PARAMETER['standard_parallel_2',60.0],PARAMETER['latitude_of_origin',52.0],UNIT['Kilometer',1000.0]]", "", "1", "1", "MEAN", "FIRST")

   arcpy.Delete_management(SCRATCH)
share|improve this question
1  
Shouldn't the last line of post arcpy.Delete_management(SCRATCH) be included in the code box? –  radek Feb 2 '12 at 16:27
1  
yes it should, I had some trouble getting the full code inserted correctly in the code box. Sorry about that. The last line is part of the original code. –  Jodi Feb 2 '12 at 17:00
    
Since you say after the 3rd iteration you have a problem, what happens when you change the iteration to start there, i.e. for month in [3, 4, 5, 6, 7, 8, 9, 10, 11]: ? Does it do another 3 iterations before hitting that error, or do you hit the error sooner ? –  Stephen Quan Feb 3 '12 at 10:56
    
If I shorten the number of months then the script will process a few more variables. The problem seems to be total number of steps which makes me suspect that the temporary (scratch) memory limit has been exceeded. –  Jodi Feb 3 '12 at 16:20

3 Answers 3

A couple of tips that might help:

  • Don't use the ArcGIS numpy module - it's rubbish to say the least. Instead use the open source netCDF4 library to read the data into numpy arrays and process those instead, and then use the NumPyArrayToRaster tool in ArcPy to turn the results back into an ArcGIS compatible Raster.
  • Rather than using the ArcPy scratch workspace, use the Python tempfile module to create a temporary directory on your disk to write to (with perhaps something like temp_path = tempfile.mkdtemp() - you can delete the directory afterwards with shutil.rmtree(temp_path)
  • As a last resort, temporary variables should drop out of scope at the end of each loop and then be collected by the garbage collecter automatically. If this doesn't work you could try invoking the garbage collector manually;

(I don't really advise this though - not very Pythonic...)

del var_name
gc.collect()
share|improve this answer
    
Thanks for your help! The files I am using are an older version of netcdf....I don't think the netCDF4 library works with these. I will try the second option. –  Jodi Feb 21 '12 at 13:57

FeatureLayers and RasterLayers are layers created in memory, not in the scratch workspace (in background processing they create a file referencing the memory position but that is all).

In order to remove those layers residing in memory, you need to explicitly delete them one by one using arcpy (they do not fall out of scope until Python exits). Note that deleting the variable holding them in a Pythonic way (del varname) will NOT work.

Your code needs to be revised as follows (changes in the end of the code block):

    for month in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]:
        ENA_Layer = SCRATCH +"\\ENA"+variable+str(month)
        NRM_Layer = SCRATCH +"\\NRM"+variable+str(month)
        PNW_Layer = SCRATCH +"\\PNW"+variable+str(month)
        PSW_Layer = SCRATCH +"\\PSW"+variable+str(month)
        SRM_Layer = SCRATCH +"\\SRM"+variable+str(month)

        # Process: Make NetCDF Raster Layer (169)
        arcpy.MakeNetCDFRasterLayer_md(ENA, variable, "x", "y", ENA_Layer, "", "time "+ str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (157)
        arcpy.MakeNetCDFRasterLayer_md(NRM, variable, "x", "y", NRM_Layer, "", "time "+     str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (145)
        arcpy.MakeNetCDFRasterLayer_md(PNW, variable, "x", "y", PNW_Layer, "", "time "+ str(month), "BY_INDEX")

        # Process: Make NetCDF Raster Layer (133)
        arcpy.MakeNetCDFRasterLayer_md(PSW, variable, "x", "y", PSW_Layer, "", "time "+ str(month), "BY_INDEX") 

        # Process: Make NetCDF Raster Layer (121)
        arcpy.MakeNetCDFRasterLayer_md(SRM, variable, "x", "y", SRM_Layer, "", "time "+ str(month), "BY_INDEX")     


        # Process: Mosaic To New Raster (25)
        arcpy.MosaicToNewRaster_management("\"" + ENA_Layer +";"+ NRM_Layer +";"+ PNW_Layer +";"+ PSW_Layer +";"+ SRM_Layer +"\"", Output_Location, variable+"_"+str(month)+"2032", "PROJCS['WGS_1984_Lambert_Conformal_Conic',GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Lambert_Conformal_Conic'],PARAMETER['false_easting',0.0],PARAMETER['false_northing',0.0],PARAMETER['central_meridian',-102.3000030517578],PARAMETER['standard_parallel_1',30.0],PARAMETER['standard_parallel_2',60.0],PARAMETER['latitude_of_origin',52.0],UNIT['Kilometer',1000.0]]", "", "1", "1", "MEAN", "FIRST")

        ##########################################
        # NEW LINES TO ADD TO DELETE MEMORY LAYERS
        # Delete layers from memory    
        arcpy.Delete_management(ENA_Layer)
        arcpy.Delete_management(NRM_Layer)
        arcpy.Delete_management(PNW_Layer)
        arcpy.Delete_management(PSW_Layer)
        arcpy.Delete_management(SRM_Layer)

This is not explicitly mentioned in the help files from ESRI, but it is necessary if you want to deal with data in memory. If you monitor the memory use from the task manager you will notice the drop as soon as the Delete statements are hit (in my case, from 1.6GB use it drops to 100MB in every loop).

share|improve this answer
    
+1 Welcome to our site! Your brief explanations and illustrations with carefully commented code are just the kinds of answers we look for here. –  whuber Sep 18 '12 at 16:30

A year later, the better answer is to use the Multidimensional Supplemental Tools, which are described in this blog post. The tools ship with a specially compiled version of NetCDF4-Python, and include a number of useful features for interacting with NetCDF data sources.

If you're doing analysis that is more complex with NetCDF objects, also look into the MGET tools which are written with multidimensional data in mind, and integrate nicely into ArcGIS (plus it's a great toolset for a variety of other problems).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.