I have a short script that I am using to process roughly 5,000 NetCDF files so that I can extract a single raster layer. This is not difficult, and I have a short script that does it as follows:
x_dimension = "lon" y_dimension = "lat" band_dimension = "" dimension = "time" valueSelectionMethod = "BY_VALUE" outLoc = "C:/Out_location" import_path = 'C:/import_location' for root,dirs,files, in os.walk(os.path.abspath(import_path)): for f in files: outfile = os.path.join(root,f) write_file = ('%s' %outLoc)+f +".img" layer = 'name' arcpy.MakeNetCDFRasterLayer_md(outfile, variable, x_dimension, y_dimension, layer,'#', '#', valueSelectionMethod) arcpy.CopyRaster_management(layer,write_file, "#", "#", "#", "NONE", "NONE", "") del(write_file) arcpy.Delete_management(layer) print 'Done'
This script works fine, but it has an aggregious memory leak that only allows it to process roughly 125 files at a time.
What is happening is that either function being used to convert my netCDF to raster automatically caches the raster it converts into memory and hence accumulates a raster with each loop that is run (which I doubt since I am using a temporary variable for that raster name that is overwritten with each loop); or that the function that creates a raster copy is caching the raster it creates for, for some reason, in memory.
Hence, I am trying to configure it such that my memory leak is solved so that I can process the entirety of my 5000 files at once.
I am beginner at using python, but from what I understand a way to halt a memory leak via using subprocesses, but quite frankly, even after my research, I am not sure how to do this.
Things I have tried:
I have tried deleting the temporary file that is created by raster creation tool but nothing happens
I have input a string that is taken as the name of the output raster for the NetCDF to raster tool that is input into the Raster copy tool and is overwritten every loop
- I have tried deleting the output raster from the copy raster function in every loop which eliminated the leak, except it deleted it from my save location as well which indicates that this is a linked to raster creation function.
I am at a loss, hence any expedited solutions would be great!