Running ArcToolbox without ArcMap?

Question

I am following the accepted answer for the question: Batch conversion of netCDF to Raster in ArcGIS using Python, obviously, to convert netcdfs. The code taken from the above answer (pasted below) works fine on a single NetCDF. However, if I run the same function on many .nc files by making inNetCDF variable a parameter, ArcMap breaks down. It crashes with ArcGIS 10.1 and hangs with ArcGIS 10.2.2 (on two different machines) after generating about 500 or 600 images.

I suspected that the issue is that ArcMap can't handle this many layers (more than 1000 considering the extracted nc layers). So I added:

      arcpy.env.addOutputsToMap = False

to forbid adding output to the content index, but the crashes still remain.

Is it possible to run the involved ArcToolbox functions like CopyRaster_management outside of ArcMap from command line python and, if not, how do I avoid the crash problem?

--code--

def extractAllNetCDF():

    variable = "RRt_10m"
    x_dimension = "lon"
    y_dimension = "lat"
    band_dimension = ""
    dimension = "time"
    valueSelectionMethod = "BY_VALUE"

    outLoc = "E:/New Folder/"
    inNetCDF = "E:/netCDFFiles/RRt.nc"

    nc_FP = arcpy.NetCDFFileProperties(inNetCDF)
    nc_Dim = nc_FP.getDimensions()

    for dimension in nc_Dim:
        top = nc_FP.getDimensionSize(dimension)
        for i in range(0, top):
            if dimension == "time":

                dimension_values = nc_FP.getDimensionValue(dimension, i)
                nowFile = str(dimension_values)

                #THIS IS THE NEW CODE HERE
                dv1 = ["time", dimension_value]
                dimension_values = [dv1]
                #END NEW CODE

                arcpy.MakeNetCDFRasterLayer_md(inNetCDF, variable, x_dimension, y_dimension, nowFile, band_dimension, dimension_values, valueSelectionMethod)
                arcpy.CopyRaster_management(nowFile, outLoc + nowFile + ".img", "", "", "", "NONE", "NONE", "")
                print dimension_values, i 

---

Here's the outcome using python from the command line. The script works well except for needing a few imports like arcpy and sys. There was still a crash, but the number of processed images increased 10 fold to about 7000. This time, memory seems to be at issue, with 1.8G consumed before the crash. I am trying the 64 bit background processing as suggested.

Answer 1

You can definitely run your code outside of ArcMap and you probably should.

You can save your code in a .py file in any text editor or IDE (take a look at PyScripter if you don't want to buy a commercial one, I am using Wing IDE). You can then run your code outside of ArcMap. Provided that you have Python path set and have associated Python with .py files, you can just double-click it in the Windows Explorer.

By default, Python 32bit is installed along with ArcGIS. However, when working with larger datasets, you might run out of memory which might be the case for you. A good thing to check is to have the Task Manager open and see how much run RAM is used by python.exe process running. You might get some strange things happening already at 2GB RAM consumption (due to 32bit application architecture). When you run large data processing within ArcMap there is always a certain layer of overhead which you definitely want to avoid.

Consider installing 64bit Python. Make sure you run your script .py file with Python 64bit. You can use this command in cmd:

C:\Python27x64\python.exe C:\GIS\yourscript.py

This will make sure you run your script with Python 64bit. I was able to process large datasets while Python was consuming up to 15GB of RAM with no problem at all. I don't know how resource intensive is generating those rasters, however if your machine lacks resources, you could try to complete your data processing in multiple runs (provided that the RAM use is getting bigger and bigger as more files are being processed in the same run).