0

I am processing more than 5k NetCDF files in ArcPy, using ArcGIS Desktop 10.5. I am doing it using the Python console of ArcMap. The process runs well, but when around 113 files are done, the process breaks down. The ArcMap gives an ERROR(attached at the end).

Does anyone know if there is a limitation with the number of files to process?

The process is:

  1. Read the netCDF files --> arcpy.MakeNetCDFRasterLayer_md

  2. Apply Zonal Statistics as table -->arcpy.gp.ZonalStatisticsAsTable_sa

  3. Export the results to a CSV

The code is:

in_fd="is the netCDF files directory. More than 5k files."
shape= "is the shapefile where the zonal statistics will be applied."

names=[]
path=[]
dic={}

for root,folder,files in os.walk(in_fd):
for file in files:
    if os.path.isfile(os.path.join(root,file)) and file.endswith('.nc'):
        path.append(os.path.join(root,file))
        names.append(os.path.basename(file))


arcpy.env.workspace = r"Z:\EMSV-064_IM_for_EMSN071_ForestFireHazard_Germany\5_Process\3_Task2\1_Data\4_FWI\MIKEL\FWI_mv.gdb"
arcpy.env.overwriteOutput = True
for i in range (0,len(path)):
    print("#############")
    print("Empieza el:   " + names[i])
    ras=arcpy.MakeNetCDFRasterLayer_md(in_netCDF_file=path[i], variable="dc", x_dimension="longitude", y_dimension="latitude", out_raster_layer=names[i][:-3], band_dimension="", dimension_values="", value_selection_method="BY_VALUE")

    table=arcpy.gp.ZonalStatisticsAsTable_sa(shape, "Id", ras, os.path.join(r"Z:\EMSV-064_IM_for_EMSN071_ForestFireHazard_Germany\5_Process\3_Task2\1_Data\4_FWI\MIKEL\Statistics_DC","id_"+str(names[i][13:21])), "DATA", "ALL")



    with arcpy.da.SearchCursor(table,"*") as cursor:
        for row in cursor:
            dic[names[i][13:21]]=row[4],row[5],row[7],row[8]



    ls=[]
    vls=[(k,)+v for k,v in dic.items()]
    for row in vls:
        ls.append(row)


    if os.path.exists(r"Z:\EMSV-064_IM_for_EMSN071_ForestFireHazard_Germany\5_Process\3_Task2\1_Data\4_FWI\MIKEL\CSV_mv\FWI_DC.csv"):
        os.remove(r"Z:\EMSV-064_IM_for_EMSN071_ForestFireHazard_Germany\5_Process\3_Task2\1_Data\4_FWI\MIKEL\CSV_mv\FWI_DC.csv")
    with open(r"Z:\EMSV-064_IM_for_EMSN071_ForestFireHazard_Germany\5_Process\3_Task2\1_Data\4_FWI\MIKEL\CSV_mv\FWI_DC.csv","wb")as out:
        csv_out=csv.writer(out)
        csv_out.writerow(['Fecha','MIN','MAX','MEAN','STD'])
        for row in ls:
            csv_out.writerow(row)

    arcpy.Delete_management(ras)
    arcpy.Delete_management(table)

The error is:

enter image description here

3
  • 1
    SAE messages are really an issue for Esri Tech Support. You didn't specify the version in use, or give an indication of trying a standalone script, or trying the 64-bit Background Geoprocessing environment, so there isn't much GIS SE can offer.
    – Vince
    Mar 29, 2021 at 12:54
  • 1
    You say this happens around the 113th file. What happens when you process the 112th to 115th files individually? And the the same four files using the same code tweaked to only do those four?
    – PolyGeo
    Mar 29, 2021 at 23:43
  • @PolyGeo. If I process the files individually, it works well, and if I run the process with 112th, 113th, 114th and 115th files also. I suggest that It is a space issue. I sent the error report to Esri, asking about the problem.
    – MikV89
    Mar 30, 2021 at 6:32

1 Answer 1

1

Your issues sounds very similar to a problem I had.

I very recently had to do a large amount of processing using the zonal stats as table tool and was hitting a catastrophic error that was completely bombing out my code, no error trapping could capture it.

I discovered the issue was with the zonal stats tool unable to process zones that were overlapping polygons. Rather failing gracefully and allowing me to trap the problem it just killed the main code.

With some research on the internet I finally managed a work around which I document here on another Q&A. I spawn subprocesses and run the zonal stats code in the subprocess, when it inevitably failed it returned to the calling code which was able to skip and continue.

1
  • Thanks, @Hornbydd, it's very useful.
    – MikV89
    Apr 6, 2021 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.