2

I am new to scripting with ArcGIS. Although this might be an easy thing for those that know Python well, I haven't figured out how to use the subprocess module in a script.

I actually think I'm having a memory problem. I am processing a lot of shapefiles and raster datasets, and the whole processing should go on for the night, but it always stops with an error after processing about the same number of files. I read in one of the forum's answers that subprocesses.call is an old trick to solving ArcGIS memory leaks, I just don't know how to add it to the script.

The script is longer, and it's supposed to run overnight, but it never does, it always stops and I have to restart ArcCatalogue to be able to continue. Could I get some help with using the subprocess module?

In summary, the script is processing a large number of shapefiles containing point features to extract the cells from a raster that overlays those points. Within these shapefiles the features indicate areas burned by a fire (the area 75m around the point is considered burned). All shapefiles have exactly the same name but are saved in different folders. Each shapefile corresponds to a single fire, but I have to group the files before extracting the values from the raster, that's why they are in for loops.

Script:

## SIMULATION PARAMETERS
firesFolder = "C:\\Simulations\\fires"
outputFolder = "C:\\Simulations\\merge"
rasterFolder = "C:\\Simulations\\luswfr_values"
extractCSVFolder = "C:\\Simulations\\output_csv"
startPBS = 1
endPBS = 5  
startIgn = 1
endIgn = 30
fireSeasons = [2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012]
bufferDistance = "75 Meters"

# Importing modules
import os
import arcpy
from arcpy.sa import *
import glob

# Confirm license
arcpy.CheckOutExtension("Spatial")

print("\n\nRetrieving shapefiles to merge...")

# Create list of fires to merge
inputDict = {}
for pbs in range(startPBS, endPBS):
    for ign in range(startIgn, endIgn):
        for season in fireSeasons:
            inputDict[str(pbs) + str(ign) + str(season)] = glob.glob(firesFolder + "\\PBS{0}, ignitions{1}, weather {2}-{3}\\*\\spread_events.shp".format(pbs, ign, season, season+1))

print("Shapefiles retrieved\n\nMerging shapefiles...")

# Merge files
for pbs in range(startPBS, endPBS):
    for ign in range(startIgn, endIgn):
        for season in fireSeasons:
            #assuming this print statement should be indented
            print("\tStarting Merge: PBS{0}, Ign{1}, Weather{2}-{3}".format(pbs, ign, season, season+1))
            arcpy.Merge_management(inputDict[str(pbs) + str(ign) + str(season)], outputFolder + "\\PBS{0}_{1}_{2}".format(pbs, ign, str(season)[2:]))

print("Shapefiles merged\n\nBuffer Analysis of shapefiles...")

# Buffer analysis
arcpy.env.workspace = outputFolder
for pbs in range(startPBS, endPBS):
    for ign in range(startIgn, endIgn):
        for season in fireSeasons:
            print("\tStarting buffer analysis: PBS{0}_{1}_{2}".format(pbs, ign, str(season)[2:]))
            arcpy.Buffer_analysis("PBS{0}_{1}_{2}.shp".format(pbs, ign, str(season)[2:]), 
                              outputFolder + "\\BuffPBS{0}_{1}_{2}".format(pbs, ign, str(season)[2:]),
                              bufferDistance)

print("Buffer analysis complete\n\nPolygon to Raster Conversion...")

# Polygon to raster conversion
for pbs in range(startPBS, endPBS):
    for ign in range(startIgn, endIgn):
        for season in fireSeasons:
            print("\tStarting Polygon to raster conversion: BuffPBS{0}_{1}_{2}".format(pbs, ign, str(season)[2:]))
            arcpy.PolygonToRaster_conversion("BuffPBS{0}_{1}_{2}.shp".format(pbs, ign, str(season)[2:]),
                                         "intensity",
                                         outputFolder + "\\PBS{0}_{1}_{2}".format(pbs, ign, str(season)[2:]),
                                         "CELL_CENTER", "NONE", 50)

print("Polygon to Raster Conversions complete\n\nExtracting by mask...")

# Extract by mask
inRaster = rasterFolder
maskDict = {}
for pbs in range(startPBS, endPBS):
    for ign in range(startIgn, endIgn):
        for season in fireSeasons:
            print("\tStarting extract of: PBS{0}_{1}_{2}".format(pbs, ign, str(season)[2:]))
            #assuming this maskDict should be indented
            maskDict[str(pbs) + "_" + str(ign) + "_" + str(season)] = ExtractByMask(inRaster, outputFolder + "\\PBS{0}_{1}_{2}".format(pbs, ign, str(season)[2:]))

print("Extract by mask complete\n\nSaving Mask Extracts...")

# Save mask extracts
for pbs in range(startPBS, endPBS):
    for ign in range(startIgn, endIgn):
        for season in fireSeasons:
            for mask in maskDict:
                print("\tSaving mask VPBS{0}_{1}_{2}".format(pbs, ign, str(season)[2:]))
                #assuming this maskDict should be indented
                maskDict[str(pbs) + "_" + str(ign) + "_" + str(season)].save(outputFolder + "\\VPBS{0}_{1}_{2}".format(pbs, ign, str(season)[2:]))

print("Mask saving complete\n\nReclassifying files...")

closed as off-topic by PolyGeo Oct 6 '16 at 9:49

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This problem cannot or can no longer be reproduced. Changes to the system or to the asker's circumstances may have rendered the question obsolete, or the question does not include a procedure to enable potential answerers to reproduce the same symptoms. Such questions are off-topic as they are unlikely to help future readers, but editing them to include more details can lead to re-opening." – PolyGeo
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 4
    at which process does the script typically end? merge? buffer? poly to raster? - and it looks like the maskDict in the last 2 sections should be at the same indent level as the print statement? – fluidmotion Oct 11 '14 at 15:56
  • 2
    It would be helpful for you to summarize what the script is doing. The comments alone are not too helpful. – Aaron Oct 11 '14 at 17:01
  • The process ends at Merge, after merging 443 files (so right at the beginning). Yes, maskDict is at the same indent level as the print statement. – Vero Oct 23 '14 at 7:48
  • A quick summary of what the script is doing: The script is processing a very large number of shapefiles with point features and getting them to extract the cells in a raster that overlap those points and the neighbouring cells. I have a large number of shapefiles with points that indicate area burned by a fire (the area 75m around the point is considered burned). All shapefiles have exactly the same name but are saved in different folders. Each shapefile corresponds to a single fire, but I have to group the fires before extracting the values from the raster, that's why they are in for loops. – Vero Oct 23 '14 at 8:15
  • 2
    The script stops with an error? Or the script hangs? If the former, what's the error? – ForeverWintr Jan 19 '16 at 22:43