I am quite new to scripting for ArcGIS, so don't mind the styling and layout.
I am trying to calculate 600.000 routes by looping a network analyst route analysis. The script extracts two pointID's from a database, selects their spatial component from a feature dataset based on this pointID, calculates the route and writes it to a layer file by using an append.
When I run the file (in ArcGIS Python Window), I get an out of memory error after 400+ files. When I examine the memory-use of the process, I can see a definite build-up until 2.5GB usage until the error.
This is my code:
# step 1: centroids geographical area arcpy.AddMessage("Performing Feature to Point") arcpy.FeatureToPoint_management(geographicalArea, outGeographicalAreaCentroids) arcpy.MakeFeatureLayer_management(outGeographicalAreaCentroids, outGeographicalAreaCentroidsLyr) # step 2: create new empty feature class to append routes into arcpy.CreateFeatureclass_management(outLocation, routesFC, "POLYLINE", templateRoute) # step 3: go through table and perform routing for each row cursor = arcpy.SearchCursor(dbf) for row in cursor: outRouteLayer = arcpy.na.MakeRouteLayer(networkDataset, outRouteLayerName, "Minutes", hierarchy = "USE_HIERARCHY") outRouteLayer = outRouteLayer.getOutput(0) subLayerNames = arcpy.na.GetNAClassNames(outRouteLayer) stopsLayerName = subLayerNames["Stops"] herkomst = row.getValue("Herkomstzo") expression1 = "\"Z\" = %d" % herkomst arcpy.SelectLayerByAttribute_management(outGeographicalAreaCentroidsLyr, "NEW_SELECTION", expression1) bestemming = row.getValue("Bestemming") expression2 = "\"Z\" = %d" % bestemming arcpy.SelectLayerByAttribute_management(outGeographicalAreaCentroidsLyr, "ADD_TO_SELECTION", expression2) arcpy.CopyFeatures_management(outGeographicalAreaCentroidsLyr, outGeographicalAreaCentroidsSelection) arcpy.MakeFeatureLayer_management(outGeographicalAreaCentroidsSelection, outGeographicalAreaCentroidsSelectionLyr) arcpy.na.AddLocations(outRouteLayer, stopsLayerName, outGeographicalAreaCentroidsSelectionLyr) if herkomst == bestemming: arcpy.AddMessage("Origin zone is equal to the destination zone") else: arcpy.na.Solve(outRouteLayer) outputRoute = "C:\Users\Koos\Documents\Werk\Onderzoek\Data\WORKSPACE\Routes\outRouteRoutes" arcpy.MakeFeatureLayer_management(r"route\Routes", outputRoute) arcpy.AddField_management(outputRoute, "Orig", "DOUBLE") arcpy.CalculateField_management(outputRoute, "Orig", herkomst) arcpy.AddField_management(outputRoute, "Dest", "DOUBLE") arcpy.CalculateField_management(outputRoute, "Dest", bestemming) arcpy.Append_management(outputRoute, routesFCLocation, "NO_TEST") # delete layers from memory arcpy.Delete_management(outRouteLayer) arcpy.Delete_management(herkomst) arcpy.Delete_management(expression1) arcpy.Delete_management(bestemming) arcpy.Delete_management(expression2) arcpy.Delete_management(outGeographicalAreaCentroidsSelection) arcpy.Delete_management(outGeographicalAreaCentroidsSelectionLyr) arcpy.Delete_management(outputRoute) del row del cursor
There are more steps, but they are not relevant.
I read a lot of topics about the out of memory problems and memory leaks, etc. But I can't seem to solve this problem.
First I tried to remove every file made in the proces with the arcpy.Delete_management function. This seems to drop the memory use a little bit every step, but not enough, so the memory usage still builds up. Next, I tried to run my script in PythonWin and IDLE (thinking the memory issue would maybe not occur), but then I got errors saying the spatial network analyst (solve tool) is not licensed (a whole other issue).
Are there any suggestions as to how I can solve this issue and calculate the routes? I read on one topic that it is possible to extract the code in the loop to a seperate script and then summon this script in the loop, would this be an option? And if yes, how is this done?
Edits: For a similar 'out of memory'-problem when calculating service areas, chunking the calculation has proved a valuable solution (the memory resets after every chunk). However, this still not resolves the routing issue, since chunking the route calculation will not help. Is there a possibility to partition the loop, thus resetting the memory after every 'loop-chunk'?
Code used to check license:
class LicenseError(Exception): pass try: if arcpy.CheckExtension("Spatial") == "Available": arcpy.CheckOutExtension("Spatial") print "Checked out \"Spatial\" Extension" else: raise LicenseError except LicenseError: print "Spatial license is unavailable" except: print arcpy.GetMessages(2)
Code used to determine input variables (example):
inputMap = "C://Users//Koos//Documents//Werk//Data script//Inputs" facility = inputMap + "//Totale_opvang_Vlaanderen.shp" geographicalArea = inputMap + "//BASISZONERING_OVL.shp" networkDataset = inputMap + "//belbel________nw_ND.nd"