Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I've run OD cost matrix analysis using GUI tools n ArcGIS 10.

My goal is relatively simple - find closest 50 neighbors for each origin point.

The dataset I'm working with is relatively large:

  • network dataset consists of ~ 1.16 million elements
  • there are ~ 100k origins
  • and ~ 1.3 million destinations.

Origins are part of the destinations dataset.

The limits I set for the geoprocessing tool are: default_number_destinations_to_find set to 51 (since first 'neighbor' will be the same, overlapping point) and default_cutoff = 50000 (since I know from previous analyses that this value cannot be larger than that).

In order to save memory (I think) I go with output_path_shape set to NO_LINES - I do not need graphical representation of OD matrix - just tabular data (correct me if I'm wrong here).

It takes a while to process this task (~ 24h), but nonetheless I get the results. So far so good.

In order to automate this task I tried putting all this steps into python script and came up with:

try:

    import arcpy
    from arcpy import env

    # Check out any necessary licenses
    arcpy.CheckOutExtension("Network")

    # set environment settings
    env.workspace = "D:\\GIS\\04-buildings.gdb"
    env.overwriteOutput = True

    # Local variables:
    NW_FIX_ND = "D:\\GIS\\02-network-restrict.gdb\\NW_FIX\\NW_FIX_ND"
    DESTINATIONS = "buildings\\destinations"
    ORIGINS = "buildings\\origins_1"
    OD_COST_MATRIX = "OD Cost Matrix"
    LINES = "OD Cost Matrix\\Lines"
    bn_1 = "results\\bn_1"

    # Make OD Cost Matrix Layer
    arcpy.MakeODCostMatrixLayer_na(NW_FIX_ND, "OD Cost Matrix", "Length", "50000", "51", "Length", "ALLOW_UTURNS", "restriction", "NO_HIERARCHY", "", "NO_LINES")

    # Add Destinations
    arcpy.AddLocations_na(OD_COST_MATRIX, "Destinations", DESTINATIONS, "Name h_gebaeude_id #;CurbApproach # 0", "5000 Meters", "", "TLM_STRASSE_RESTR_FIX SHAPE;NW_FIX_ND_Junctions NONE", "MATCH_TO_CLOSEST", "APPEND", "NO_SNAP", "5 Meters", "EXCLUDE", "TLM_STRASSE_RESTR_FIX #;NW_FIX_ND_Junctions #")

    # Add Origins
    arcpy.AddLocations_na(OD_COST_MATRIX, "Origins", ORIGINS, "Name h_gebaeude_id #;TargetDestinationCount # 51;CurbApproach # 0;Cutoff_Length # #", "5000 Meters", "", "TLM_STRASSE_RESTR_FIX SHAPE;NW_FIX_ND_Junctions NONE", "MATCH_TO_CLOSEST", "CLEAR", "NO_SNAP", "5 Meters", "EXCLUDE", "TLM_STRASSE_RESTR_FIX #;NW_FIX_ND_Junctions #")

    # Solve
    arcpy.Solve_na(OD_COST_MATRIX, "SKIP", "TERMINATE")

    # Select OD Lines
    arcpy.SelectData_management(OD_COST_MATRIX, "Lines")

    # Select
    arcpy.Select_analysis(LINES, bn_1, "")

    print "Completed successfully"

except Exception as e:

    # If an error occurred, print line number and error message
    import traceback, sys
    tb = sys.exc_info()[2]
    print "An error occured on line %i" % tb.tb_lineno
    print str(e)

However, when I attempt to run this script I get the following error:

An error occured on line 39
ERROR 030024: Solve returned a failure.
Out of memory.
Failed to execute (Solve).

My questions here - why does arcpy fail when GUI could do the job? My reasoning was that by running the task in python and avoiding the graphical overhead - it should even work faster? Any ideas on what I could do to make arcpy handle this problem?

share|improve this question
1  
I feel your pain, I too have hit the out of memory wall when attempting to process large vector datasets with arcpy. Things like dissolving, symmetrical difference, intersect - it just cannot handle them. I'm considering attempting my processing with Python and shapely, a colleague is going to try it in GRASS. –  Chad Cooper Mar 7 '12 at 14:30
add comment

1 Answer

I found the answer is due to python.exe having a limit of about 2GB. A solution is here:

http://gisgeek.blogspot.com.au/2012/01/set-32bit-executable-largeaddressaware.html

share|improve this answer
    
Thanks for a pointer. Will check this solution out on my GIS desktop asap! –  radek Mar 14 at 14:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.