Dissolving polygons fails on computers with small amount of RAM

Question

When I'm trying to dissolve a polygon through an arcpy script the operation fails with the 999999 error on computers with small amount of RAM [~3 GB] (known issue).

Unfortunately neither:

• arcpy.Dissolve fails; need a arcpy dissolve to multipart workaround
• Why is the Dissolve Tool not working? nor
• Dissolve not working in ArcGIS 10 work for me, so I was thinking about dividing the polygon into smaller pieces first, processing them and then merging back? But I have absolutely no idea how to do that. Any ideas? Maybe there's another approach that hasn't come to my mind?

An example polygon for testing can be found here, I'm dissolving on the "Godziny" field with the default settings:

arcpy.Dissolve_management(path_to_original_shp, path_to_dissolved_shp, "Godziny")

ADDED: Unfortunately when I tested Geog's solution and it worked it was just a fluke. Still 99 out of 100 dissolves fail. However the conditions of failing are quite weird. If I run dissolving as a tool in ArcMap it dissolves just fine (in about 30 seconds). If I run it from python's interpreter (doesn't matter whether in ArcMap or not) it also dissolves fine. However If I run it from my script as one of the functions in a processing chain it fails (it's stuck for about 20-30 min with memory usage growing slowly but constantly over time and then gives 999999 error - "Invalid topology").

The way I run this script is a bit unusual, because I have a GUI built with wxpython that I run in ArcMap, from which the processing chain with dissolving is run from. I run it on a Windows XP SP3 machine with 3 GB of RAM. If I use a machine with more RAM and Windows 7 or 8 (64 bit but without 64 bit background geoprocessing) everything works just fine. In addition to that, if I comment out dissolving and the rest of the chain, run it, then comment out the first part and run it from the dissolve point it also works fine. That gave me an idea that I might be somehow locking the resources, so I've tried doing gc.collect(), but with no success.

I've just tried running the script from the command line and it also works fine... So I guess there's something wrong when dissolving from my extension GUI, however I have no idea how my extension could influence the function as I'm not modifying the environment in any way.

The processing chain I'm doing looks like this:

    arcpy.CheckOutExtension("spatial")

    # getting necessary licences
    ccm1_cut = ExtractByMask(project_manager.conf_dict["CCM1"], project_manager.conf_dict["hordist"])
    ccm2_cut = ExtractByMask(project_manager.conf_dict["CCM2"], project_manager.conf_dict["hordist"])
    ccm3_cut = ExtractByMask(project_manager.conf_dict["CCM3"], project_manager.conf_dict["hordist"])
    ccm4_cut = ExtractByMask(project_manager.conf_dict["CCM4"], project_manager.conf_dict["hordist"])
    ccm5_cut = ExtractByMask(project_manager.conf_dict["CCM5"], project_manager.conf_dict["hordist"])

    # merging ccm layers
    ccm_merged = Con(IsNull(ccm1_cut), Con(IsNull(ccm2_cut), Con(IsNull(ccm3_cut), Con(IsNull(ccm4_cut),
                    Con(IsNull(ccm5_cut), project_manager.conf_dict["lu_speed"], ccm5_cut), ccm4_cut), ccm3_cut), ccm2_cut), ccm1_cut)

    walk_speed_cut = ExtractByMask(project_manager.conf_dict["walkspeed"], project_manager.conf_dict["hordist"])

    travel_speed_raster = walk_speed_cut / ((Float(ccm_merged)) / 100)
    travel_speed_raster.save(travel_speed)

    # creating path distance
    ipp = project_manager.get_ipp_path()
    path_dist_raster = PathDistance(ipp, travel_speed)

    # normalising to hours
    path_dist_norm_raster = Divide(path_dist_raster, 3600)
    path_dist_norm_raster.save(travel_time_normalised)

    # creating hour classes
    remap_range = RemapRange(create_remap_range_ccm())
    temp_reclass = Reclassify(travel_time_normalised, "VALUE", remap_range, "NODATA")
    temp_reclass.save(travel_time_classes)

    # converting travel time raster to polygons
    arcpy.RasterToPolygon_conversion(travel_time_classes, travel_time_classes_shp, "SIMPLIFY", "VALUE")

    # calculating time value
    arcpy.AddField_management(travel_time_classes_shp, "Godziny", "SHORT")
    arcpy.CalculateField_management(travel_time_classes_shp, "Godziny", "[GRIDCODE]", "VB")

    # dissolving polygons
    arcpy.Dissolve_management(travel_time_classes_shp, travel_time_classes_dissolved, "Godziny")
    ...

Answer 1

The Dissolve tool can create a Godzilla by combining smaller (but still fairly large) features into one feature. This is known as the combinatorial problem. The Dissolve tool has logic that prevents it from creating a Godzilla (you’ll receive the warning code 000059) but this logic is based on the machine’s available memory at the time Dissolve is run. So, while the output may not be a Godzilla on the machine where Dissolve ran, it may be on another machine with less available memory. learn more about Dicing Godzillas (The Dice tool takes input features and a vertex limit and outputs a new feature class with diced features, as illustrated below. The Dice tool works with multipoints, lines, and polygons.)

Answer 2

I would create a 4 region polygon and 'clip' out each of the four portions into separate shapefiles then run the 'dissolve'. Then 'merge' them back together and you would likely have to run one final 'dissolve'.

I would create the 4 polygons via the fishnet tool based on the envelope of the input set of polygons.

here is a link. http://forums.arcgis.com/threads/40500-Split-polygon

Answer 3

ArcGIS Desktop 10.2 appears to have some geoprocessing memory-management issues. Repeatedly running geoprocessing operations in a single Python process will eventually crash with the spurious 'Invalid Topology' error. How long this takes depends on system resources and the complexity of the operations being run.

One solution that works, taken from here, is to run the geoprocessing operations as their own process. This example is for dissolve, but erase and clip can be done in the same way.

import subprocess


def call_dissolve_in_separate_process(input_fc, new_fc_loc):
    args = (input_fc, new_fc_loc)
    cmd = r"C:\Python27\ArcGIS10.2\python.exe C:\projects\my_project\standalone_dissolve.py " + " ".join(args)
    chld = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    r = chld.communicate()
    if chld.returncode != 0:
        print str(chld.returncode)
        # Here you can raise some kind of error, or recursively try again.
    else:
        print '\nDissolve and saving took\n%s seconds\n' % (time.time()-start_time)
        return new_fc_loc

In a separate module called standalone_dissolve.py

"""Dissolve in a separate subprocess."""
import sys
from arcpy import Dissolve_management


args = sys.argv
script, input_fc, new_fc_loc = args

def main(input_fc, new_fc_loc):
    try:
        print 'Trying dissolve'
        Dissolve_management(input_fc, new_fc_loc)
        print 'Dissolve worked'
        return 0
    except Exception as ex:
        print ex.message
        print 'Dissolve failed, trying again!'  # There are other things wrong with geoprocessing, this recursive approach is from the 'if at first you don't succeed, try it again until successful school of arcpy scripting thought'. Your needs may differ.
        main(input_fc, new_fc_loc)
    return 1  # Someone suggest a more elegant way

main(input_fc, new_fc_loc)

This is messy, but you get complete resources cleanup after each geoprocessing operation completes.

Answer 4

Sometines it helps to work with a layer instead of directly on the data.You should try to creatre such a layer before you run the tools (with arcpy.MakeFeatureLayer_management), then you delete this layer (with arcpy.Delete_management, deleting the layer, not the feature class, of course)

It can also help to avoid multipart features using the "SINGLE_PART" option and to reduce the number of vertices using "simplify (you did simplify at the stage of the raster to polygon conversion.

Finally, there is a work around to dissolve if you have the advanced licence, but it takes time. Use polygon to line with the option identify the left and right polygons, join the original fields that you need for the dissolve, select all the lines that have a different value left and right, the convert those lines to polygons: You have dissolved your dataset.

Answer 5

I had this problem.

Here's what arcmap does in background and you should do the same.

1. 10000 features slices: select top 10000 features ,make a layer, dissolve
2. Clean-up : always , but always use "arcpy generated names" ( there is a function, don't remember it ), do not reuse names (layers or what ever) and delete unused/reused layers at the end of each iteration cycle.
3. Change the workspace and the default working gdb.

" I have a GUI built with wxpython " -> get rid of the gui in wx. There are know problems for this combination. arcpy module uses some COM/OLE objects somewhere. " From version 0.5 Wx::ActiveX is compatible with Win32::OLE objects " -> are you using the 64bit version ? Are you using Arcgis 10.1 ?!

Cheers! Hope this helps you understand the core problem.

Answer 6

My guess would be it is a garbage collection problem - one of the classes isn't going out of scope long enough for collection to occur before it is re-initialised. Managed frameworks don't always get round to it when their resources are at maximum usage and the fact it only occurs during the script version and then somewhat randomly (which might be down to other system operations) suggests this cause to me.

As for solutions, its hard to change if the code concerned is wrapped up in a proprietary class. If you can force GC somehow (wrap it all in another class perhaps an put that out of scope?) that might work.

Or perhaps create rasters for the parameters in the calculatefield_management class and use a raster calc instead, then convert to polys at the end, thus avoiding dissolve all together?