Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

When I'm trying to dissolve a polygon through an arcpy script the operation fails with the 999999 error on computers with small amount of RAM [~3 GB] (known issue).

Unfortunately neither this, that nor that work for me, so I was thinking about dividing the polygon into smaller pieces first, processing them and then merging back? But I have absolutely no idea how to do that. Any ideas? Maybe there's another approach that hasn't come to my mind?

An example polygon for testing can be found here, I'm dissolving on the "Godziny" field with the default settings:

arcpy.Dissolve_management(path_to_original_shp, path_to_dissolved_shp, "Godziny")

ADDED: Unfortunately when I tested Geog's solution and it worked it was just a fluke. Still 99 out of 100 dissolves fail. However the conditions of failing are quite weird. If I run dissolving as a tool in ArcMap it dissolves just fine (in about 30 seconds). If I run it from python's interpreter (doesn't matter whether in ArcMap or not) it also dissolves fine. However If I run it from my script as one of the functions in a processing chain it fails (it's stuck for about 20-30 min with memory usage growing slowly but constantly over time and then gives 999999 error - "Invalid topology").

The way I run this script is a bit unusual, because I have a GUI built with wxpython that I run in ArcMap, from which the processing chain with dissolving is run from. I run it on a Windows XP SP3 machine with 3 GB of RAM. If I use a machine with more RAM and Windows 7 or 8 (64 bit but without 64 bit background geoprocessing) everything works just fine. In addition to that, if I comment out dissolving and the rest of the chain, run it, then comment out the first part and run it from the dissolve point it also works fine. That gave me an idea that I might be somehow locking the resources, so I've tried doing gc.collect(), but with no success.

I've just tried running the script from the command line and it also works fine... So I guess there's something wrong when dissolving from my extension GUI, however I have no idea how my extension could influence the function as I'm not modifying the environment in any way.

The processing chain I'm doing looks like this:

    arcpy.CheckOutExtension("spatial")

    # getting necessary licences
    ccm1_cut = ExtractByMask(project_manager.conf_dict["CCM1"], project_manager.conf_dict["hordist"])
    ccm2_cut = ExtractByMask(project_manager.conf_dict["CCM2"], project_manager.conf_dict["hordist"])
    ccm3_cut = ExtractByMask(project_manager.conf_dict["CCM3"], project_manager.conf_dict["hordist"])
    ccm4_cut = ExtractByMask(project_manager.conf_dict["CCM4"], project_manager.conf_dict["hordist"])
    ccm5_cut = ExtractByMask(project_manager.conf_dict["CCM5"], project_manager.conf_dict["hordist"])

    # merging ccm layers
    ccm_merged = Con(IsNull(ccm1_cut), Con(IsNull(ccm2_cut), Con(IsNull(ccm3_cut), Con(IsNull(ccm4_cut),
                    Con(IsNull(ccm5_cut), project_manager.conf_dict["lu_speed"], ccm5_cut), ccm4_cut), ccm3_cut), ccm2_cut), ccm1_cut)

    walk_speed_cut = ExtractByMask(project_manager.conf_dict["walkspeed"], project_manager.conf_dict["hordist"])

    travel_speed_raster = walk_speed_cut / ((Float(ccm_merged)) / 100)
    travel_speed_raster.save(travel_speed)

    # creating path distance
    ipp = project_manager.get_ipp_path()
    path_dist_raster = PathDistance(ipp, travel_speed)

    # normalising to hours
    path_dist_norm_raster = Divide(path_dist_raster, 3600)
    path_dist_norm_raster.save(travel_time_normalised)

    # creating hour classes
    remap_range = RemapRange(create_remap_range_ccm())
    temp_reclass = Reclassify(travel_time_normalised, "VALUE", remap_range, "NODATA")
    temp_reclass.save(travel_time_classes)

    # converting travel time raster to polygons
    arcpy.RasterToPolygon_conversion(travel_time_classes, travel_time_classes_shp, "SIMPLIFY", "VALUE")

    # calculating time value
    arcpy.AddField_management(travel_time_classes_shp, "Godziny", "SHORT")
    arcpy.CalculateField_management(travel_time_classes_shp, "Godziny", "[GRIDCODE]", "VB")

    # dissolving polygons
    arcpy.Dissolve_management(travel_time_classes_shp, travel_time_classes_dissolved, "Godziny")
    ...
share|improve this question
    
I think you are going to find this a frustrating experience which you could resolve very easily by just making more RAM a requirement of your application. Everything you describe seems to fit with that being your limiting factor, and the cost of more RAM versus development time costs to try and work within 3Gb RAM seems to be a false economization. –  PolyGeo yesterday
    
@PolyGeo It seems that this is the case and working on a workaround at this point is just pointless. –  Michal yesterday
    
add comment

5 Answers 5

The Dissolve tool can create a Godzilla by combining smaller (but still fairly large) features into one feature. This is known as the combinatorial problem. The Dissolve tool has logic that prevents it from creating a Godzilla (you’ll receive the warning code 000059) but this logic is based on the machine’s available memory at the time Dissolve is run. So, while the output may not be a Godzilla on the machine where Dissolve ran, it may be on another machine with less available memory. learn more about Dicing Godzillas (The Dice tool takes input features and a vertex limit and outputs a new feature class with diced features, as illustrated below. The Dice tool works with multipoints, lines, and polygons.)

share|improve this answer
    
Didn't know about the Godzillas. That was exactly what I was looking for. Thank you very much. –  Michal Jul 6 at 15:29
    
Dicing worked for me at first, but it was pure luck, because it usually fails about 99/100 times. However thanks to you, I've found out about Godzillas, which I'm thankful for. –  Michal yesterday
    
... and that's why I'm going to award you with the bounty. I may have not solved the problem, but at least I've understood it. –  Michal yesterday
add comment

I would create a 4 region polygon and 'clip' out each of the four portions into separate shapefiles then run the 'dissolve'. Then 'merge' them back together and you would likely have to run one final 'dissolve'.

I would create the 4 polygons via the fishnet tool based on the envelope of the input set of polygons.

here is a link. http://forums.arcgis.com/threads/40500-Split-polygon

share|improve this answer
    
Dear user, you're very kind, however I'm looking for a solution to my problem rather than a processed layer. The one I'm giving link to is just an example polygon (I've modified the question so it won't mislead other users). The thing with 'clip' is, I would need to clip the polygon with some other polygon. Meanwhile the polygon to dissolve will have a different location each time, so I can't create a generic mask for clipping it. Or am I missing something? –  Michal Jul 6 at 13:32
    
amended answer to split based on different locations. –  user two seven two three nine Jul 6 at 13:35
    
Unfortunately I can't find a constant and reliable number of regions and after exceeding a certain threshold the execution time is too long. –  Michal yesterday
add comment

Sometines it helps to work with a layer instead of directly on the data.You should try to creatre such a layer before you run the tools (with arcpy.MakeFeatureLayer_management), then you delete this layer (with arcpy.Delete_management, deleting the layer, not the feature class, of course)

It can also help to avoid multipart features using the "SINGLE_PART" option and to reduce the number of vertices using "simplify (you did simplify at the stage of the raster to polygon conversion.

Finally, there is a work around to dissolve if you have the advanced licence, but it takes time. Use polygon to line with the option identify the left and right polygons, join the original fields that you need for the dissolve, select all the lines that have a different value left and right, the convert those lines to polygons: You have dissolved your dataset.

share|improve this answer
    
Unbelievable, simply unbelievable, after creating a layer dissolving worked perfectly... Do you by chance have any idea why it helped? –  Michal Jul 16 at 7:50
    
Nevermind the problem persists... I will try your other suggestions –  Michal Jul 16 at 8:55
    
can you please be more specific with what worked and didn't work with the layers. –  radouxju Jul 17 at 7:13
    
Making a feature layer works if I have only ArcMap running and a fresh windows session (a lot of RAM is available), which unfortunately is not acceptable. Simplify with satisfying precision doesn't help at all, if I simplify more it helps but the outcome is not acceptable. The last solution works perfectly, however it takes too much time (my code will be use in search and rescue operations), it needs more than an hour (dissolve on PCs with a lot of RAM takes up to a minute). Unfortunately described circumstances makes your proposals unusable for me. –  Michal yesterday
add comment

I had this problem.

Here's what arcmap does in background and you should do the same.

  1. 10000 features slices: select top 10000 features ,make a layer, dissolve
  2. Clean-up : always , but always use "arcpy generated names" ( there is a function, don't remember it ), do not reuse names (layers or what ever) and delete unused/reused layers at the end of each iteration cycle.
  3. Change the workspace and the default working gdb.

" I have a GUI built with wxpython " -> get rid of the gui in wx. There are know problems for this combination. arcpy module uses some COM/OLE objects somewhere. " From version 0.5 Wx::ActiveX is compatible with Win32::OLE objects " -> are you using the 64bit version ? Are you using Arcgis 10.1 ?!

Cheers! Hope this helps you understand the core problem.

share|improve this answer
    
I would need to go really low with the selected number of features (as low as 100 region) and with such low selection it makes the process last too long, which is not acceptable for me. –  Michal yesterday
add comment

My guess would be it is a garbage collection problem - one of the classes isn't going out of scope long enough for collection to occur before it is re-initialised. Managed frameworks don't always get round to it when their resources are at maximum usage and the fact it only occurs during the script version and then somewhat randomly (which might be down to other system operations) suggests this cause to me.

As for solutions, its hard to change if the code concerned is wrapped up in a proprietary class. If you can force GC somehow (wrap it all in another class perhaps an put that out of scope?) that might work.

Or perhaps create rasters for the parameters in the calculatefield_management class and use a raster calc instead, then convert to polys at the end, thus avoiding dissolve all together?

share|improve this answer
    
I tried making GC to collect garbage, however that didn't work at all. The second solution seems reasonable, however I've lost just too many time at this point to find a workaround and I'm just going to give up and try to work on an undissolved layer. –  Michal yesterday
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.