5

There are concerns about the ability of ArcGIS 10 to fulfill a requisite of mine.

ET Geowizards has been tried, although it doesn't have the same capability of ArcGIS e.g. I cannot aggregate all of the points I have with ET, to the scale I have them plotted.

There is a memory leak, which means looping through 700 objects, performing:

  1. Agg Points.
  2. Buffer.
  3. Add Field.
  4. Update cursor.

Starts off taking 5-9 secs per object, and continues on to 2 mins per (similarly sized) object.

In SP2, it appears AggPoints no longer works to create an FC on the fly. There is more, but it's too long a list to compile!

Code, simplied with no buffer, add field or cursor. geom is a collection or arcpy points

def createGeom(geom, scratchDB):
    filetime = (str(time.time())).split(".")
    outfile = "fc" + filetime[0]+filetime[1]
    outpath = scratchDB + "tmpV.gdb/Polygon/"  
    outFeatureAggClass = outpath+outfile +"_Agg"
    arcpy.AggregatePoints_cartography(geom, outFeatureAggClass,"124000 meters")

Seems to be that the size of the file geodatabase we're writing to is one of the main issues; I think the problem is that performance degrades as the local file geodatabase fills up - significantly.

Any ideas how to tune local file geodatabase?

Improve this question
14
  • can you be more specific or give some examples highlighting your concerns with ArcGIS 10?
    – artwork21
    Aug 1, 2011 at 13:00
  • Comment added. I would love to do a concave hull, but the only code kicking around isn't suitable for the grid of points I am using, as it's simply too large. I am using world grids of around 60km each.
    – Hairy
    Aug 1, 2011 at 13:47
  • downvote because the Q is too broad. I think people are upvoting on the title alone. It's a theme many are interested in (myself included!), but it doesn't make a good question.
    – matt wilkie
    Aug 2, 2011 at 16:50
  • 1
    In terms of cursors, I have found ArcGIS to be a nightmare when working in Python. C# has been much more amenable to the task of large cursor processes, though.
    – Nathanus
    Aug 2, 2011 at 17:55
  • The cursor is small, and I am using it to update one object. I have also removed the cursor to show it isn't that, which it isn't. Calling arcpy causes the leaks.
    – Hairy
    Aug 3, 2011 at 6:58

1 Answer 1

Reset to default
3

If you show the simplest possible form of the code, it might benefit from using a dictionary instead of cursor, or in-memory workspace, or a change the workflow (for example aggregate & buffer then cursor instead of cursor then aggregrate & buffer [ref]), or... In any case, start here: Performance of ArcGISScripting and large spatial data sets

Improve this answer
9
  • Matt, I though I had made it clear the workflow? The cursor only ever updates one record, the record it has created with the agg points and buffer; I cannot update the data, without first adding the fields, and creating the dataset, which I am doing with the agg points, and buffer. I don't find this sueful at all matt, to be honest.
    – Hairy
    Aug 3, 2011 at 6:56
  • Matt's suggestions are sound. When you say there is a memory leak, how much memory is it actually using? If you are using an in-memory workspace, take care to either overwrite or delete anything you create there when it is no longer needed so that you don't fill up your RAM and start hitting the pagefile instead. We would need to see how you are calling your createGeom in a loop to say whether/how that could be improved.
    – blah238
    Aug 3, 2011 at 7:55
  • Also are you hitting an SDE feature class in this script? I ran into an actual memory leak in a very specific situation involving a direct connection to an SDE feature class which I described here. It's possible, though unlikely, that the same leak is happening here.
    – blah238
    Aug 3, 2011 at 8:01
  • I think you can see no in-memory workspace is being used. It's using a set of points, to create a featureClass. They are all being written to a local FGDB. I said Matt wasn't being useful, as I had described the workflow clearly, before he wrote his post. He also got the order wrong using the cursor.
    – Hairy
    Aug 3, 2011 at 12:27
  • @Hairy apologies for lack of usefulness. It appears there's enough of that to flow in both directions, as what was described as a clear workflow for you was anything but for me. ;-) In any case, on with the troubleshooting! (in additional comments)
    – matt wilkie
    Aug 3, 2011 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.