6

I'm having a memory leak issue. Upon investigating this more, it seems the file based geodatabase I write to in a loop, is growing very large, and as it grows, it significantly degrades the performance of the scripts I am running.

Any ideas how to optimize the configuration of the fgdb? Or how to speed it all up? I am not writing 'in_memory', I am using aggregatePoints to create a temp featureclass (which I delete), and I buffer this FC, which I keep.

However, it seems to get slower, and slower, and slower...

def createGeom1(geom, scratchDB):
    filetime = (str(time.time())).split(".")
    outfile = "fc" + filetime[0] + filetime[1]
    outpath = scratchDB + "tmp1.gdb/Polygon/"
    outFeatureAggClass = outpath+outfile + "_Agg"
    arcpy.AggregatePoints_cartography(geom, outFeatureAggClass, "124000 meters")

geom is a collection of Points, scratch the scratch area (local gdb I am using).

Just looping through a list of files, I call a procedure that creates alist of geoms (and doesn't degrade) and then call this. Doing this, will see this function, createGeom, degrade significantly, and the previous one, not a bit.

9
  • Can't you flush (to the FC) at regular intervals, instead of doing it one time at last? Just a thought...
    – ujjwalesri
    Aug 9, 2011 at 7:53
  • @ujjwalesri - Sorry, I don't get what you mean?
    – Hairy
    Aug 9, 2011 at 8:32
  • Local or network disk? Aug 9, 2011 at 12:08
  • Local disk, keep well away from the network...
    – Hairy
    Aug 9, 2011 at 13:24
  • Post your machine specs including hard drive model/speed/size, RAM speed/size/timings, graphics card model/memory and processor.
    – MLowry
    Aug 9, 2011 at 13:59

4 Answers 4

4

Depending on how big your file geodatabase, you might be able to use a RAM disk (e.g. ImDisk: http://www.ltr-data.se/opencode.html/#ImDisk ). It might speed things up if your hard disks are slower.

Here's a question (mine) about RAM Disks: Does using RAM Disk improve ArcGIS Desktop performance appreciably?

The RAM Disk option is low-hanging fruit. And unfortunately, I can't link to whuber's: comment on that question, but I think it highlights where the most performance gains can be made (so maybe you should post your code in your question).

whuber: Even a high-end (ordinary) disk drive can make an appreciable difference. However, by far the most dramatic improvement in long-running GIS processes is made by improving the algorithm. In many, many cases, if your computation is taking noticeably longer than the time required to read all inputs and write all outputs, chances are you are using an inefficient algorithm.

5
  • 1
    Yeah, I have been through the code a few times and even had it code reviewed by ESRI, who said it couldn't, really, be improved. I think a hard ware review is out of the questions, but I might push for a solid state drive lol. I saw those posts before, but thanks for adding them here for completeness. Thanks for the help.
    – Hairy
    Aug 9, 2011 at 11:46
  • 1
    yes, if the inefficient algorithms are in Esri's geoprocessor, it doesn't matter how perfect your python code is. Aug 9, 2011 at 11:59
  • it's actually a fair point! I put a test harness together for ESRI, which simply looped through a list of text files containing a collection of points, to which it went to one procedure to aggregate the points. One call to arcpy caused the memory to get all beat up.
    – Hairy
    Aug 9, 2011 at 12:15
  • When the process starts, it uses very limited memory and performs incredibly quickly ~2 seconds a polygon. However, this slowly creeps up as data is added to the fgdb until it's almost 30 seconds a polygon for more or less the same size unit.
    – Hairy
    Aug 9, 2011 at 13:39
  • Solid state drives are faster but the arcgis software is the bottleneck in performance (re-write required). If no need write to the geodatabase then compressed (rather than compacting) data is a direct-access format so faster. webhelp.esri.com/arcgisserver/9.3/java/index.htm#geodatabases/…
    – Mapperz
    Aug 9, 2011 at 14:36
2

Check out item #4 of my answer for this related question Performance of ArcGISScripting and large spatial data sets I bet you are loading inside an EditSession and that you are not using recycling cursors :)

Update based on comments:

Although ArcPy and ArcObjects are syntactically different, semantically it is the same.  You asked if performance of FileGDB degrades as it fills up. The answer is no it does not. What degrades is when cursors you are using whether explicit (i.e by declaring them yourself) or implicitly (by using a canned arcpy function that creates them) are not released properly. There is a price for using the canned functions - a trade off of simplicitly for resource control (you barely have any). This is not inherent to FileGDB, but to ArcGIS architecture as a whole. The moment you choose to use ArcPy, you are making the statement "I'd rather trade-off the complexity of using a lower level API for performance". The higher you go, the more you trade. ArcPy is pretty much at the top of it all... one function and you are done! Nevertheless, that thread still went to the ArcObjects initialization process, and the associated workspace factory allocation, and the associated file handle open for every scratch workspace (a full gdb!) and until the internal memory freeing routines kicks in (in .net you can explicitly call the gargabe collector and in C++ is inmediate, not sure about the Python-Com memory model honestly) you will have these gdbs and resources staying resident. So you probably have several dbs, and fc handles in memory every time you loop. That is the price of ArcPy and it is up to you to decide whether it is worth it.

14
  • That's for ArcObjects, not ArcPY Ragi
    – Hairy
    Aug 10, 2011 at 6:52
  • ArcPy are just python wrappers on top of ArcObjects, so the same concepts apply, except that you have less explicit control of triggering the right behaviors. Aug 10, 2011 at 8:41
  • I know they are, which introces a lot of issues. However, the use of cursors is 'syntactically' different, so I don't have the same options, it also isn't the issue, as I get this by simply calling the AggPoints function inside the call, without a cursor.
    – Hairy
    Aug 10, 2011 at 8:57
  • Updated my answer to include comments to your comments :) Aug 10, 2011 at 15:28
2

I know an answer has been accepted already, but I thought I'd add my experience and some code I use to get around it. I have a project that generates a large volume of intermediate data. The size is small, but the number of features/rasters is significant. After having my program benchmark where the performance loss was occurring, it was clear that it was happening on the FGDB writes. At the start of the program, an FGDB write would take ~2 seconds. After adding about 100-150 features, it would take about 6 seconds, and increase approximately linearly from there.

I solved this by creating a simple gdb class that tracks the number of features and creates a new gdb for that purpose if we hit the threshold. NOTE that it requires a bit of adaptation because it takes a variable name in another module as a parameter and sets that variable's value to the path to the gdb. In this case the module is "config" as seen in the method switch() below. If you correct that to your own module (or just switch it to set a name in the current module), this can be adapted. It also generates names by splitting off the FGDB name from the input variable, so you'll need to seed that first. Otherwise, you can adjust the code in init to behave differently.

db_size_threshold = 150

class gdb:
    def __init__(self,tconfig=None):
        self.name = None
        self.config_var = tconfig
        self.parent_folder = None
        self.db_count = 1
        self.features_count = 0

        name_parts = os.path.split(getattr(config,self.config_var))
        self.name = os.path.splitext(name_parts[1])[0]
        self.parent_folder = name_parts[0]

        self.switch() # make a new db to start

    def check(self):
        '''checks if we've hit the threshold - if not, increment its count'''

        if self.features_count > db_size_threshold:
            self.switch()
        else:
            self.features_count += 1

    def switch(self):
        '''creates a new gdb for this object''' 

        new_name = self.next_db()

        arcpy.CreateFileGDB_management(self.parent_folder,"%s_%s.gdb" % (self.name,self.db_count))
        self.features_count = 0

        setattr(config,self.config_var,new_name)

    def next_db(self):
        '''returns the name of the next db after checking for existence - this could possibly use arcpy.CreateUniqueName instead'''
        new_name = os.path.join(self.parent_folder,"%s_%s.gdb" % (self.name,self.db_count))
        while arcpy.Exists(new_name):
            self.db_count += 1
            new_name = os.path.join(self.parent_folder,"%s_%s.gdb" % (self.name,self.db_count))

        return new_name

I've found 150 to be a good tradeoff between complexity and performance for my project. Write times tend to fluctuate between about 3 and 7 seconds with that threshold. Set db_size_threshold to whatever balance you are looking to strike on your own hardware.

0

There's a memory leak in ArcGIS 10 which is being fixed in SP3 apparently.

Also, I decided I would delete the 'in_memory' data and compact the database on each loop. which actually sped the application up. Then, when I run the script again, I delete the fgdb and recreate it. it's sped it all up by 30%. However, once the memory leak has been fixed, we expect much better gains in performance. Arcpy is a pig in loops...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.