Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I am looping through an amount of point arrays, creating data on the fly.

The features/arrays, are largely similar in size. One thing I have noticed, is that when I start the process, the time it takes to perform, rises from about 8 seconds, to around 1 minute. I can't see why this is happening, but you cna see the times slowly increase if you put a timer around the loop.

Is there a way to increase this performance? Is there a flush method? I believe I am properly handling my objects and deleting them after use, but I cannot for the life of me see why it is happening?

Any ideas?

The code is refactored, so there may be some typos, etc.

def createGeom(geom,header,band,scratchDB,buffer, dist):

filetime = (str(time.time())).split(".")
outfile = "fc" + filetime[0]+filetime[1]
outpath =  scratchDB + "tmp.gdb/Polygon/"
print outpath
sCon = ["1", "16", "2.0"] 
ts = header.split(" ")
ti = ts[6]
Current = ts[9]
LL = ts[2][2:5]
LU = ts[2][8:11]

outFeatureAggClass = outpath+outfile+"_agg"
outFeatureClass = outpath+outfile

arcpy.AggregatePoints_cartography(geom, outFeatureAggClass, dist)
arcpy.Buffer_analysis(outFeatureAggClass, outFeatureClass, buffer)



arcpy.AddField_management(outFeatureClass, "Name", "Text")
arcpy.AddField_management(outFeatureClass, "ID", "SHORT")
arcpy.AddField_management(outFeatureClass, "LL", "SHORT")
arcpy.AddField_management(outFeatureClass, "LU", "SHORT")        
arcpy.AddField_management(outFeatureClass, "Tim", "SHORT") 
arcpy.AddField_management(outFeatureClass, "Band", "SHORT")
arcpy.AddField_management(outFeatureClass, "Lvl", "TEXT") 
arcpy.AddField_management(outFeatureClass, "Cur", "SHORT")


rows = arcpy.UpdateCursor(outFeatureClass)
for row in rows:
    row.name = outfile
    row.ID = 1000
    row.LL = int(LL)
    row.LU = int(LU)
    row.Tim = int(ti)   
    row.Band = int(band)
    row.Lvl = sCon[int(band)]
    row.Cur = Current
    rows.updateRow(row)

del row
del rows    

UPDATE

I am now running this in ArcMap, and it's taking significantly less time, to a factor or about 3 or 4; what took 30 mins, now takes 7.

What's that all about?

EDIT

ESRI have agreed to look at the issue. I'll park this for now, ready to update it with any progress.

share|improve this question
    
Can you post some code? –  MerseyViking Jul 7 '11 at 13:44
    
nps, Yer 'tis... –  Hairy Jul 7 '11 at 13:49

3 Answers 3

You might want to try putting del row inside of the loop as the last call after rows.updateRow{row}. Your object deletion does not take place until you are outside of the loop, so it might just be the rather large collection of row objects that you are creating. I would think that the row objects are being deleted as you go out of scope, but that might not be the case.

You can also try using the garbage collection module to debug and/or fix the issue: http://docs.python.org/library/gc.html

Some other issues... do all of your casting outside the loop. e.g. LL = int(LL) before the loop, so that inside the loop you just do 'row.LL = LL' It looks like you are doing no calculations inside the loop that would change any of these values from row to row, so there is no need to do the casting inside the loop.

Edit: I did not realize that you were looping this against many files. The problem might be in your outer loop call, not in your function call. Since it works correctly in ArcMap, but not outside of ArcMap, you might be reloading arcpy or creating new geoprocessor objects with each call to a file. This is a very costly operation that gets much worse the more times you execute it.

I ran into a similar problem with an operation that I was trying to call every 20 seconds outside of ArcMap. Every time that I called the inner function from the outer timing loop, arcpy was being reloaded even though I imported arcpy into the outer timing loop. As a result, my inner function kept running slower and slower and was impossible to finish in 20 seconds. To get around this, I imported arcpy into the outer timing loop, and then passed arcpy as an argument into the inner function. Once I did this, my inner function was executing in under 3 seconds every time.

So, for your function, you might want to try changing its definition to:
def createGeom(geom,header,band,scratchDB,buffer, dist, arcpy): and then when you call it, make sure you pass arcpy as the 7th argument.

As far as I can tell, inside of ArcMap, arcpy is loaded once and then maintained for any calls against it, even if called by an inner function. Hence the much better performance when executed in ArcMap.

share|improve this answer
    
GC doesn't work, the rows loop is only ever run once, the casting is not taking anytime at all. I have now asked the question of esri, but it's insane that this kit is doing this, not handling it's own memory. Could it be indexes? –  Hairy Jul 8 '11 at 9:31
    
I seem to remember there being a known bug about cursors not releasing memory properly until the entire process is complete. This is avoidable in .NET, from what I've found, but large cursor processes in Python have been nothing but frustrating to me. –  Nathanus Jul 8 '11 at 15:34
    
I agree with @blord - where your del row is now, it just deletes the last row, rather than each iteration of the loop. –  djq Jul 8 '11 at 18:54
    
@celenius - There is only ever one row. I have changed it for completeness, but it only updates the one FC you have created, which is one polygon. –  Hairy Jul 11 '11 at 7:37
    
Ah, I see. I suppose it makes no difference as the loop is overwriting row each time and you are just deleting the last one. –  djq Jul 11 '11 at 12:32

Hmmm... it seems odd to use the del command, but the official docs use it. Other than that I can't see anything obvious. Are you getting the slowdown as the loop iterates, or is it every time you call your createGeom() function?

You could try commenting out the updateRow() call, just to see if it's slowing it down or whether it is something else. Also, are you able to wrap the loop in a transaction? My knowledge of arcpy is limited, but in general database-type updates are more efficient if the whole thing is in a transaction.

share|improve this answer
    
I put the Del commands in, to see if they have made a difference, which they haven't really. When I do the update row, it is only ever one row that is updating. I have a list of files (72) which I loop through and call createGeom for each file; it does this operation once. Puttinga timer around the operations sees the first one clear in 9 seconds, but the last one in 45 seconds, each one taking slightly longer than the previous one, which strikes me as an issue with a memory thing. I'll look at GC now - thanks –  Hairy Jul 8 '11 at 7:04
    
No real difference. I am disagrregating all of the functionality, to see if one is doing it, one function call, but they're all doing it! –  Hairy Jul 8 '11 at 9:29
    
Are you closing each file after you've finished with createGeom()? It may be an issue with arcPy keeping hold of some data until the file is closed. –  MerseyViking Jul 8 '11 at 11:05
    
Yes, as I open it, I gather the data, then I close it sorry, just to note I have updated the question, to reflect the differences when doing this in ArcMap! –  Hairy Jul 8 '11 at 11:12
up vote 1 down vote accepted

I have just heard back from ESRI who have confirmed this is a bug which is repaired in SP2 of ArcGIS 10.

Installing SP2 today, so will report back. The polygons were drawing at 5 seconds, ending up at 50 seconds, so this represents a considerable win for us, as all will draw around the 5 second mark.

Thanks for the advice peeps.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.