Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

All,

I was wondering if anyone has any advice or "tales from the dark side" about changing the AutoCommitInterval property of the ITransactionsOptions interface? It's the sort of thing I generally leave well alone!

I am developing some code that populates dictionaries with a lot of information. The dictionary key is an integer and the item is an ArrayList with potentially several thousand entries and I can have many thousands of key/item entries in my dictionary. Ultimately I want to write the lot to a File Geodatabase Table. I was hitting the out of memory exception so a solution was to periodically write the contents of the dictionary to the table, clear out the dictionary and continue in my loop. This works but the process of writing the dictionary data to the table is quite slow (as it's writing millions of rows) and I was looking for ways to improve performance.

My code is using an insert cursor with row buffering and I believe that section of code is as optimal as I can get it, so I was looking at the best practice page on the ESRI website when I picked up on the idea of changing the Auto Commit Interval. I've never tinkered around with that before and I was wondering if people had any advice, it looks like I can set that to zero and do the commit right at the end, is this advisable, does it actually improve write performance?

Thanks

Duncan

share|improve this question
add comment

1 Answer 1

All autocommit does is keep an internal counter of the amount of inserts you have done and then flushes when that counter is set. This is specially useful for ArcSDE since you can basically buffer the inserts on the client and then send them in one big chunk for insertion (instead of getting chatty communication in an insert-by-insert basis).

Since you are using FileGDB, the impact that you will get is not as dramatic as what you are looking for.

If you are inserting a lot of features in batch, I would recommend you put the table in LoadOnlyMode and do the whole thing outside of an EditSession.

The LoadOnlyMode tells FileGDB not to update the indexes on every insert and to wait until the very end, when you turn LoadOnlyMode off, to rebuild them (much faster).

Editing outside of an EditSession tells FileGDB that you don't care about doing the inserts in a transactional basis (no undo/redo, or internal object pool, etc).

share|improve this answer
    
Ragi, Thanks for the pointer to the LoadOnlyMode, not used that one before. I have implemented it in my code but I've not observed any significant improvement. I've also discovered that the ITransactionsOptions Interface works only with ArcSDE workspaces. –  Hornbydd Jul 14 '12 at 20:29
    
Yeah, I guess it doesn't mean anything if you are not using an ArcSDE-based datasource. I am a bit skeptic that the batch insertion in LoadOnlyMode is not proving to give you a speed increase. Are you using an insert Cursor with RowBuffers? forums.esri.com/Thread.asp?c=93&f=992&t=79743 –  Ragi Yaser Burhum Jul 14 '12 at 20:54
    
I am using insert cursors with rowbuffers. I'm wondering now if I am just being "a bit optimistic" as one Table generates 41 million rows! Which is quite a lot of data... :) –  Hornbydd Jul 14 '12 at 23:32
2  
hahahahahaha yeah, 41 million rows can take quite sometime. Another way you could speed this up is by using multiple processes in different disks, at the end, you could append them together. –  Ragi Yaser Burhum Jul 14 '12 at 23:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.