Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I am currently writing a python script using the arcgisscripting module to process a reasonably large data set (~10,000 records in total) normalised over a small number of tables, 8 in total. The process consists of creating a feature based on coordinate tuples (x,y) and creating a graph (nodes and lines) using the relationships in the other 7 tables for guidance. The final output is a personal geodatabase (pgdb/fgdb) with nodes and edges spatial data sets that visually represent the relationships.

My initial attempt was to use queries of the new geodatabase tables and SearchCursor record sets to populate link tables (InsertCursor) for the many-to-many relationships that occur. This worked very well, except for the 15-20 min processing time.

Using the cProfiler module in Python, it was apparent that 'thrashing' a personal geodatabase when performing the search queries to populate the link tables with requests for cursors (Search and Insert cursors) caused the appalling performance.

With a little refactoring I have managed to get the processing time below 2.5 minutes. The trade off was partial construction of the geodatabase schema in code and limiting the requests for arcgisscripting cursors to InsertCursors once all relationships were collated.

My question is one of performance;

  • What techniques have people used to maintain reasonable compute times when working with large data set?
  • Are there any ESRI recommended methods that I've missed in my search for optimisation?

    I understand the overhead incurred when creating a arcgisscripting cursor, particularly if it is from a personal geodatabase, though after a lengthy search for performance related answers from this site and Google I am under the impression that performance isn't at the forefront of peoples endeavours.

  • As a user of ESRI products, does one expect and condone these performance lags?

UPDATE

After some work with this product I have accumulated a list of optimization techniques that have taken a process of converting spatial information from a propriety format to a geodatabase. This has been developed for personal and file geodatabase. Tidbits:

  1. Read you data and rationalize it in memory. This will cut your time in half.

  2. Create feature classes and tables in memory. Use the feature dataset keywork 'in_memory' to use your memory as ram disk, perform your functions there and then write out to disk

  3. To write out to disk use the CopyFeatureclass for feature classes, and CopyRow for tables.

These 3 things took a script that converted 100,000+ features to a geodatabase from 30 minutes to 30 - 40 seconds, this includes relationship classes. They are not to be used lightly, most of the methods above use a lot of memory, which could cause you issues if you are not paying attention.

share|improve this question
1  
Have you tried using a different storage format? How does a file geodatabase perform? –  Derek Swingley Apr 5 '11 at 20:59
    
A file geodatabase performs slightly worse than a personal geodatabase. I have spent yesterday setting up and tuning an ArcSDE instance to test performance on an enterprise format. I'll keep you posted on my findings –  OptimizePrime Apr 6 '11 at 13:49
1  
This doesn't help you now, but in 10.1 cursor performance in Python has been improved by a huge factor (something in the order of magnitude range on average cases) with the new data access module. –  Jason Scheirer Apr 27 '11 at 20:05
    
In_memory uses an the InMemoryWorkspace edndoc.esri.com/arcobjects/9.2/ComponentHelp/esriDataSourcesGDB/… which, after an arbitrary number of rows, dumps everything to a ScratchWorkspaceFactory (i.e. FileGDB) and relies on FileGDB to do all the work –  Ragi Yaser Burhum Apr 28 '11 at 22:17
add comment

4 Answers

up vote 41 down vote accepted

Although this question was already answered, I thought I could chime in an give my two cents.

DISCLAIMER: I worked for ESRI at the GeoDatabase team for some years and was in charge of maintaining various parts of GeoDatabase code (Versioning, Cursors, EditSessions, History, Relationship Classes, etc etc).

I think the biggest source of performance problems with ESRI code is not understanding the implications of using different objects, particularly, the "little" details of the various GeoDatabase abstractions! So very often, the conversation switches to the language being used as a culprit of the performance issues. In some cases it can be. But not all the time. Let's start with the language discussion and work our way back.

1.- The programming language that you pick only matters when you are doing something that is complicated, in a tight loop. Most of the time, this is not the case.

The big elephant in the room is that at the core of all ESRI code, you have ArcObjects - and ArcObjects is written in C++ using COM. There is a cost for communicating with this code. This is true for C#, VB.NET, python, or whatever else you are using.

You pay a price at initialization of that code. That may be a negligible cost if you do it only once.

You then pay a price for every subsequent time that you interact with ArcObjects.

Personally, I tend to write code for my clients in C#, because it is easy and fast enough. However, every time I want to move data around or do some processing for large amounts of data that is already implemented in Geoprocessing I just initialize the scripting subsystem and pass in my parameters. Why?

  • It is already implemented. So why reinvent the wheel?
  • It may actually be faster. "Faster than writing it in C#?" Yes! If I implement, say, data loading manually, it means that I pay the price of .NET context switching in a tight loop. Every GetValue, Insert, ShapeCopy has a cost. If I make the one call in GP, that entire data loading process will happen in the actual implementation of GP - in C++ within the COM environment. I don't pay the price for context switching because there is none - and hence it is faster.

Ah yes, so then the solution if to use a lot of geoprocessing functions. Actually, you have to be careful.

2. GP is a black box that copies data (potentially unnecessarily) around

It is a doubled-edged sword. It is a black box that does some magic internally and spits out results - but those results are very often duplicated. 100,000 rows can easily be converted into 1,000,000 rows on disk after you ran your data through 9 different functions. Using only GP functions is like creating a linear GP model, and well...

3. Chaining too many GP functions for large datasets is highly inefficient. A GP Model is (potentially) equivalent to executing a query in a really really really dumb way

Now don't get me wrong. I love GP Models - it saves me from writing code all the time. But I am also aware that it is not the most efficient way of processing large datasets.

Have you every heard of a Query Planner? It's job is to look at the SQL statement you want to execute, generate an execution plan in the form of a directed graph that looks a heck of a lot like a GP Model, look at the statistics stored in the db, and choose the most optimal order to execute them. GP just executes them in the order you put things because it doesn't have statistics to do anything more intelligently - you are the query planner. And guess what? The order in which you execute things is very dependent on your dataset. The order in which you execute things can make the difference between days and seconds and that is up to you to decide.

"Great" you say, I will not script things myself and be careful about how I write stuff. But do you understand GeoDatabase abstractions?

4.Not understanding GeoDatabase abstractions can easily bite you

Instead of pointing out every single thing that can possibly give you a problem, let me just point out a few common mistakes that I see all the time and some recommendations.

  • Understanding the difference between True/False for Recycling cursors. This tiny little flag set to true can make runtime orders of magnitude faster.
  • Put your table in LoadOnlyMode for data loads. Why update the index on every insert?
  • Understand that even though IWorkspaceEdit::StartEditing looks the same in all workspaces, they are very different beasts on every datasource. On an Enterprise GDB, you may have versioning or support for transactions. On shapefiles, it will have to be implemented in a very different way. How would you implement Undo/Redo? Do you even need to enable it (yes, it can make a difference in memory usage).
  • The difference between batch operations, or single row operations. Case in point GetRow vs GetRows - this is the difference between doing a query to get one row or doing one query to fetch multiple rows. A tight loop with a call to GetRow means horrible performance and it is culprit #1 of performance issues
  • Use UpdateSearchedRows
  • Understand the difference between CreateRow and CreateRowBuffer. Huge difference in insert runtime.
  • Understand that IRow::Store and IFeature::Store triggers super heavy polymorphic operations. This is probably reason #2 culprit of really slow performance. It doesn't just save the row, this is the method that makes sure your geometric network is OK, that the ArcMap Editor gets notified that a row has changed, that notifies all relationship classes that have anything to do with this row validate to make sure that the cardinality is valid, etc. You should not be inserting new rows with this, you should be using an InsertCursor!
  • Do you want (need) to do those inserts in an EditSession? It makes a huge difference if you do or not. Some operations require it (and make things slower), but when you don't need it, skip the undo/redo features.
  • Cursors are expensive resources. Once you have a handle to one, you are guaranteed that you will have Consistency and Isolation and that has a cost.
  • Cache other resources like database connections (don't create and destroy your Workspace reference) and Table handles (every time you open or close one - several metadata tables need to be read).
  • Putting FeatureClasses inside or outside a FeatureDataset makes a huge difference in performance. It is not meant as an organizational feature!

5.And last and not least...

Understand the difference between I/O bound and CPU bound operations

I honestly thought about expanding more on every single one of those items and perhaps doing a series of blog entries that covers every single one of those topics, but my calendar's backlog list just slapped me in the face and started yelling at me.

My two cents.

share|improve this answer
2  
+1, great information. –  blah238 Apr 28 '11 at 3:43
4  
thanks. I should have been doing work instead of writing this post haha –  Ragi Yaser Burhum Apr 28 '11 at 6:11
3  
+1 Thank you very very much for your input Mr Burhum. This is the type of response I was aiming to receive. If I could vote it twice I could!! What ArcGISScripting (python) users should take from this answer is, though the links reflect ArcObjects and .Net concepts, the underlying COM objects are the same, understanding these objects will help you better plan code in any language. Lots of great information here!! –  OptimizePrime Apr 29 '11 at 18:34
1  
@OptimizePrime That is a great summary. And you are right - you can't ignore ArcObjects implications if you want to squeeze performance out of ESRI products –  Ragi Yaser Burhum Apr 29 '11 at 22:40
add comment

Generally, for performance computations, I try to stay away from using any ESRI related stuff. For your example, I would suggest doing the process in steps, the first step reading the data into normal python objects, doing the calculations, and then the final step converting to the final ESRI spatial format. For ~10k records, you could probably get away with storing everything in memory for the processing, which would give a definite performance gain.

share|improve this answer
    
Thank you for your response. It is a good suggestion. I've started to refactor the code to perform the required procedures before using arcgisscripting. After working with the software since its ArcInfo days I find it frustrating that CPU performance and Hardware capability increase, ArcGIS Map/Info/Editor X.X performance is stagnate. Maybe the introduction of GPU's may boost things. Though a good refactor of the ESRI code base may help also –  OptimizePrime Apr 6 '11 at 14:00
add comment

Depending on the hardware you have, you might also consider the option displayed in the ESRI Geocoder Example; it gives you a framework to breakup a large dataset and run multiple instances of python to give you almost a multi-threaded approach. I saw geocoding performance go from 180,000 per hour in a single Python instance to over one million thanks to spinning up 8 parallel processes on my machine.

I have seen that getting away as much as I can and keeping data work in the database, and functional work in my tables and just using what is explicit GIS in the ESRI realm give major performance increases.

share|improve this answer
    
These are great idea's. I have the opportunity to thread some processes in this script, but I'm finding my bottle necks are initialising the COM libraries and Geodatabase I/O. In regards to I/O, I've reduced the need to a single write. If I spend any more time optimizing, my boss is going to have a fit ;) So I'm leaving threading as the last squeeze of performance if he asks for more. At the moment I'm processing 60,000 features a minute. –  OptimizePrime Apr 29 '11 at 18:46
add comment

You might find these other forum posts interesting as they are in the optimization context but for raster data and overall:

Compiling Python Scripts That Use ArcGIS Geoprocessing Tools

Processing time using ArcGIS Hydrology toolbox tools in standalone Python script vs. ArcCatalog

the gp.scratchworkspace setting made a big difference for me in some python code I wrote for doing watershed delineations.

Could you post some code examples that demonstrate numbers 1 and 2 in your UPDATE to your original question? I'd be interested to see the mechanics of that (though I'm assuming you're dealing with feature class data only here)

thanks, Tom

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.