Performance of ArcGIS Engine using multiple file geodatabases as opposed to one?

Question

I'm trying to decide the best way to organize my data for an ArcGIS Engine application. I am particularly interested in map display and query speed. Currently I have all of my data separated into separate file geodatabases based on theme. So I have Transportation.gdb, Utilities.gdb, etc. The data doesn't necessarily need to be organized based on themes, and I'm considering putting it all in one file geodatabase.

I will be doing my own testing, but I wanted to throw the question out to the community.

In general, is using a single file geodatabase faster than using multiple (roughly 7) smaller ones? I'm interested in any other pros/cons as well.

NOTE: the software and all data will be on the customer's local machine. No data served on the web or over a network, and the amount of data is fairly small (roughly 100,000 features).

Answer 1

I am going to go the other way and actually say that no, it is not a good performance improvement to separate the GeoDatabases for this particular use-case you described .

You have to remember that there is a cost associated with a connection to a DB. In the case of the GeoDatabase, it is loading all the related metadata tables. So whenever you separate your data into multiple GDBs, you are just increasing that cost, because now you have to open multiple versions of these tables (one for each DB). Multiplexing to query the different DBs usually may also mean i/o with cache that gets invalidated.

Nevertheless, there are a few cases when having multiple DBs may work better. For example. Consider the case of a personal gdb (not filegdb) that is 700MB vs two that are 350MB a piece. The MS Jet driver (what is used to interact with .mdb files) will memory map files smaller than 500MB - so if the machine has enough memory, you will be interacting with DBs fully in memory vs any disk i/o. Much much faster. The 700MB file will not be memory mapped.

Taking this case out of the equation, then it doesn't make sense to do separate dbs. ArcMap, as it is looping through the layers, will query each layer sequentially, so you don't have any parallelism going on.

You are better off rebuilding your FileGDB Indexes instead.

And yes, an SSD would definitely help.

Answer 2

Actually it's normally the other way around; smaller databases query faster. It's like asking if you can find stuff faster if you throw everything in a big heap in the basement rather than sort it into individual filing cabinets. When you have individual databases, that's like having 6 filing cabinets that you can disregard straight from the outset, and don't need to look through. Of course this assumes that you know which database needs querying - if you need to look through all of them anyway, then one big one might indeed be faster (because it can optimize the dataset as a whole).

Answer 3

At one time, I had a similar setup with ArcReader on devices that weren't spec'd very well for GIS and were lucky to maintain a stable network connection to the GIS server(we're talking unstable wired connections...not wireless).

I had numerous databases that were generally broken about by "theme", and also by frequency of update. I broke them out by daily, monthly, annually, or tri-annually (which was Aerial/Planimetric update schedule). Since they were updated via robocopy, I didn't want to move any data that was unnecessary onto these devices.

If you are in an environment where you don't have robust geodatabase replication capability or you are simply receiving the file geodatabase for distribution, it may be easier to manage by breaking out your data storage in this way.

To answer your performance question: I never noticed any speed decreases by breaking out my data stores into separate file geodatabases. That doesn't mean there wasn't any, but if there was, it was not human-perceptible. It is worth noting that these configurations had all file geodatabases on 1 hard disk--you might get a performance gain if you had them spread across SCSI/SSD devices.

Answer 4

I once had about five ArcGIS Server WebADF web applications that each covered a different geographic area, but they all shared common datasets. The killer was that the apps were all dynamic (nothing was cached) and we had oil and gas wells in them that could number in the hundreds of thousands (millions actually for the whole US). Doing queries on the entire dataset was painful - actually they would usually just timeout. Clipping the data out for each area and putting it in a separate datastore kept our performance up and our customers happy. Like you, we also kept the file geodatabases stored on the HDD on the server, which also helped ALOT. We had a automated process that clipped the data out to each file geodatabase each night.

Not exactly an answer, but more of a case study in something sort of similar to what you are thinking about doing. If we wouldn't have had so many dynamic features to deal with, we might have not had to do that. Sometimes doing things a little out of the ordinary is necessary.