Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I'm trying to decide the best way to organize my data for an ArcEngine application. I am particularly interested in map display and query speed. Currently I have all of my data separated into separate file geodatabases based on theme. So I have Transportation.gdb, Utilities.gdb, etc. The data doesn't necessarily need to be organized based on themes, and I'm considering putting it all in one file geodatabase.

I will be doing my own testing, but I wanted to throw the question out to the community.

In general, is using a single file geodatabase faster than using multiple (roughly 7) smaller ones? I'm interested in any other pros/cons as well.

NOTE: the software and all data will be on the customer's local machine. No data served on the web or over a network, and the amount of data is fairly small (roughly 100,000 features).

share|improve this question

4 Answers 4

up vote 4 down vote accepted

I am going to go the other way and actually say that no, it is not a good performance improvement to separate the GeoDatabases for this particular use-case you described .

You have to remember that there is a cost associated with a connection to a DB. In the case of the GeoDatabase, it is loading all the related metadata tables. So whenever you separate your data into multiple GDBs, you are just increasing that cost, because now you have to open multiple versions of these tables (one for each DB). Multiplexing to query the different DBs usually may also mean i/o with cache that gets invalidated.

Nevertheless, there are a few cases when having multiple DBs may work better. For example. Consider the case of a personal gdb (not filegdb) that is 700MB vs two that are 350MB a piece. The MS Jet driver (what is used to interact with .mdb files) will memory map files smaller than 500MB - so if the machine has enough memory, you will be interacting with DBs fully in memory vs any disk i/o. Much much faster. The 700MB file will not be memory mapped.

Taking this case out of the equation, then it doesn't make sense to do separate dbs. ArcMap, as it is looping through the layers, will query each layer sequentially, so you don't have any parallelism going on.

You are better off rebuilding your FileGDB Indexes instead.

And yes, an SSD would definitely help.

share|improve this answer
1  
Oh. The memory mapping of <500mb .mdb's is interesting. I had written off personal gdb's as not good for anything other than re-ordering and re-naming fields in ms-access instead of the painful add-a-copy-and-delete process needed in arcgis. Maybe now I have another reason to use them from time to time. Is the 500mb tipping point file on disk size or something else? (e.g. a jpeg can be 30kb on disk yet consume multi megabytes of ram when open). –  matt wilkie Jan 26 '12 at 6:16
1  
As far as I remember, this was behavior from the Jet engine itself, and not ESRI triggered. Also, it was slightly smaller than 500MB. Good question about file size vs memory. I think it was file size - but I do not remember exactly, to be honest with you –  Ragi Yaser Burhum Jan 26 '12 at 16:26

Actually it's normally the other way around; smaller databases query faster. It's like asking if you can find stuff faster if you throw everything in a big heap in the basement rather than sort it into individual filing cabinets. When you have individual databases, that's like having 6 filing cabinets that you can disregard straight from the outset, and don't need to look through. Of course this assumes that you know which database needs querying - if you need to look through all of them anyway, then one big one might indeed be faster (because it can optimize the dataset as a whole).

share|improve this answer

At one time, I had a similar setup with ArcReader on devices that weren't spec'd very well for GIS and were lucky to maintain a stable network connection to the GIS server(we're talking unstable wired connections...not wireless).

I had numerous databases that were generally broken about by "theme", and also by frequency of update. I broke them out by daily, monthly, annually, or tri-annually (which was Aerial/Planimetric update schedule). Since they were updated via robocopy, I didn't want to move any data that was unnecessary onto these devices.

If you are in an environment where you don't have robust geodatabase replication capability or you are simply receiving the file geodatabase for distribution, it may be easier to manage by breaking out your data storage in this way.

To answer your performance question: I never noticed any speed decreases by breaking out my data stores into separate file geodatabases. That doesn't mean there wasn't any, but if there was, it was not human-perceptible. It is worth noting that these configurations had all file geodatabases on 1 hard disk--you might get a performance gain if you had them spread across SCSI/SSD devices.

share|improve this answer

I once had about five ArcGIS Server WebADF web applications that each covered a different geographic area, but they all shared common datasets. The killer was that the apps were all dynamic (nothing was cached) and we had oil and gas wells in them that could number in the hundreds of thousands (millions actually for the whole US). Doing queries on the entire dataset was painful - actually they would usually just timeout. Clipping the data out for each area and putting it in a separate datastore kept our performance up and our customers happy. Like you, we also kept the file geodatabases stored on the HDD on the server, which also helped ALOT. We had a automated process that clipped the data out to each file geodatabase each night.

Not exactly an answer, but more of a case study in something sort of similar to what you are thinking about doing. If we wouldn't have had so many dynamic features to deal with, we might have not had to do that. Sometimes doing things a little out of the ordinary is necessary.

share|improve this answer
    
Thanks for the answer. It doesn't quite match my situation, but it's good insight for other people with a similar situation. I failed to mention that all the data will be on the customer's local machine, along with the software. No data is being served over the internet (other then when they need to install updates for the software). Also, the amount of data I'm working with is a tiny fraction of the amount you were working with. –  Tanner Jan 11 '12 at 21:11
4  
I didn't think you were serving over the web, but even having the FGDBs out on a network share could slow things down with data going over the pipes. If you're not working with huge datasets, I don't think separate FGDBs will do you much good - it might be more of a pain than it would be worth. –  Chad Cooper Jan 11 '12 at 21:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.