Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

Let us say, for the sake of argument, that you are working with a table that is over 2 GB in size (so .dbf is automatically ruled out... right?). To even create such a document, ArcGIS requires that you make it a "table" inside a File Geodatabase. Ignoring the hours-long conversion time in such an endeavor, working with the final product can prove to be an even larger nightmare. ArcGIS simply does not work for me when the size of a dataset gets to be several gigabytes. Can anyone suggest alternative programs or methods for processing such large sets of data? I am trying to avoid working with joins between 40-odd million records and the headache-inducing wait times involved there-in. I have been told in passing that using "backend RDBMS" programs can achieve this, but my own research has proven ineffectual.

I turn to you, StackExchange.

Edit: To further specify the requirements of the data, the table itself will have nothing to truly make it "spatial." There will be fields for X/Y Co-ordinates, but they will be simple double/float fields. I am looking for a proper way to do joins and relates across very large tables, in essence. The data is not spatial in the strictest sense; SDE is not something I need consider, I think.

I am hoping that any solution will allow me to host the data locally without creating a server or other such mumbo jumbo. Only I will need to access it, and if it needed to be shared, I would expect a simple export would be possible to a more ubiquitous format than ESRI's geodatabases.

share|improve this question
    
What kind of processing do you plan to do? –  whuber Mar 2 '11 at 22:00
1  
Did you wait until your data is loaded before indexing the join field? Also, you might use this as an excuse to get a solid state drive. –  Kirk Kuykendall Mar 3 '11 at 15:05
add comment

1 Answer

up vote 3 down vote accepted

The short answer is yes dbf is out of the question.
Next up is what your data needs to accomplish.
The answer will determine the capability you can depend on from other systems and what you will be required to depend on from ESRI.
In short if you have any special objects (i.e. network or com objects) in your data or non-standard data types you will probably be stuck with ESRI processsing.

If the answer is no here you are in an extreme amount of luck.
Just a few good options as I am sure many more will chime in...

Oracle
Oracle spatial
Postgres
Postgres/postgis
mssql server spatial paid
mssql server express

All of this was on the premise that a multi GB db is what you have to deal with and that you need to do some joins and process the data in some manner.
A bit of a rethink is required to give a comprehensive answer...

OK so you can use your data in (what I would term as flat tables another term would be business tables).
These tables can live in the rdbms of your choice and not have any special GIS schema attached.
You can perform your joins and do whatever it is you do in the realm of the db (via sql or other application).

However, when you want to display this data (I will assume it is point data) you will need an engine that can do what you need. Arcmap is one of those engines.
You might accomplish the best case management in a couple of ways...

1 you could create table views and just display the portion of the data you want.
2. you could create several queries and re-apply those to a minimal dataset in arcmap to see just the rows you want.
3. you could just open the data and wait on it.

share|improve this answer
1  
I'm looking at postgres now, actually! –  Nathanus Mar 2 '11 at 22:26
    
lots of nice stuff in there. so to ask whuber's question again. What kind of processing are you doing? –  Brad Nesom Mar 2 '11 at 22:43
    
I don't think I follow the meaning of "what kind of processing." I'm just looking to display, join, and query the data without dealing with all of ESRI's baggage when doing so. –  Nathanus Mar 3 '11 at 16:37
    
We got that from your question. "Can anyone suggest alternative programs or methods for processing such large sets of data?" See answer for a redirect. –  Brad Nesom Mar 3 '11 at 20:02
    
I think this pretty much covers all the bases I had in mind. Thank you. –  Nathanus Mar 3 '11 at 22:04
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.