Let us say, for the sake of argument, that you are working with a table that is over 2 GB in size (so .dbf is automatically ruled out... right?). To even create such a document, ArcGIS requires that you make it a "table" inside a File Geodatabase. Ignoring the hours-long conversion time in such an endeavor, working with the final product can prove to be an even larger nightmare. ArcGIS simply does not work for me when the size of a dataset gets to be several gigabytes. Can anyone suggest alternative programs or methods for processing such large sets of data? I am trying to avoid working with joins between 40-odd million records and the headache-inducing wait times involved there-in. I have been told in passing that using "backend RDBMS" programs can achieve this, but my own research has proven ineffectual.

I turn to you, StackExchange.

Edit: To further specify the requirements of the data, the table itself will have nothing to truly make it "spatial." There will be fields for X/Y Co-ordinates, but they will be simple double/float fields. I am looking for a proper way to do joins and relates across very large tables, in essence. The data is not spatial in the strictest sense; SDE is not something I need consider, I think.

I am hoping that any solution will allow me to host the data locally without creating a server or other such mumbo jumbo. Only I will need to access it, and if it needed to be shared, I would expect a simple export would be possible to a more ubiquitous format than ESRI's geodatabases.

  • What kind of processing do you plan to do?
    – whuber
    Commented Mar 2, 2011 at 22:00
  • 1
    Did you wait until your data is loaded before indexing the join field? Also, you might use this as an excuse to get a solid state drive. Commented Mar 3, 2011 at 15:05

1 Answer 1


The short answer is yes dbf is out of the question.
Next up is what your data needs to accomplish.
The answer will determine the capability you can depend on from other systems and what you will be required to depend on from ESRI.
In short if you have any special objects (i.e. network or com objects) in your data or non-standard data types you will probably be stuck with ESRI processsing.

If the answer is no here you are in an extreme amount of luck.
Just a few good options as I am sure many more will chime in...

Oracle spatial
mssql server spatial paid
mssql server express

All of this was on the premise that a multi GB db is what you have to deal with and that you need to do some joins and process the data in some manner.
A bit of a rethink is required to give a comprehensive answer...

OK so you can use your data in (what I would term as flat tables another term would be business tables).
These tables can live in the rdbms of your choice and not have any special GIS schema attached.
You can perform your joins and do whatever it is you do in the realm of the db (via sql or other application).

However, when you want to display this data (I will assume it is point data) you will need an engine that can do what you need. Arcmap is one of those engines.
You might accomplish the best case management in a couple of ways...

1 you could create table views and just display the portion of the data you want.
2. you could create several queries and re-apply those to a minimal dataset in arcmap to see just the rows you want.
3. you could just open the data and wait on it.

  • 1
    I'm looking at postgres now, actually!
    – Nathanus
    Commented Mar 2, 2011 at 22:26
  • lots of nice stuff in there. so to ask whuber's question again. What kind of processing are you doing?
    – Brad Nesom
    Commented Mar 2, 2011 at 22:43
  • I don't think I follow the meaning of "what kind of processing." I'm just looking to display, join, and query the data without dealing with all of ESRI's baggage when doing so.
    – Nathanus
    Commented Mar 3, 2011 at 16:37
  • We got that from your question. "Can anyone suggest alternative programs or methods for processing such large sets of data?" See answer for a redirect.
    – Brad Nesom
    Commented Mar 3, 2011 at 20:02
  • I think this pretty much covers all the bases I had in mind. Thank you.
    – Nathanus
    Commented Mar 3, 2011 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.