Are there any efficient methods for processing ArcGIS-based database tables outside of the ArcEtc program library?

Question

Let us say, for the sake of argument, that you are working with a table that is over 2 GB in size (so .dbf is automatically ruled out... right?). To even create such a document, ArcGIS requires that you make it a "table" inside a File Geodatabase. Ignoring the hours-long conversion time in such an endeavor, working with the final product can prove to be an even larger nightmare. ArcGIS simply does not work for me when the size of a dataset gets to be several gigabytes. Can anyone suggest alternative programs or methods for processing such large sets of data? I am trying to avoid working with joins between 40-odd million records and the headache-inducing wait times involved there-in. I have been told in passing that using "backend RDBMS" programs can achieve this, but my own research has proven ineffectual.

I turn to you, StackExchange.

Edit: To further specify the requirements of the data, the table itself will have nothing to truly make it "spatial." There will be fields for X/Y Co-ordinates, but they will be simple double/float fields. I am looking for a proper way to do joins and relates across very large tables, in essence. The data is not spatial in the strictest sense; SDE is not something I need consider, I think.

I am hoping that any solution will allow me to host the data locally without creating a server or other such mumbo jumbo. Only I will need to access it, and if it needed to be shared, I would expect a simple export would be possible to a more ubiquitous format than ESRI's geodatabases.

Answer 1

The short answer is yes dbf is out of the question.
Next up is what your data needs to accomplish.
The answer will determine the capability you can depend on from other systems and what you will be required to depend on from ESRI.
In short if you have any special objects (i.e. network or com objects) in your data or non-standard data types you will probably be stuck with ESRI processsing.

If the answer is no here you are in an extreme amount of luck.
Just a few good options as I am sure many more will chime in...

Oracle
Oracle spatial
Postgres
Postgres/postgis
mssql server spatial paid
mssql server express

All of this was on the premise that a multi GB db is what you have to deal with and that you need to do some joins and process the data in some manner.
A bit of a rethink is required to give a comprehensive answer...

OK so you can use your data in (what I would term as flat tables another term would be business tables).
These tables can live in the rdbms of your choice and not have any special GIS schema attached.
You can perform your joins and do whatever it is you do in the realm of the db (via sql or other application).

However, when you want to display this data (I will assume it is point data) you will need an engine that can do what you need. Arcmap is one of those engines.
You might accomplish the best case management in a couple of ways...

1 you could create table views and just display the portion of the data you want.
2. you could create several queries and re-apply those to a minimal dataset in arcmap to see just the rows you want.
3. you could just open the data and wait on it.