Let us say, for the sake of argument, that you are working with a table that is over 2 GB in size (so .dbf is automatically ruled out... right?). To even create such a document, ArcGIS requires that you make it a "table" inside a File Geodatabase. Ignoring the hours-long conversion time in such an endeavor, working with the final product can prove to be an even larger nightmare. ArcGIS simply does not work for me when the size of a dataset gets to be several gigabytes. Can anyone suggest alternative programs or methods for processing such large sets of data? I am trying to avoid working with joins between 40-odd million records and the headache-inducing wait times involved there-in. I have been told in passing that using "backend RDBMS" programs can achieve this, but my own research has proven ineffectual.
I turn to you, StackExchange.
Edit: To further specify the requirements of the data, the table itself will have nothing to truly make it "spatial." There will be fields for X/Y Co-ordinates, but they will be simple double/float fields. I am looking for a proper way to do joins and relates across very large tables, in essence. The data is not spatial in the strictest sense; SDE is not something I need consider, I think.
I am hoping that any solution will allow me to host the data locally without creating a server or other such mumbo jumbo. Only I will need to access it, and if it needed to be shared, I would expect a simple export would be possible to a more ubiquitous format than ESRI's geodatabases.