I'm using ArcGIS 10.0 on Windows 7 64-bit with 4GB of RAM.
I have some very large tables in CSV format to import to ArcGIS, they all have about 30 fields, upwards of 5 million records per table (a few have double that or more), and file sizes up to about 5 GB. I am trying to import each of them into a file geodatabase as separate tables so I can, ultimately, link them to a feature class and analyze the results in the tables according to their location.
The problem is that ArcGIS seems to just quit importing records at a certain point. I'm using the "Table to Table" tool under Conversion > To Geodatabase, but the "Copy Rows" tool has the same problem. Even if I just add the CSV file directly to ArcGIS without trying to convert it to an FGDB table first, the problem is the same. One of my tables has about 11 million records, and ArcGIS only imports about 10 million of them. ArcGIS doesn't tell me that any error has occurred, the tool just finishes as if nothing is wrong.
I've tried it a few times now and the number of records that make it into the FGDB table is always the same, and doesn't appear to be a file size limit I've ever heard of (not a square of 2 or 16). ArcGIS was able to import another CSV with about 6 million records and all the records came through (though with the problems I'm having with the larger table, the smaller one is kind of suspect now too). ESRI's web site lists the following size limits in a file geodatabase, and I'm far from hitting any of them:
- File geodatabase size: No limit
- Table or feature class size: 1 TB (default), 4 GB or 256 TB with keyword
- Number of feature classes and tables: 2,147,483,647
- Number of fields in a feature class or table: 65,534
- Number of rows in a feature class or table: 2,147,483,647
- Geodatabase name length: Number of characters the operating system allows in a folder
- Feature class or table name length: 160 characters
- Field name length: 64 characters
- Text field width: 2,147,483,647
All I really need to do to these tables are add a couple fields, delete a couple others, and generate values for the new fields (sums of a few of the existing fields). I'm using ArcGIS for it because I'm familiar with the field calculator and I know (or knew, until now) that it could handle tables consisting of millions of records, whereas most other desktop software I have handy (MS Access/Excel) chokes on that many records. So I'm open to using some other piece of software to manipulate the original table and then exporting the (much smaller) resulting table to ArcGIS. Really, the fact that I'm having this problem and that ArcGIS is not giving me any errors or warnings that the problem is even occurring makes me want to handle this data outside ArcGIS as much as possible.