I have a relatively large result file from my network analysis in ArcGIS 10.5. Data represents 100 closest neighbors for ~1.5 mio locations. The dataset is stored in a file geodatabase and it contains 'no geography' since I specified
NO_LINES option in
MakeODCostMatrixLayer_na tool so I treat it as a table.
I would like to move this data out of GIS for further processing in Stata (and or R). Since Stata is not GIS.. and I also wanted to avoid shapefile for data of this size I thought first about using txt or csv file as an intermediate way.
Learning from previous solutions, I tried three solutions:
- Exporting via GUI in ArcCatalog
- Exporting via (64-bit) Python script with 'Export Feature Attribute to ASCII' (Spatial Statistics toolbox)
- Exporting via Python script with 'Table To Table' (Conversion toolbox)
- Exporting via Python script with 'Copy rows' (Data Management toolbox)
- Exporting via Python script with a cursor loop (as suggested by Caleb1987 here)
Oddly enough, all these solutions give me a resulting file with 30 mio observations (exactly the same number) instead of 150 mio.
What can be causing a problem like that? Are there any better solutions for exporting files of this size?
Note: all scripts were tested with smaller datasets and everything worked smoothly so I am pretty confident there are no mistakes there.
Update: I managed to test one more solution via
ogr2ogr. Using command line to connect to database and export to CSV:
ogr2ogr -f "CSV" "path\output.csv" "path\database.gdb" "feature_class"
This time I get to export 47mio records and the tool fails with many error messages similar to:
ERROR 1: Error occurred in filegdbtable.cpp at line 1284