I am attempting to automate an ArcMap process with Python and I am having difficulty with a particular part of a process taking too long when performed on larger datasets.
The process is as follows:
Step 1: Begin with 4 Feature Layers of geospatial data.
Step 2: Use Spatial Joins to add together for comparison.
arcpy.SpatialJoin_analysis(input_table_1, input_table_2, output_table)
Step 3: Change these into temporary feature layers and use AddJoin_management to join them together
Step 4: Use AddJoin_management to add these new feature layers together to make a complete master table with all of the previous spatially joined tables.
arcpy.AddJoin_management(temp_table_1, "field", temp_table_2, "field")
Now the further steps require summary statistics performed on this master table but joined tables may not be used as input for summary stats.
I attempted to work around this using the CopyRows_management function and performing summary statistics on the identical table output:
However, while this works fine on a smaller table of approx 100k rows, it is now needed to be performed on a feature layer with over 5m entries and it takes virtually the entire night.
Are there any known ways to either:
- Convert temporary feature layers to a permanent table faster, or;
- Perform any form of editing on a joined table?
I have already attempted:
- Running from the command line
- Writing master joined table to csv
- Using compression on gdb with Compact_management