I am attempting to automate an ArcMap process with Python and I am having difficulty with a particular part of a process taking too long when performed on larger datasets.

The process is as follows:

Step 1: Begin with 4 Feature Layers of geospatial data.

Step 2: Use Spatial Joins to add together for comparison.

arcpy.SpatialJoin_analysis(input_table_1, input_table_2, output_table)

Step 3: Change these into temporary feature layers and use AddJoin_management to join them together

arcpy.MakeFeatureLayer_management(output_table, temp_table)

Step 4: Use AddJoin_management to add these new feature layers together to make a complete master table with all of the previous spatially joined tables.

arcpy.AddJoin_management(temp_table_1, "field", temp_table_2, "field")

Now the further steps require summary statistics performed on this master table but joined tables may not be used as input for summary stats.

I attempted to work around this using the CopyRows_management function and performing summary statistics on the identical table output:

arcpy.CopyRows_management(joined_table, copy_rows_output_table)

However, while this works fine on a smaller table of approx 100k rows, it is now needed to be performed on a feature layer with over 5m entries and it takes virtually the entire night.

Are there any known ways to either:

  1. Convert temporary feature layers to a permanent table faster, or;
  2. Perform any form of editing on a joined table?

I have already attempted:

  • Running from the command line
  • Writing master joined table to csv
  • Using compression on gdb with Compact_management
  • I attempted to use the Statistics_analysis function on the master table but this results in error messages as it is a joined table. I attempted to work around this by using the CopyRows function but this is what is very slow.
    – user146631
    Jul 23 '19 at 14:32
  • 2
    Have you tried adding attribute indices to the fields that you are joining on?
    – Hornbydd
    Jul 23 '19 at 19:15
  • 2
    I'd add missing fields and populate them using update cursor. Joined table is often a pain.
    – FelixIP
    Jul 23 '19 at 19:43
  • @FelixIP I attempted this but running UpdateCursor on a feature layer of 5m values runs even slower.
    – user146631
    Jul 25 '19 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.