Optimizing combining multiple feature classes into one via arcpy

Question

I have about 1000 feature classes spread over about 50 geodatabases which I am trying to append into an empty layer in a separate geodatabase.

When I previously did this on shapefiles, the process completed quickly. However due to earlier processes (requiring Query Tables) I had to move the data into a GDB, and this attempt to append the layers has been running for about 50 minutes.

Are there any recommendations to decrease the time required to merge these feature classes? Should I copy the feature classes to shapefiles first?

The layers all have the same field set; I am not using any field mappings.

General process:

1. Loop through the GDBs using arcpy.da.walk; add feature class path/name into a list
2. Create a feature class in the output GDB using one of the fcs above as a template
3. Use arcpy.Append_management(fclist, out_layername)

Note: While the first step (obtaining the list of FCs via arcpy.da.walk) takes a considerable amount of time, that is acceptable - I'd like to improve the append or merge step.

The specific code snippet I am trying to improve is:

# Set the workspace environment setting
arcpy.env.workspace = os.path.join(rootworkingdir, gdbName)
out_path = gdbpath
out_name = fcname
geometry_type = "POLYLINE"   
template = fclist[0]
# Execute CreateFeatureclass
arcpy.CreateFeatureclass_management(
    out_path, out_name, geometry_type, template)
out_name = out_path + "/" + out_name
arcpy.Append_management(fclist, out_layername)

The total number of expected features is about 5000.

Answer 1

As commented by @Vince:

There is never any reason to copy a file geodatabase table to a shapefile to populate a file geodatabase table (unless you want your data corrupted). The only thing you can do to reduce time is to drop the spatial index (and any other indexes) of the target table before you start inserts.

Don't forget that converting a geodatabase feature class to a shapefile is likely to truncate some of its field names for starters, but a number of other shapefile/dBase limitations are likely to come into play.

Answer 2

Using Merge_management in step-wise fashion (described below) was faster in this case than the Append_management for all of the layers at once. This did not require removing the indexes (via ListIndexes and RemoveIndex_Management or RemoveSpatialIndex_management, although removing the indexes per @Vince may speed processing; I didn't have a chance to test. Note: The Merge method was much faster within ArcMap (interactively) than via arcpy.

For each geodatabase:

1. Identify the desired feature classes
2. create a temporary list of fcs
3. Merge the fcs into a new layer in the final geodatabase

Once all GDBs are processed, merge the 50 layers from step 3 above.