Merging shapefiles from subfolders to corresponding new folder using ArcPy?

Question

The dataset have shapefiles saved in different folders, as shown below.

The numbers of shapefiles in each folder are not equal. The content of one folder is shown below.

I need to merge all shapefiles in each folder and save the merge results in their corrsponding folder based on the FacIDxx, as shown below.

The following code works fine for the requirements mentioned above, but it takes lots of time when the dataset folder has many shapefiles to be merged.

import arcpy

#resetting the workspace
def find_all_fcs(workspace):
    prev_workspace = arcpy.env.workspace

    arcpy.env.workspace = workspace
    fcs = arcpy.ListFeatureClasses()

    arcpy.env.workspace = prev_workspace
    return fcs

filein0=r"C:\Users\TL\Desktop\812test\SplitAllRoutes"
arcpy.CreateFolder_management("C:/Users/TL/Desktop/812test", "Merge")
fileinIntersect = r"C:\Users\TL\Desktop\812test\Intersect"
fcs = find_all_fcs(filein0)
for fc in fcs:
    tempname_Intersect = fileinIntersect+'\\'+fc[:-4]
    fcs_Intersect = find_all_fcs(tempname_Intersect)
    arcpy.CreateFolder_management(r"C:\Users\TL\Desktop\812test\Merge", fc[:-4])
    input_Merge = []
    for fc_Intersect in fcs_Intersect:
        tempname_toMerge = tempname_Intersect+'\\'+fc_Intersect
        input_Merge.append(tempname_toMerge)
    output_Merge = r"C:\Users\TL\Desktop\812test\Merge"+'\\'+fc[:-4]+'\\'+fc[:-4]+'.shp'
    arcpy.Merge_management(input_Merge, output_Merge)

I wonder if someone can help me improve the efficiency of the code.

Answer 1

os.path is faster than arcpy for filesystem manipulation - use that to search your folders and set up the new folders. To speed up the merge, use append_management. Create the new folder, copy the first shapefile, then append the rest of the shapefiles to that one. Append will also allow you to keep master files that can be added to over time (if needed). Merge has to create a new feature class before merging the data. Append does not do this, so may be slightly faster. The processing time is going to depend on the size of the data.