0

The dataset have shapefiles saved in different folders, as shown below. enter image description here

The numbers of shapefiles in each folder are not equal. The content of one folder is shown below. enter image description here

I need to merge all shapefiles in each folder and save the merge results in their corrsponding folder based on the FacIDxx, as shown below. enter image description here enter image description here

The following code works fine for the requirements mentioned above, but it takes lots of time when the dataset folder has many shapefiles to be merged. 

import arcpy

#resetting the workspace
def find_all_fcs(workspace):
    prev_workspace = arcpy.env.workspace

    arcpy.env.workspace = workspace
    fcs = arcpy.ListFeatureClasses()

    arcpy.env.workspace = prev_workspace
    return fcs

filein0=r"C:\Users\TL\Desktop\812test\SplitAllRoutes"
arcpy.CreateFolder_management("C:/Users/TL/Desktop/812test", "Merge")
fileinIntersect = r"C:\Users\TL\Desktop\812test\Intersect"
fcs = find_all_fcs(filein0)
for fc in fcs:
    tempname_Intersect = fileinIntersect+'\\'+fc[:-4]
    fcs_Intersect = find_all_fcs(tempname_Intersect)
    arcpy.CreateFolder_management(r"C:\Users\TL\Desktop\812test\Merge", fc[:-4])
    input_Merge = []
    for fc_Intersect in fcs_Intersect:
        tempname_toMerge = tempname_Intersect+'\\'+fc_Intersect
        input_Merge.append(tempname_toMerge)
    output_Merge = r"C:\Users\TL\Desktop\812test\Merge"+'\\'+fc[:-4]+'\\'+fc[:-4]+'.shp'
    arcpy.Merge_management(input_Merge, output_Merge)

I wonder if someone can help me improve the efficiency of the code.

|improve this question
  • 1
    What does your ArcGIS Runtime tag have to do with your question? Also, you seem to be asking for a code review when such are at best problematic and more likely specifically off-topic: meta.gis.stackexchange.com/questions/4304/… At minimum I think you should identify what the most time consuming step is in your code. Then you can perhaps ask more specifically about that. – PolyGeo♦ Aug 16 '16 at 2:02
  • Thanks for your information. Actually, I don't know what is ArcGIS Runtime tag, but I will take a look at it. When I mentioned the runtime is long, I noticed that the merge step in my code for the folder with a lot of shapefiles (2500+) needed to be merged. – Tianxin Li Aug 16 '16 at 2:49
  • 1
    I think the time taken for the rest of your code will be insignificant in comparison to merging 100+ shapefiles. I think your question should focus on the performance of the Merge tool which would make it a duplicate of gis.stackexchange.com/questions/28244/… – PolyGeo♦ Aug 16 '16 at 2:52
1

os.path is faster than arcpy for filesystem manipulation - use that to search your folders and set up the new folders. To speed up the merge, use append_management. Create the new folder, copy the first shapefile, then append the rest of the shapefiles to that one. Append will also allow you to keep master files that can be added to over time (if needed). Merge has to create a new feature class before merging the data. Append does not do this, so may be slightly faster. The processing time is going to depend on the size of the data.

|improve this answer
  • Thanks for your information. Based on your description, I believe you totally understand my questions and I think you have done something similar before. I will take a look at the grammar of os.path and append_management. I will post further questions as needed for this question. Good luck with your study and work. – Tianxin Li Aug 16 '16 at 2:52
  • gis.stackexchange.com/questions/28244/… suggests that the jury may still be out on Merge/Append performance comparison. I would have guessed Append NO_TEST would be quickest but I cannot recall ever measuring. – PolyGeo♦ Aug 16 '16 at 3:00
  • @Tianxin Li You're welcome. If you find the info useful, could you please upvote my answer. If you post more comments I will improve my answer until it solves the problem. Good luck to you as well. – jbalk Aug 16 '16 at 3:32
  • Absolutely. Thanks for your help. It took me about 10 hours to run the code I put on above. I haven't got a chance to improve the code based on your suggestions, but I will find a time to do that. – Tianxin Li Aug 16 '16 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.