I'm currently running a script that is using the arcpy.Merge_management function to merge roughly ~1500 shapefiles together.

When I tried this with much smaller test runs, it ran perfectly.

Now that I'm running all the files it is taking a VERY long time.

I let it run over the weekend, and it is not even half way complete, even though it is still running.

Does anyone have any suggestions on how I could possibly speed up the process? Or would able to suggest another method to merge the files together in a timely manner?

All the files being merged have the exact same schema. None of the files have any more than 150 records. The majority of the files being between 1-20 records.

Here is the code snippet of this part of the script:

# Set the directory and name for the fully merged MESQ_Streets File

outputfile = "Q:\GIS\Admin\Shapefiles\Streets\MESQ_STREETS_SINGLELINE\MESQ_STREETS_SINGLE_LINE.shp"

# This merges the Roads back together

workspace = "Q:\GIS\Admin\Shapefiles\Streets\MESQ_STREETS_SINGLELINE\Rejoined\\"
arcpy.env.workspace = workspace
     arcpy.Merge_management(arcpy.ListFeatureClasses(), outputfile)
except Exception as e:
     print "This did not work", e

closed as unclear what you're asking by PolyGeo Aug 27 '15 at 5:24

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    How many features do you have in each shapefile? You can try using arcpy searchCursor and InsertCursor to reduce the overhead of running each geoprocessing session. Additionally you can try FME. – dexgecko Aug 24 '15 at 15:45
  • 1
    Are these shapefiles part of a GP workflow? Are you utilizing 64-bit background geoprocessing? – Paul Aug 24 '15 at 16:16
  • 3
    Another consideration is your final dataset, how big will it be? If you are merging 15K datasets each with thousands of rows and many fields your final dataset may exceed the design limitations of a shapefile. I've always thought 2GB was some magic limit. It may be more prudent to merge into a file geodatabase featureclass? You need to tells more info about your data as asked by @dexgecko. – Hornbydd Aug 24 '15 at 16:23
  • 2
    Do the shapefiles all have the same schema? maybe try setting up an empty dataset with the same schema inside a file geodatabase, and then append the shapefiles to that, with the schema type option set to No Test – Adam Aug 24 '15 at 23:57
  • 1
    Try merging the data into an in_memory workspace – MapMan Aug 25 '15 at 23:18