ArcGIS "Merge" tool is slower on arcpy than when starts it with ArcGIS interface

Question

I am writing python script on ArcGIS 10.2.2 and found some strange bug.

I use merge tool to process 498 shape files (they are located on a hard drive). Using arcpy script it takes 6 minutes, but when I run the same tool with the same shape files it takes 6 seconds to complete the task in ArcMAP interface!

The output of the tool goes to in_memory workspace in both situations.

I tried to use copy feature tool to put all shape files into memory before I run merge tool - no luck.

Also I tried to use make feature layer tool and use output files as input to merge tool - no luck again.

How to make this tool works same fast as in ArcMAP interface?

UPGRADE 1:

I use standart ArcGIS 10.2.2 python installation: IDLE 2.7.5, Python version 2.7.5, Tk version 8.5.2.

When I run PolyGeo's script from IDLE interface, it gives:

ArcPy import took 3.96323227335 seconds

Merge took 316.568876475 seconds

Runnig PolyGeo's script from ArcMap's python interface, it gives:

ArcPy import took 9.65935383235e-06 seconds

Merge took 364.072804059 seconds

Running Merge tool with same shapefiles from ArcMAP's interface gives:

Elapsed Time: 6,39 seconds

Answer 1

I think your first troubleshooting step should be to quantify the time taken for your IDE to import ArcPy (which ArcMap already has imported by default) because you will not be able to eliminate that time from your IDE workflow.

You can use a code snippet like this to do that:

import time

start = time.clock()
import arcpy
elapsed = (time.clock() - start)
print("ArcPy import took " + str(elapsed) + " seconds")

start = time.clock()
arcpy.Merge_management(<your_parameters_go_here>)
elapsed = (time.clock() - start)
print("Merge took " + str(elapsed) + " seconds")

If the import arcpy is not the culprit then perhaps it is the number of feature classes. To test that try something similar with perhaps 2, 4, 8, 16, 32, 64, 128, 256 feature classes at a time. This is not expected to show you why 498 is slow but could give you some ideas about whether the time increase as you try to do more feature classes is linear or something else.

Something else that I am curious about is how you place your shapefiles onto the Merge tool dialog. I am thinking that you must be dragging them into your map as layers first, and then dragging the layers into the Merge tool. In which case, for a fair comparison your ArcPy code would need to be using MakeFeatureLayer too.