Understanding why arcpy.CreateScratchName() gets slower with more file geodatabases in folder?

Question

In my ArcGIS Pro 1.4.1 application, I noticed that a short section of ArcPy code that creates a scratch name, and then uses that to create a file geodatabase (always in the same folder with the same root name), goes from taking a few seconds the first time it is run to taking 2 minutes around the 30th time.

I have isolated that it is CreateScratchname rather than CreateFileGDB that is slowing down by running this code:

import arcpy,time,os

workingFolder = r"C:\Temp\TestFolder"
for i in range(100):
    scratchName = arcpy.CreateScratchName("Working",".gdb","Workspace",workingFolder)
    baseName = os.path.basename(scratchName.split(".")[0])  
    workingGDB = r"{0}\{1}.gdb".format(workingFolder,baseName)
    start = time.clock()
    arcpy.CreateFileGDB_management(workingFolder,baseName)
    elapsed = (time.clock() - start)
    print("Creating working geodatabase {0} took {1} seconds".format(workingGDB,int(round(elapsed))))

to create this output where you will see that the time for CreateFileGDB is consistently 1 second whereas the time for CreateScratchName is 3 seconds for the first time, then climbs from 1 second to 4-5 seconds by the 25th iteration:

Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> 
======== RESTART: C:\CompilationPlots\Scripts\CreateWorkingGDBtest.py ========
Creating scratchname C:\Temp\TestFolder\Working0.gdb took 4 seconds
Creating working geodatabase C:\Temp\TestFolder\Working0.gdb took 3 seconds
Creating scratchname C:\Temp\TestFolder\Working1.gdb took 0 seconds
Creating working geodatabase C:\Temp\TestFolder\Working1.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working2.gdb took 0 seconds
Creating working geodatabase C:\Temp\TestFolder\Working2.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working3.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working3.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working4.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working4.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working5.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working5.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working6.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working6.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working7.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working7.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working8.gdb took 1 seconds
Creating working geodatabase C:\Temp\TestFolder\Working8.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working9.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working9.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working10.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working10.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working11.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working11.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working12.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working12.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working13.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working13.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working14.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working14.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working15.gdb took 2 seconds
Creating working geodatabase C:\Temp\TestFolder\Working15.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working16.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working16.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working17.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working17.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working18.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working18.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working19.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working19.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working20.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working20.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working21.gdb took 3 seconds
Creating working geodatabase C:\Temp\TestFolder\Working21.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working22.gdb took 4 seconds
Creating working geodatabase C:\Temp\TestFolder\Working22.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working23.gdb took 5 seconds
Creating working geodatabase C:\Temp\TestFolder\Working23.gdb took 1 seconds
Creating scratchname C:\Temp\TestFolder\Working24.gdb took 4 seconds
Creating working geodatabase C:\Temp\TestFolder\Working24.gdb took 1 seconds
>>> 

The above would not be such a concern except that in my application I go on to fill the working geodatabase with about 50 feature classes and tables at each iteration, and this seems to make CreateScratchName run even slower.

Is the slowing down of CreateScratchName demonstrated in my code above, and anecdotally exacerbated by the size of each file geodatabase once they have been filled, to be expected?

Answer 1

I just ran into this myself. I would imagine that it's because ArcGIS Pro has to scan the specified directory every time CreateScratchName is called to determine the next unique name. And I agree that adding data to those GDBs slows CreateScratchName down even more, the more data you add to each GDB, the slower it gets... I'm not sure why, perhaps ArcGIS recurses into those GDBs for some reason.

An alternative to CreateScratchName might be to determine a unique GDB name yourself, with the uuid module or determining the next sequential number.

Here's a plot of the time taken to call CreateScratchName as the number of GDBs increases x the number of feature classes in each GDB:

# Tested with ArcGIS Pro 2.8 running in Visual Studio Code

import arcpy
import os
import time
import pandas as pd
import matplotlib.pyplot as plt

workingFolder = r"D:\Temp\TestFolder"

n_gdbs = 25
n_fcs = 25
step = 5

data = []

for n_fc in range(0, n_fcs+1, step):

    if os.path.exists(workingFolder):
        arcpy.management.Delete(workingFolder)
    os.mkdir(workingFolder)
    
    last_gdb = None
    for i in range(0, n_gdbs+1, step):
        start = time.perf_counter()
        scratchName = arcpy.CreateScratchName("Working",".gdb","Workspace",workingFolder)
        elapsed = (time.perf_counter() - start)
        data += [{"featureclasses": n_fc, "gdbs": i, "time_elapsed": elapsed}]

        print(f"featureclasses: {n_fc}, gdbs: {i}, time_elapsed: {round(elapsed, 2)}")

        baseName = os.path.basename(scratchName.split(".")[0])  
        workingGDB = os.path.join(workingFolder,baseName)

        if last_gdb is None:  # Create the GDB / FCs
            last_gdb = scratchName
            arcpy.CreateFileGDB_management(*os.path.split(scratchName))
            if n_fc > 0:
                for j in range(n_fc):
                    arcpy.CreateFeatureclass_management(scratchName, f"feature_class{j}", "POLYGON", spatial_reference=4326)
        else:  # Copy the GDB just to speed things up
            arcpy.management.Copy(last_gdb, scratchName)

    arcpy.management.Delete(workingFolder)  # Clean up before starting again

df = pd.DataFrame(data)

groups = df.groupby('featureclasses')
fig, ax = plt.subplots()
ax.margins(0.05)
for name, group in groups:
    ax.plot(group.gdbs, group.time_elapsed, '-', label=name)
ax.legend(title='N FeatureClasses')
plt.xlabel('N GDBs')
plt.ylabel('Time elapsed (s)')

plt.show()