2

I'm trying to automate a workflow that currently costs a co-worker about 45 minutes. General run down is this person copies files from a network drive onto a local one, and process it through four separate ArcGIS models, the code below does not comprise all of the geoprocesses that are run against these files, but the point in which the bog down is happening. As it stands, each model takes about 12 minutes to run. The code below works, but it literally takes about an hour and a half on the same exact data. I'm using arcpy.da.Walk because I figured I could save all the clicks and just run it in the background. Pretty new to coding, but been doing it about 2 years - could it be the fnmatch? My gut tells me it's the walk, and building the list - wondering if there is an alternative.

I have attempted to replicate the workflow (copying the directories locally, and not over the network).

def main():
    try:
        import arcpy, os, sys, fnmatch, traceback
        arcpy.env.overwriteOutput = True
        rootDir = arcpy. GetParameterAsText(0)
        outDir = arcpy.GetParameterAsText (1)
        mssn = '"12"'
        sensor = '"Geo"'
        dissolveFld = ["Date"]
        search = "*pan*.shp"
        outName = "test"

        shpMergeLst = []
        for root, dir, files in arcpy.da.Walk(rootDir):
            for filename in fnmatch.filter(files, search):
                shpMergeLst.append(os.path.join(root, filename))

        if shpMergeLst:
            outShp = os.path.join(outDir, os.path.basename(outName[0][:-4]) + "_Merge.shp")
            arcpy.Merge_management(shpMergeLst, outShp)

Don't have system specs, but currently running ArcGIS 10.1 Advanced

  • 1
    Please edit the question: Have you tried measuring the timing at each point in your code flow? What is the directory fragmentation of the "rootDir" parameter location? Please confirm that the rootDir is not a network drive. – Vince Dec 8 '15 at 20:27
  • 1
    Is there an except to your try? Perhaps there is a hang up with something that waits for a long time then throws an error. Also, it looks like you're running it as a tool. Do you have 64-bit background processing installed and are running it in the background? Depending on the size of the files, you may be able to load them all into memory to perform the mergers as it may improve performance. – Branco Dec 8 '15 at 20:43
  • Thanks for the response: There is an exception, which is just the traceback information for debugging purposes. It is not a 64 bit background, the files are small, but there is a convoluted directory tree with hundreds of thousand of files. 'in_memory' has been attempted, and it doesn't make a difference, apparently there are arc tools that have to be hard written for processing, in my research... multiple geoprocesses on a merge process is one of them (although I can't source it, and can't confirm that validity). – geodranic Dec 8 '15 at 23:17
  • Ran some timing code.... it's definitely hemmed at arcpy.da.Walk and fnmatch filtering – geodranic Dec 9 '15 at 0:19
6

I agree with @Paul above - the best way to optimise this code is to ensure that it never runs. If you're happy to install third party packages I'd also take a look at the the formic package, which is a

... Python implementation of Apache Ant FileSet and Globs including the directory wildcard **

(Credit to this answer on SO for pointing me at the library)

In the same vein as the answer above you can use formic to build your file matcher like so:

import formic

t3a = time.time()
res3 = list(formic.FileSet(include="*pan*.shp", exclude=["**/*.gdb/"]))
t3b = time.time()
print("Formic walker:\t\t{}".format(t3b-t3a))

assert(res1==res2==res3)

And on my machine I get the three results for times:

Native walker:    333.040999889
Optimized walker: 277.563000202
Formic walker:    0.770999908447

Do note that the substantial difference comes from Formic not checking whether or not the file is actually a valid shapefile (and this certainly won't work for geodatabases). The idea behind this is that it's Easier to Ask for Forgiveness than Permission. Still, that doesn't always hold out with ArcGIS, so you may find it easier to filter the function as the paths are generated:

def check_formic(maindir, searchpat):
    for path in formic.FileSet(
            include=searchpat, exclude=["**/*.gdb/"],
            directory=maindir
    ):
        if arcpy.Describe(path).dataType == "ShapeFile":
            yield path

t4a = time.time()
res4 = list(gen4)
t4b = time.time()
print("Formic with type checking:\t\t{}".format(t4b-t4a))

Which gives:

Formic with type checking:              30.4530000687

Still faster than using the arcpy.da.Walk method by a significant margin.

Note All my tests were done in a folder structure with 43 shapefiles in various places.

  • Goodness, that's a very significant speedup! I'll have to test it on my directory tomorrow. – Paul Dec 9 '15 at 6:02
  • @Paul Yeah, it works well, assuming it's only shapefiles you're after. I suspect as soon as you had to start trawling for datasets in file and personal geodatabases or SDE connections you'd have to write a lot of boilerplate, and lose a lot of the time savings there. – om_henners Dec 10 '15 at 5:03
5

Given your parameters, remember that The fastest code is the code that never runs!

Unless your folder hierarchy is very complex, chances are there are some directories you know you can skip. For example, my D: drive, which houses quite a bit of data (~40k files, 1.3k folders, 240GB), contains mostly file geodatabases. We don't need to search those directories, so let's skip them. Also, you might have some data (PDFs, word documents, images, etc) which the walker will also traverse, so let's restrict it to only FeatureClasses. Finally, since you are passing the list of files to Merge, all the shapefiles must be of the same type, so let's look at only Points.

import arcpy
import os
import fnmatch
import time

maindir = r"D:"

def walker(maindir, searchpat):
    for root, dirs, files in arcpy.da.Walk(maindir):
        for f in fnmatch.filter(files, searchpat):
            yield os.path.join(root, f)

def walker_optim(maindir, searchpat):
    for root, dirs, files in arcpy.da.Walk(maindir, datatype='FeatureClass', type="Point"):
        # Modify dirs in place to skip file GDBs
        dirs[:] = [d for d in dirs if not d.endswith(".gdb")]
        for f in fnmatch.filter(files, searchpat):
            yield os.path.join(root, f)


gen1 = walker(maindir, "*pan*.shp")
gen2 = walker_optim(maindir, "*pan*.shp")

t1a = time.time()
res1 = list(gen1)
t1b = time.time()
print("Native walker:\t\t{}".format(t1b-t1a))
time.sleep(15)

t2a = time.time()
res2 = list(gen2)
t2b = time.time()
print("Optimized walker:\t\t{}".format(t2b-t2a))

assert(res1==res2)

Which gives us:

Native walker:          48.3669998646
Optimized walker:       10.4809999466
  • This will absolutely be implemented in the final code optimization. However, I should clarify, my tests have been run against a single folder with no subdirectories - and the timing results have been dismal compared to the 4 aforementioned models. The directory structure will be constant, so this was absolutely helpful for optimizing, but still cant find the hem. I can watch the merge file grow incrementally, but it's at about 6kb/s totalling around a 34mb shapefile (and yes, has to be shapefile). – geodranic Dec 9 '15 at 0:41
  • 1
    @StevenFerronato, just how many files are in this single folder? It would be useful, I think, if you updated your OP with some information about the folder hierarchy, size, number of files, timing, etc. – Paul Dec 9 '15 at 0:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.