Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I am looking at cataloging some data, and I have code that works, it just seems to take a lot of time to run through all of the items. The way this data is structured is a bit of a hindrance, particularly with the amount of directories I am looking to analyze.

The data structure is similar to this:

Working Folder
 Property Folder (All with a specific label ogl###)
      Geodatabase(all the same name in each different folder)
           Feature Dataset
                3 Shapefiles (boundary, parcel, corner)
           60 Additional Shapefiles of varying types
      Misc. Folders, Tiffs, and geodatabases
 Misc. Folders

This may not have been the best structure to set up our data library, but that is above what I am trying to do, and I have no influence over that.

My goal is to do some quality control on this data, making sure that each Property Folder, Geodatabase, and Feature Class is structured correctly. To do this I need to check each Property Folder\Geodatabase individually, write to excel the names of the corresponding Feature Classes in each database, and then within each feature class the correct fields.

I have written working code to check the existence of the correct feature classes and export to excel the results of that check. To me, however, it seems like it is taking longer than it should. Here is an example of my code to accomplish that (minus the extraneous excel stuff):

import arcpy 

arcpy.env.workspace = r'E:\Working'
working_folder_list=arcpy.ListWorkspaces('ogl*','Folder')           ##Generate list of Property
                                                                    #Folders 
######This block is cataloging a blank reference folder######

arcpy.env.workspace = r'E:\Working\Reference GDB\OGL.gdb\Admin'     ##looking into the reference 
fc_ref_ad=arcpy.ListFeatureClasses('*')                             ##folder Feature Dataset
fc_ref_ad_dic={}
for fc in fc_ref_ad:
    fc_ref_ad_dic[fc]=[f.name for f in arcpy.ListFields(fc)]

arcpy.env.workspace = r'E:\Working\Blank DB 4-05-11\SGL.gdb'        ##looking into the reference
fc_ref=arcpy.ListFeatureClasses('*')                                ##geodatabse
fc_ref_dic={}
for fc in fc_ref:
    fc_ref_dic[fc]=[f.name for f in arcpy.ListFields(fc)]           ##creating a dictionary of fields
                                                                    ##per FC
for fc in fc_ref_ad:                   
    fc_ref.append(fc)                                               ##Merging both lists and dicts

for fc in fc_ref_ad_dic:
    fc_ref_dic[fc]=fc_ref_ad_dic[fc]
fc_ref.sort()
#############################################################


######This block looops through each Property Folder#########

for folder in working_folder_list:
    arcpy.env.workspace = folder + "\\ogl_data0.gdb\\Admin"         ##Listing FC in Feature Dataset
    admin_ds=arcpy.ListFeatureClasses('*')

arcpy.env.workspace = folder + "\\sgl_data0.gdb"                    ##Listing FC in Geodatabase
current_gdb=arcpy.ListFeatureClasses('*')
for fc in admin_ds:
    current_gdb.append(fc)                                          ##Merge the 2 lists

if len(current_gdb) > len(fc_ref):                                  ##Dealing with uneqaul len(lists)
    for j in range(len(current_gdb)-len(fc_ref)):
        fc_ref.append('')

##block of code to write lists to corresponding cells in excel

#############################################################


######This block goes through each FC and lists fields#######
######Approximately 4000 directories(66 Property Folders and 60 FCs######

for folder in working_folder_list:

arcpy.env.workspace = folder + "\\sgl_data0.gdb\\Admin"
fc_cur_ad=arcpy.ListFeatureClasses('*')
fc_cur_ad_dic={}
for fc in fc_cur_ad:
    fc_cur_ad_dic[fc]=[f.name for f in arcpy.ListFields(fc)]

arcpy.env.workspace = folder + "\\sgl_data0.gdb"
fc_cur=arcpy.ListFeatureClasses('*')
fc_cur_dic={}
for fc in fc_cur:
    fc_cur_dic[fc]=[f.name for f in arcpy.ListFields(fc)]

for fc in fc_cur_ad_dic:
    fc_cur_dic[fc]=fc_cur_ad_dic[fc]

##########################################

I know it is a lot to look through, and I may not be the cleanest coder. Truth be told I am still learning this stuff, and most of it I am teaching to my self on the fly. I am just wondering if there is a more efficient way I could be doing this apart from restructuring the data itself. I am aware of the arcpy.da.Walk, but I am unfamiliar with it, and not sure if it would do me any good. Perhaps I am wrong.

Either way, I am hoping someone may be able to give me some tips. I want to thank you in advance for looking.

Edit:

After trying to learn arcpy.da.Walk, I created a sample script that averages the times it takes for each method. I am a bit shocked that after 20 iterations, that the Walk() function took an average of 1 minute longer per run through my data. Here is that script:

import arcpy
import os
import time

worktimes=[]
walktimes=[]

counter=0

while counter <20:
t_start=time.clock()

arcpy.env.workspace = r'E:\Working'
working_folder_list=arcpy.ListWorkspaces('ogl*','Folder')

for folder in working_folder_list:
    arcpy.env.workspace = folder + "\\ogl_data0.gdb\\Admin"
    admin_ds=arcpy.ListFeatureClasses('*')
    for fc in admin_ds:
        print fc
        fields = arcpy.ListFields(fc)
        for f in fields:
            print f.name

    arcpy.env.workspace = folder + "\\ogl_data0.gdb"
    current_gdb=arcpy.ListFeatureClasses('*')
    for fc in current_gdb:
        print fc
        fields = arcpy.ListFields(fc)
        for f in fields:
            print f.name

workspacetime=time.clock()-t_start
worktimes.append(workspacetime)

t_start2=time.clock()
workspace = 'E:\Working'
for dirpath, dirnames, filenames in arcpy.da.Walk(workspace):
    for filename in filenames:
        fields=arcpy.ListFields(os.path.join(dirpath,filename))
        print filename
        for f in fields:
            print f.name

walktime = time.clock() - t_start2
walktimes.append(walktime)
counter+=1

worksum = 0
walksum = 0

for time in worktimes:
worksum += time

for time in walktimes:
walksum += time

workavg = worksum/len(worktimes)
walkavg = walksum/len(walktimes)

print 'Average workspace time: '+str(workavg)
print 'Average walk time: '+str(walkavg)

Is there something I'm missing, or is Walk() not as efficient as I assumed.

The average times were 104 seconds for changing workspaces, 167 using the Walk() function.

share|improve this question

2 Answers 2

If you are using ArcGIS 10.1 SP1 (or later) for Desktop then I have assumed, without any performance testing, that the most efficient way to iterate through folders and geodatabases is to use arcpy.da.Walk.

You mention that you are aware of it in your Question and I recommend that you investigate it to make your code to do this much more concise.

share|improve this answer
    
Would you mind taking a look at my edit. I know the code for Walk() is much more concise, but that process seems to be taking longer than I expected. I'm not sure If there is something I am missing... –  user27542 Mar 3 at 22:22
    
@user27542 I have just upvoted your Question because you now seem to have considered the obvious alternative to your original procedure. Unfortunately, I know it will be quite some time before I am likely to have time to look at your comparison of the two techniques. –  PolyGeo Mar 3 at 22:35
    
Thanks for the input, and no worries on taking the time yourself to look into it. Hopefully someone on here can provide some insight as to why the Walk() function isn't working as well as I had assumed. –  user27542 Mar 3 at 22:42

You can really optimize arcpy.da.walk by limiting your search results. I see you use the following impementation of da.walk:

dirpath, dirnames, filenames in arcpy.da.Walk(workspace)

Rather, try limiting your search results to only tables, for example:

dirpath, dirnames, filenames in arcpy.da.Walk(workspace, datatype = "Table")

To illustrate, the following script searches an entire directory looking for and appending the files to a list. In the first example, only polygon feature classes are specified while in the second example, all files are specified. You can see there is a 2X speed increase by simply limiting the search results:

import arcpy, os, time

workspace = r"C:\gdrive"
files = []

start = time.clock()

# Note that the files are limited to polygon feature classes
for dirpath, dirnames, filenames in arcpy.da.Walk(workspace,
                                                  datatype="FeatureClass",
                                                  type="Polygon"):
    for filename in filenames:
        files.append(os.path.join(dirpath, filename))


print "There are %s featureclasses in the workspace" % len(files)

end = time.clock()
total = end - start

print str(total) + " second to gather only feature classes"

workspace = r"C:\gdrive"
files = []

start = time.clock()

# Note that all files are game here--no filtering...
for dirpath, dirnames, filenames in arcpy.da.Walk(workspace):
    for filename in filenames:
        files.append(os.path.join(dirpath, filename))

print "There are %s featureclasses in the workspace" % len(files)

end = time.clock()
total = end - start

print str(total) + " seconds to gather all files"

enter image description here

share|improve this answer
    
In the file system I was testing the timing script on, only shapefiles and geodatabases were present. Since those are my target files I am going for, I did not do any filtering with the Walk() function. In a live environment I would, but for testing, the results would be the same regardless I think. To give you an idea, this structure has 4 gdb's, each with a feature dataset and 66 total shapefiles. Inside each shapefile there are an average of 10-15 fields. I'll have to do some more code tweaking to get to the bottom of this. –  user27542 Mar 4 at 0:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.