3

I'm having an issue where calling arcpy.ListFeatureClasses() in a workspace with 400 shapefiles takes a very long time (more than 10 minutes).

Using the os module to achieve the same effect takes less than a second:

import os
allSingleNonRoadPolys = []
for file in os.listdir(r'D:\data\allSingleNonRoadPolys'):
if file.endswith(".shp"):
    print(file)
    allSingleNonRoadPolys.append(file)

You can then achieve the same effect as ListFeatureClasses by simply prepending the workspace to the filename, each time you need to use a dataset, as seen here:

for shp in allSingleNonRoadPolys:
    arcpy.someTool(workspace + '\\' + shp)

Why would I even keep using ListFeatureClasses() and what makes it take so long? Does it do anything incredibly useful that cannot be achieved in any other way?

  • 1
    Is that a network drive? – Jason Scheirer Aug 25 '15 at 22:00
  • Just working on my D: drive; not linked to any network. Will edit the question to make the path a little easier to read. – BritishSteel Aug 25 '15 at 22:03
  • ListFeatureClasses() is especially good at listing feature classes in an Esri geodatabase. You may want to look into the glob module too, which has very nifty (and efficient) ways of listing data: shps = glob.glob(r'D:\data\allSingleNonRoadPolys\*.shp') – Aaron Aug 25 '15 at 22:27
  • 2
    400 shapefiles is 1600 files to open and validate. A slow or badly fragmented drive can make things worse. – Vince Aug 25 '15 at 22:27
  • 1
    @Aaron arcpy.da.Walk too – Jason Scheirer Aug 26 '15 at 5:37
3

Are you sure it is arcpy.ListFeatureClasses() that takes such a long time? Could it be some other piece of code? Verify with the profiler with just a dummy os.time as shown here.

On the SSD disk (2 years old, was heavily used daily), the arcpy.ListFeatureClasses() returns the list of ~800 shapefiles found in the folder specified in less than 5 secs (just tested). The total size of the folder is ~6GB. I thought that if you have large shapefiles (more than 100MB each and your folder is 40GB), then it could have slown down the run. Yet the run time is the same for a folder of ~40GB with a hundred of shapefiles and the performance was identical to what I've observed with the larger number of shapefiles of smaller size.

Remember that you can limit your search by name, feature type, and optional feature dataset. This is helpful if you don't need to list all of your feature classes (shapefiles).

Why would I even keep using ListFeatureClasses()

You would need to use arcpy.ListFeatureClasses() to work through any Esri workspace (such as a file or a multiuser geodatabase). Of course you are free to use any other methods for listing your shapefiles (as this is an open file format to interact with), but you will need to get back to this function when working with anything stored in an Esri repository later on.

  • I will have to test it again this evening, but I am sure that it is the listing function that takes so long, as I created a single script that simply sets a workspace and then lists the files. I added a print statement after each command, and the workspace print statement is returned quickly whereas the listing one takes forever. Glad to hear that it works fine in your example, so I know that the problem might be linked to the computer (which is weird, as it was newly installed and is super fast in general). I will have to ask my boss about that. – BritishSteel Aug 26 '15 at 8:29
  • Another question: would it be advised to work with geodatabase feature classes instead of shapefiles? It generally would be faster, right? – BritishSteel Aug 26 '15 at 8:30
  • I wouldn't say that it would be faster with feature classes in a gdb comparing to shapefiles. The performance difference should be neglible (seconds, not minutes as in your case). – Alex Tereshenkov Aug 26 '15 at 8:34
  • Could it be an antivirus that scans the folder while you are reading it? Try to isolate possible problems one by one. – Alex Tereshenkov Aug 26 '15 at 8:45
  • Listing ~150 feature classes in a file gdb of 6GB size took ca 4 secs for me; as I said, we are talking about seconds, definitely not minutes. – Alex Tereshenkov Aug 26 '15 at 8:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.