1

Looping through shapefile using arcpy.ListFeatureClasses() is very slow for many shapefiles. because first thing first, it writes all in memory as a list and then read each shapefile one by one. Is there any other way to read one by one shapefile without making them as a list and write it in the memory:

This is the slow script:

arcpy.env.workspace = shapefile_folder
fc_list = arcpy.ListFeatureClasses()
for shapefile in fc_list:
     print (fc) # doing the processing over shp here
Improve this question
13
  • 2
    ListFeatureClasses doesn't read the shapefiles, but it does need to verify the .shx/.dbf in parallel to the .shp.
    – Vince
    Commented Sep 25, 2020 at 14:09
  • @vince it takes very long time to read all shapefile in a folder using it . any suggestion?
    – Anahita Kp
    Commented Sep 25, 2020 at 14:14
  • 1
    I have never experienced listfeatureclasses to be slow, unless maybe when the files are on a network drive. How do you know the listing is what takes a long time and not your processing?
    – Bera
    Commented Sep 25, 2020 at 14:16
  • 1
    @Vince Do you have any references to support the claim that filesystems are not designed for 10's of thousands of entries in a single directory? This is counter to what I have seen, although I am certainly open to changing my mind.
    – Aaron ♦
    Commented Sep 25, 2020 at 15:16
  • 1
    I too have observed reduced performance in arcmap or arccatalog when there are thousands of shapefiles in a single folder. You should avoid that by storing them in some sensible sub-folder structure. Bjorn's answer below would work but as he says you loose the inbuilt validation that @Vince points out comes with listfeatureclasses(). Basically you need to be smarter with data management rather than dumping everything into a single folder
    – Hornbydd
    Commented Sep 25, 2020 at 22:13

1 Answer 1

Reset to default
2

You can get a very quick list of the shapefiles in a folder with a command like filter(lambda x: x.lower().endswith('.shp'), os.listdir(<dir>)). As some of the comments imply, this will not test to ensure that all of the results refer to complete & readable shapefiles though.

Improve this answer
1
  • I support this because python's strength in os.listdir and lambda functionality. Obviously you have to later test to ensure the validity of the shapefile, but that can be done inside the for looping.
    – JasonInVegas
    Commented Sep 26, 2020 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.