3

I am quite new to the use of ArcPy but am using it instead of ArcGIS Modelbuilder as I thought it might speed up my workflow. I have a large set of polygon features in a geodatabase (around 12.000), which I want to merge/append into one single feature layer. columnnames and data types are all the same for each file.

My script is working, but my problem is that the processing seems to gradually go slower and after around 2000 features really takes a while before moving forward.

Currently using ArcPy for my script and append query, but open to try other scripts like ogr2ogr or gdal if those are better.

I am using the following script:

import arcpy
import os
from arcpy.sa import *
from sys import argv

arcpy.env.workspace = "c:/workspace"
out = "c:/workspace/out"
schemaType = "NO_TEST"
fieldMappings = ""
subtype = ""

featureclasses = arcpy.ListFeatureClasses(wild_card="NFS_PolySpatialJoin_*", feature_type='Polygon')

for fc in featureclasses:
    print(fc)
    arcpy.Append_management(fc, out, schemaType, fieldMappings, subtype)
3
  • 12k features isn't a great many. I've found that performance improved by dropping the spatial index before Append over 100k features (then rebuilding the index after), when adding 1k, 10k,100k, and 1m features to an existing table of 1m features. Of greater importance was making sure that the target feature class was on a local file-system, not a network-shared one.
    – Vince
    Aug 17, 2021 at 17:24
  • @Vince Thanks for your sharing. How would I drop my spatial index in advance before doing the Append? Do you also recommend using a for loop? or simply use the .ListFeatureClasses function? My target feature class is in the same directory as the geodatabase, both on a local file-system.
    – PGerrits
    Aug 17, 2021 at 17:57
  • 1
    RemoveSpatialIndex_management, followed by AddSpatialIndex_management after the Append(s)
    – Vince
    Aug 17, 2021 at 22:04

2 Answers 2

5

Assuming all fields in all feature classes are the same, I suspect a data access insert cursor will be faster than the append tool.

List your fields and then use a search/insert cursor combo.

flds = [f.name for f in arcpy.ListFields (out)] ##get fields
flds += ["SHAPE@"] ##get geometry

with arcpy.da.InsertCursor (out, flds) as iCurs:
    for fc in featureclasses:
        print(fc)
        with arcpy.da.SearchCursor (fc, flds) as sCurs:
            for row in sCurs:
                iCurs.insertRow (row)
2
  • 1
    thank you. Interesting and different approach, will try it out.
    – PGerrits
    Aug 17, 2021 at 23:00
  • 1
    This was definitely the fastest processing. It has just finished the entire list in 30 minutes. Thanks so much, you have just saved me several hours/days with this.
    – PGerrits
    Aug 17, 2021 at 23:30
4

Your code is almost there but I can see a glaring problem with it...

Look at the help file for the Append tool what does it take as input? You need to get smarter in understanding the help file. The parameter section (what's called syntax in the ArcMap version) is telling exactly what the tool takes.

This tool takes a LIST of datasets.

Now look at your code, you create a list called featureclasses which is what you want to feed into the append tool. Instead of feeding the LIST into the tool you step through the list and append one at a time, which as you are discovering is horrendously inefficient!

Replace your for loop with simply this:

arcpy.Append_management(featureclasses, out, schemaType, fieldMappings, subtype)

Now you are providing a list of featureclass as input to append to write to out, which by the way would create a coverage format, if that's what you really wanted otherwise I would write to an existing file geodatabase featureclass.

2
  • @ Hornbydd Thanks very much for your answer. I will try that again and have more patience this time. I initially did as you suggested, but prefered using the for loop because it provides a print for each row. Can it be that feeding in a LIST into the tool takes more time? I am sure I have waited more than an hour before.
    – PGerrits
    Aug 17, 2021 at 19:04
  • 1
    Why don't you just print the list then call the append tool? Feeding each featurclass into the append tool one at the time is very inefficient as you are calling the tool each time when it needs to be called only once. There is a cost in time when any tool is called.
    – Hornbydd
    Aug 17, 2021 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.