Appending features in file geodatabase using ArcPy

Question

I am quite new to the use of ArcPy but am using it instead of ArcGIS Modelbuilder as I thought it might speed up my workflow. I have a large set of polygon features in a geodatabase (around 12.000), which I want to merge/append into one single feature layer. columnnames and data types are all the same for each file.

My script is working, but my problem is that the processing seems to gradually go slower and after around 2000 features really takes a while before moving forward.

Currently using ArcPy for my script and append query, but open to try other scripts like ogr2ogr or gdal if those are better.

I am using the following script:

import arcpy
import os
from arcpy.sa import *
from sys import argv

arcpy.env.workspace = "c:/workspace"
out = "c:/workspace/out"
schemaType = "NO_TEST"
fieldMappings = ""
subtype = ""

featureclasses = arcpy.ListFeatureClasses(wild_card="NFS_PolySpatialJoin_*", feature_type='Polygon')

for fc in featureclasses:
    print(fc)
    arcpy.Append_management(fc, out, schemaType, fieldMappings, subtype)

Answer 1

Assuming all fields in all feature classes are the same, I suspect a data access insert cursor will be faster than the append tool.

List your fields and then use a search/insert cursor combo.

flds = [f.name for f in arcpy.ListFields (out)] ##get fields
flds += ["SHAPE@"] ##get geometry

with arcpy.da.InsertCursor (out, flds) as iCurs:
    for fc in featureclasses:
        print(fc)
        with arcpy.da.SearchCursor (fc, flds) as sCurs:
            for row in sCurs:
                iCurs.insertRow (row)

Answer 2

Your code is almost there but I can see a glaring problem with it...

Look at the help file for the Append tool what does it take as input? You need to get smarter in understanding the help file. The parameter section (what's called syntax in the ArcMap version) is telling exactly what the tool takes.

This tool takes a LIST of datasets.

Now look at your code, you create a list called featureclasses which is what you want to feed into the append tool. Instead of feeding the LIST into the tool you step through the list and append one at a time, which as you are discovering is horrendously inefficient!

Replace your for loop with simply this:

arcpy.Append_management(featureclasses, out, schemaType, fieldMappings, subtype)

Now you are providing a list of featureclass as input to append to write to out, which by the way would create a coverage format, if that's what you really wanted otherwise I would write to an existing file geodatabase featureclass.