Merge shapefiles into geodatabase in ArcPy

Question

I am trying to merge a number of shapefiles with the following code:

import arcpy, os, datetime

#Set working directory
cd = r"C:\Users\..."
arcpy.env.workspace = cd
os.chdir(cd)
arcpy.env.overwriteOutput = True

directory = cd + "/shapefiles"
shapefiles = []
for root, dirs, files in os.walk(directory):
    files = [ fi for fi in files if fi.endswith(".shp") ]
    for filename in files:
        shapefile = (os.path.join(directory, filename))
        shapefiles.append (shapefile)


arcpy.Merge_management(shapefiles, cd + "all_districts.shp")

This fails due to Error 001156 which arises because some fields in some shapefiles exceed the maximum text length. According to this question, it is not possible to increase the maximum text length and the answers suggest that one should merge the shapefiles within a geodatabase.

I am new to geodatabases, so I do not know how to do this. How can I merge all shapefiles from my folder into a geodatabase?

Answer 1

I think the problem you're seeing is that the first feature class acts as a schema template for all the following feature classes in the Merge tool field map. Suppose you have two feature classes, each with a text field named the same, but one is longer than the other. If the first feature class contains the longer field, there is no problem, since the shorter field will fit within. However, if the first feature class contains the shorter field, you will get error 001156, since the longer field will not fit within the schema of the shorter field.

There is probably a shorter answer than this (I'm not great with the field mapping(s) objects), but you can get around your problem by changing the field map before executing Merge:

>>> fcs = ['len_5','len_10'] # feature classes
... out = r'in_memory\output' # output
... field_maps = [] # container for all field map objects
... fms = arcpy.FieldMappings() # field mappings object
... for field in arcpy.ListFields(fcs[0],field_type='String'): # get 'template' string fields from first feature class
...         fm = arcpy.FieldMap() # make field mapping
...         fm.addInputField(fcs[0],field.name) # add field to field mapping
...         f_name = fm.outputField # get field mapping field
...         f_name.name = field.name # set field name
...         f_name.length = 255 # set field length
...         fm.outputField = f_name # set field mapping field
...         field_maps.append(fm) # add to field mapping list
... for fc in fcs[1:]: # loop through rest of feature classes
...     for field in arcpy.ListFields(fc,field_type='String'): # loop through string fields
...         for field_map in field_maps: # look through field maps
...             if field_map.outputField.name == field.name: # find right field map
...                 field_map.addInputField(fc,field.name) # add the field to the corresponding field map object
... for field_map in field_maps: # loop through field map objects
...     fms.addFieldMap(field_map) # add each field map to the field mappings object
... arcpy.Merge_management(fcs,out,fms) # execute Merge, using field map

Answer 2

I haven't tried it, but I would suggest to copy the shapefiles into the gdb first and than merge the feature classes. That doesn't solve the problem. You have to go for fieldmappings, or create a new feature class and read all the other rows and add it to the new feature class. As I find it always difficult to handle fieldmappings in python I show the second way here:

import arcpy, os

cd = r"C:\Users\..."
arcpy.env.workspace = cd
gdb = "mygdb.gdb"
arcpy.CreateFileGDB_management(cd,gdb)
shapefiles = arcpy.ListFeatureClasses()


spatial_reference = arcpy.Describe(shapefiles[0]).spatialReference
geometry_type = arcpy.Describe(shapefiles[0]).shapeType

arcpy.CreateFeatureclass_management (cd+os.sep+gdb,"semifinal_fc",geometry_type,"","","",spatial_reference)
fieldlist = []
fieldtypelist = []
for fc in shapefiles:
fields = arcpy.ListFields(fc)
for field in fields:
    if field.name not in fieldlist:
        fieldlist.append(field.name)
        if field.type in("TEXT","FLOAT","DOUBLE","SHORT","LONG","DATE","BLOB","RASTER","GUID","String","Integer","Double","Single"):
            if field.type == "String":
                fieldtype = "TEXT"
            elif field.type == "Integer":
                fieldtype = "SHORT"
            elif field.type == "Double":
                fieldtype = "DOUBLE"
            elif field.type == "Single":
                fieldtype = "DOUBLE"
            arcpy.AddField_management(cd+os.sep+gdb+os.sep+"semifinal_fc",field.name,fieldtype)



for fc in shapefiles:
    arcpy.CopyFeatures_management(fc,cd+os.sep+gdb+os.sep+fc.split(".")[0])

arcpy.env.workspace = cd+os.sep+gdb
fcs = arcpy.ListFeatureClasses()
fcs.remove("semifinal_fc");
fcs.insert(0,"semifinal_fc")

output = "final_fc"
arcpy.Merge_management(fcs,output)

for fc in fcs :
    arcpy.Delete_management(fc)

Answer 3

The field content should not get longer after a merge and they obviously seem to work in the existing shapefiles. I Think something else is wrong.

Try this:

import arcpy,glob,os
arcpy.env.overwriteOutput=1

#Change paths and names below
shapefile_folder=r'C:\Test\shapefiles'
output_folder=r'C:\Test'
output_database=r'database.gdb'
output_featureclass_name=r'all_districts'

arcpy.CreateFileGDB_management(out_folder_path=output_folder, out_name=output_database)
shapelist=glob.glob(shapefile_folder+r'\\*.shp')
arcpy.Merge_management(inputs=shapelist, output=os.path.join(output_folder,output_database,output_featureclass_name))