Python Script to convert a mass of KMLs to a single fGDB

Question

I have a mass of KML files, approx 350+ that I need in a single fGDB. I lifted a string of code from ESRI help to attempt to do this - I got it working to the point of creating all the individual gGDBs (one per KML) but its failing with an invalid character in a variable somewhere. Below is the code, and then the error that it throws out. I've added some print commands to show whats in the variables right before it fails. Help? Also, this is my first time trying to use python for actual work, so I apologize in advance for n00bish mistakes.

# Name: BatchKML_to_GDB.py
# Description: Converts a directory of KMLs and copies the output into a single fGDB.
#              A 2 step process: first convert the KML files, and then copy the featureclases

# Import system models
import arcpy, os

# Set workspace (where all the KMLs are)
 arcpy.env.workspace="C:\\Users\\mperez\\Desktop\\IT-Paths"

# Set local variables and location for the consolidated file geodatabase
outLocation = "V:\\2155\\active\\215510456\\GDB"
MasterGDB = r'AllKMLLayers.gdb'
MasterGDBLocation = os.path.join(outLocation, MasterGDB)

# Create the master FileGeodatabase
 arcpy.CreateFileGDB_management(outLocation, MasterGDB)

# Convert all KMZ and KML files found in the current workspace
 for kmz in arcpy.ListFiles('*.KM*'):
  print "CONVERTING: " + os.path.join(arcpy.env.workspace,kmz)
  arcpy.KMLToLayer_conversion(kmz, outLocation)


# Change the workspace to fGDB location
arcpy.env.workspace = outLocation

# Loop through all the FileGeodatabases within the workspace
wks = arcpy.ListWorkspaces('*', 'FileGDB')
# Skip the Master GDB
wks.remove(MasterGDBLocation)

for fgdb in wks:  

  # Change the workspace to the current FileGeodatabase
  arcpy.env.workspace = fgdb    

  # For every Featureclass inside, copy it to the Master and use the name from the original fGDB
  featureClasses = arcpy.ListFeatureClasses('*', '', 'Placemarks')
  for fc in featureClasses:
    print r"COPYING: " + fc + r" FROM: " + fgdb 
    fcCopy = fgdb + os.sep + 'Placemarks' + os.sep + fc
    print r"fcCopy = " + fcCopy
    print r"MasterGDBLocation = " + MasterGDBLocation
    print r"fgdb = " + fgdb
    arcpy.FeatureClassToFeatureClass_conversion(fcCopy, MasterGDBLocation, fgdb[fgdb.rfind(os.sep)+1:-4])


# Clean up
del kmz, wks, fc, featureClasses, fgdb

Output from shell showing error:

COPYING: Points FROM: V:\2155\active\215510456\GDB\canal-move-2-golden-gate-blvd.gdb
fcCopy = V:\2155\active\215510456\GDB\canal-move-2-golden-gate-blvd.gdb\Placemarks\Points
MasterGDBLocation = V:\2155\active\215510456\GDB\AllKMLLayers.gdb
fgdb = V:\2155\active\215510456\GDB\canal-move-2-golden-gate-blvd.gdb

Traceback (most recent call last):
  File "C:\Users\mperez\Desktop\batch_kml_to_gdb.py", line 46, in <module>
    arcpy.FeatureClassToFeatureClass_conversion(fcCopy, MasterGDBLocation, fgdb[fgdb.rfind(os.sep)+1:-4])
  File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\conversion.py", line 1547, in FeatureClassToFeatureClass
    raise e
ExecuteError: ERROR 000354: The name contains invalid characters
Failed to execute (FeatureClassToFeatureClass).

Answer 1

In the traceback you can see where the error in the code was raised (the last line), and every function in the call leading to that error - just go backwards up the list.

From this you can see that the error was in line 46 of your code;

arcpy.FeatureClassToFeatureClass_conversion(fcCopy, MasterGDBLocation, fgdb[fgdb.rfind(os.sep)+1:-4])

You know the third argument in Feature Class to Feature Class is the feature class name. Your first step should be to take a look at os.path.basename and os.path.splitext for cleaner ways to get the name.

Your second step should be to look at ValidateTableName in arcpy, which when you pass it the proposed name and workspace, will return a valid name.

Answer 2

This is not meant to be an answer, just a contribution (taken from my comment on om_henners answers)

import arcpy
invalidcharacters = """`~@#$%^&()-+=,{}.![]; /\:*?'<>|"""
valid = arcpy.ValidateTableName(invalidcharacters)
print valid

Output:

______________________________

(that means every character in `~@#$%^&()-+=,{}.![]; /\:*?'<>| is an invalid character)

Answer 3

I know this is a very old question by now, but here is a script I've put together that can deal with a bunch of corner cases of the KML/KMZ to GDB conversion process.

I hope someone else finds this useful!

Btw, not sure if this is obvious, but it might be worth mentioning that the code below was written for Python 3.

# This script was heavily modified from the script available in the ArcGIS Pro documentation:
# https://pro.arcgis.com/en/pro-app/latest/tool-reference/conversion/kml-to-layer.htm
# Also includes information from the following GIS Stack Overflow question: 
# https://gis.stackexchange.com/questions/55259/python-script-to-convert-a-mass-of-kmls-to-a-single-fgdb?noredirect=1&lq=1

# Converts a directory of KMZs/KMLs and copies the output into a single GDB

# Import system modules
import arcpy
import os
import pathlib
import string

def export_multiple_kmzs_to_single_gdb(kmz_folder, output_gdb, temp_folder):
    '''
    Function that reads an input folder with multiple KMZ/KML files and exports
    all the geometric data contained in those files into a single GDB file.
    
    Parameters
    ----------
    kmz_folder : str
        The input folder that contains all the KMZs/KMLs 
        that will be converted
    output_gdb : str
        Path for the consolidated GeoDataBase (.gdb) file
    temp_folder : str
        Temporary folder where multiple GDB and LRYX files will be generated
        and later deleted.
    
    Returns
    -------
    None.
    
    Outputs
    -------
    This function creates a ".gdb" file that contains all the geometric 
    features from the KMZ/KML files in the input directory. 
    The data from the KMZs/KMLs are all blended into one layer per geometry 
    type. For example, suppose the following file structure:
        -"A.kmz" has points, 
        -"B.kmz" has points and linestrings,
        -"C.kmz" has points and polygons,
        -"D.kmz" has points. 
    In that case, the output GDB file will have 3 layes:
        -One layer containing all the points (called "All_Points"), another 
        containing all the polylines (called "All_Polylines"), and yet another 
        containing all the polygons (called "All_Polygons").
    
    '''
    # Fishing out the output folder and the GDB filename
    output_gdb_path = pathlib.Path(output_gdb)
    output_folder = str(output_gdb_path.parent)
    master_gdb    = output_gdb_path.stem + output_gdb_path.suffix
    
    # Creating the temp folder in case it doesn't exist
    temp_newly_created = False
    if not os.path.exists(temp_folder):
        # Flag to indicate that this folder should be deleted at the end of
        # the process
        temp_newly_created = True
        os.mkdir(temp_folder)
    
    # Creating master GDB. This will store many temporary layers and the 
    # finalized combined layers.
    master_gdb_fullpath = os.path.join(output_folder, master_gdb)
    if os.path.exists(master_gdb_fullpath):
        arcpy.Delete_management(master_gdb_fullpath)
    arcpy.CreateFileGDB_management(output_folder, master_gdb)
    
    # Get list of all KMZs/KMLs in the input folder
    list_of_kmzs = [f for f in os.listdir(kmz_folder) 
                    if ((f[-4:].lower()=='.kmz') or (f[-4:].lower()=='.kml'))]
    
    # Set that contains the names of the consolidated layers created thus far
    # (i.e., "All_Lines", "All_Polygons", ...). This set starts empty and 
    # gets populated as the execution progresses. 
    consolidated_layers = set()
    
    # Looping over every KMZ/KML file
    for i,kmz in enumerate(list_of_kmzs):
        # Convert KMZ/KML file into a separate temporary GDB
        kmz_fullpath = os.path.join(kmz_folder, kmz)
        kmz_gdb_name = os.path.join(temp_folder, kmz)[:-3] + 'gdb'
        if os.path.exists(kmz_gdb_name):
            arcpy.Delete_management(kmz_gdb_name)
        print(f"CONVERTING: {kmz_fullpath}")
        arcpy.env.workspace = kmz_folder
        arcpy.KMLToLayer_conversion(kmz_fullpath, temp_folder)
        
        feature_classes = arcpy.ListFeatureClasses('*', '', 'Placemarks')
        
        arcpy.env.workspace = kmz_gdb_name
        
        # Looping over every Featureclass from the newly-created GDB
        feature_classes = arcpy.ListFeatureClasses('*', '', 'Placemarks')
        
        # If the input KMZ/KML is empty, the `feature_class` object from the 
        # function above will contain None. This needs to be fixed for the 
        # script to work.
        if feature_classes is None:
            feature_classes = []
            
        for this_fc in feature_classes:
            
            arcpy.env.workspace = kmz_gdb_name
            
            # Adding the "kmz_filename" column
            arcpy.management.AddField(in_table=this_fc,
                                      field_name='kmz_filename', 
                                      field_type='TEXT', 
                                      field_length=1024)
            
            # Updating the values of the "kmz_filename" column
            with arcpy.da.UpdateCursor(this_fc,'kmz_filename') as rows:
                for row in rows:
                    row[0] = kmz
                    rows.updateRow(row)
                    del(row)
            del(rows)
            
            # Creating the name for the new layer
            new_layer_name = kmz_gdb_name[kmz_gdb_name.rfind(os.sep) + 1:-4]
            
            # Fixing some invalid characters in the layer name
            valid_characters = set(string.ascii_letters + string.digits + '_')
            invalid_characters = set([x for x in new_layer_name if x not in valid_characters])
            for this_char in invalid_characters:
                new_layer_name = new_layer_name.replace(this_char,'_')
            new_layer_name = arcpy.ValidateTableName(new_layer_name)
            
            # Ensuring that the first character isn't a number
            if new_layer_name[0] in string.digits:
                new_layer_name = '_' + new_layer_name
            
            # Copying the Featureclass from the newly-created GDB to the 
            # Master GDB
            print(f"COPYING: {this_fc} FROM: {kmz_gdb_name}")
            fcCopy = os.path.join(kmz_gdb_name, 'Placemarks', this_fc)
            
            arcpy.FeatureClassToFeatureClass_conversion(
                fcCopy, master_gdb_fullpath, new_layer_name)
            
            # Copying the features from each individual Featureclass into the 
            # main "All_Points" or "All_Polylines" Featureclasses (i.e., the
            # consolidated layers).
            master_layer_name = f'All_{this_fc}'
            arcpy.env.workspace = master_gdb_fullpath
            print(f"COPYING: {this_fc} TO: {master_layer_name}")
            
            # If the consolidated layer has already been created, append 
            # the KMZ/KML's data to it. If not, the layer needs to be created
            # and the set of consolidated layers needs to be updated.
            if master_layer_name in consolidated_layers:
                arcpy.management.Append(new_layer_name, master_layer_name)
            else:
                arcpy.CopyFeatures_management(new_layer_name, master_layer_name)
                consolidated_layers.add(master_layer_name)
            
            # Delete the temporary Featureclass layer inside the master GDB
            arcpy.Delete_management(new_layer_name)
            
        # Deleting the newly-generated GDB and lyrx files
        arcpy.env.workspace = master_gdb_fullpath
        if os.path.isdir(kmz_gdb_name):
            arcpy.Delete_management(kmz_gdb_name)
        lyrx_filename = kmz_gdb_name[:-3] + 'lyrx'
        if os.path.isfile(lyrx_filename):
            os.remove(lyrx_filename)
    
    # If the temp folder didn't exist at the beginning of this process, it
    # should be deleted at the end.
    if temp_newly_created:
        os.rmdir(temp_folder)