0

I want to update metadata (the abstract/description) on an Esri ArcGIS feature class. I am using the Esri arcpy python lib. When I try this using a file geodatabase it works fine. When I try it using a SDE definition (SQL Server connection) it fails with the following error message:

ExecuteError: Error using XSLT transformation "c:\program files (x86)\ArcGIS\desktop10.3\Metadata/Stylesheets/gpTools/merge imported metadata with existing.xslt": An error occurred while loading document '__TestData\myfeatureclass_xslttransforma3.xml'. See InnerException for a complete description of the error. Failed to execute (Merge Import With Existing). Failed to execute (ImportMetadata).

It fails on the line that imports the metadata into the layer. Here is the complete code:

import traceback
import os
import sys
import arcpy
import xml.etree.ElementTree as ET

def XSLTFilePath():
    installDir = arcpy.GetInstallInfo("desktop")["InstallDir"]
    return os.path.join(installDir, r"Metadata\Stylesheets\gpTools\exact copy of.xslt")

def metadataScratchFilePath(featureClassName):
    metadata_temp_folder = arcpy.env.scratchFolder
    return os.path.join(metadata_temp_folder, featureClassName + ".xml")

def nodeIsDefined(node):
    if node == None:
        return False
    elif len(node) == 0:
        return False
    return True

def updateMetaDataFileAbstract(metadata_file,abstractText):
    # update or add abstract info to the metadata
    doc = ET.parse(metadata_file)
    root_node = doc.getroot()
    dataIdInfo_node = root_node.find("dataIdInfo")
    if nodeIsDefined(dataIdInfo_node):
        abstract_node = dataIdInfo_node.find("idAbs")
    else:
        dataIdInfo_node = ET.SubElement(root_node, "dataIdInfo")
        abstract_node = ET.SubElement(dataIdInfo_node, "idAbs")
    abstract_node.text = abstractText
    tree = ET.ElementTree(root_node)
    tree.write(metadata_file)

def updateFeatureClassAbstract(featureClassLocation,featureClassName,abstractText):
    # set up scratch file location for XML
    metadata_file = metadataScratchFilePath(featureClassName)
    if os.path.exists(metadata_file):
        os.remove(metadata_file)
    # create XML scratch file
    arcpy.env.workspace = featureClassLocation
    arcpy.XSLTransform_conversion(featureClassName, XSLTFilePath(), metadata_file)
    # update XML file with new abstract
    updateMetaDataFileAbstract(metadata_file,abstractText)
    # write updates to the feature class
    arcpy.ImportMetadata_conversion(metadata_file, "FROM_ARCGIS", featureClassName,
                                    Enable_automatic_updates=False)

try:
    print("This works...")
    USERPROFILE = os.getenv('USERPROFILE')
    WS_FILE = USERPROFILE + '\\Documents\\ArcGIS\\Default.gdb'
    updateFeatureClassAbstract(WS_FILE,"myfeatureclass","This is my new abstract text")
    print("This doesn't...")
    updateFeatureClassAbstract('My_DB.sde', "myfeatureclass", "This is my new abstract text")
    # normal completion
except:
    crash_info = traceback.format_exc()
    crash_msg = "Script Crashed."
    print(crash_msg)
    print(crash_info)
    print(crash_msg)
    print(crash_info)
    sys.exit(99)
    # abnormal completion
  • yes that's the complete error message. When running the import against the SDE, it seems like it's trying to use the wrong xml file. – Darren Mar 3 '17 at 16:35
  • 1
    Can you expand on what you mean by "it seems like it's trying to use the wrong xml file"? Please edit your question to include this plus your system details requested above. – Midavalo Mar 3 '17 at 16:36
  • I'm using ArcGIS 10.3.1 for Desktop, and the SDE database is SQL Server 11.0.5058. In the code the xml file is created, so I don't know why it is looking for the one named in the error message – Darren Mar 3 '17 at 16:57
  • 1
    Please edit your question to include any new information or clarification. Comments are for potential answerers to request clarification or ask questions, your response should be as edits to your question. – Midavalo Mar 3 '17 at 16:58
  • 1
    Please also include your ArcGIS Server version (your SDE). – Midavalo Mar 3 '17 at 17:00
3

I do the following to edit my organizations sde metadata. I use arcpy.ExportMetadata_conversion() tool to export the metadata into an xml file using the ARCGIS2FGDC translator. Another script writes the relevant values (Purpose, Abstract, Tags, Credits,etc) from the xml to a formatted xls file where I make the changes. Then I read the xls file and edit the original exported xml files. I then import it back into my sde feature classes using the arcpy.MetadataImporter_conversion() and then arcpy.UpgradeMetadata_conversion() tools. It works flawlessly.

Here is my script which I use to edit my metadata. I use identifiers in the tags which have ":" in them to group and export data to upload to a repository so that part of the code would not apply to you. I have some extra stuff in there too.

import os, csv, shutil, time, datetime, sys, re
from lxml import etree as ET
from operator import itemgetter
import arcpy

class MetaData(object):
    def __init__(self, fcFile, xmlOutputDir, overwriteOutput = False):
        ''' initialaize this class with the path of the feature class file and the
        output path for the meta data xml file that the tool outputs to read and edit the 
        meta data.

        MetaData(pathOfFeatureClass, pathOfOutputDirectory)
        '''
        self.overwriteOutput = overwriteOutput
        self.fcFile = fcFile
        self.xmlOutputDir = xmlOutputDir
        self.xmlFileName = ''
        self.exportMetadata(self.fcFile, self.xmlOutputDir)
        self.parser = ET.XMLParser(remove_blank_text=True)
        self.tree = ET.parse(self.xmlFileName, self.parser)
        self.root = self.tree.getroot()
        self.meta_data = {}
        self.defaultPath = os.path.dirname(self.xmlFileName)
        self.defaultName = os.path.splitext(os.path.basename(self.xmlFileName))[0]

        self.title = None
        self.abstract = None
        self.purpose = None
        self.uselimits = None
        self.credits = None
        self.tags = None
        self.identifiers = None

        self.__getIdInfo()
        self.__getSpdoInfo()
        self.__getSprefInfo()
        self.__getEaInfo()
        self.__getMetaInfo()
        self.__descItem()

    def __getXMLValue(self, val, attrib = False):
        if val is not None:
            if attrib:
                return val.strip()
            elif val.text:  
                return val.text.strip()
            else:
                return ''
        else:
            return ''

    def warningMessage(self):
        print "Don't forget to commit changes to the xml file by using 'object.write()'"

    def __getIdInfo(self):
        data = {}
        self.title = self.__getXMLValue(self.root.find("idinfo/citation/citeinfo/title")) #the name by which the data set is known.
        self.abstract = self.__getXMLValue(self.root.find("idinfo/descript/abstract"))
        self.purpose = self.__getXMLValue(self.root.find("idinfo/descript/purpose"))
        self.uselimits = self.__getXMLValue(self.root.find("idinfo/useconst"))
        self.credits = self.__getXMLValue(self.root.find("idinfo/datacred"))
        self.tags = [tag.text for tag in self.root.findall("idinfo/keywords/theme/themekey") if ":" not in tag.text]
        self.identifiers = [tag.text for tag in self.root.findall("idinfo/keywords/theme/themekey") if ":" in tag.text]

        title = self.root.find("idinfo/citation/citeinfo/title") #the name by which the data set is known.
        geoform = self.root.find("idinfo/citation/citeinfo/geoform")
        abstract = self.root.find("idinfo/descript/abstract") #a brief narrative summary of the data set.
        purpose = self.root.find("idinfo/descript/purpose") #a summary of the intentions with which the data set was developed.
        bounding_westbc = self.root.find("idinfo/spdom/bounding/westbc")
        bounding_eastbc = self.root.find("idinfo/spdom/bounding/eastbc")
        bounding_northbc = self.root.find("idinfo/spdom/bounding/northbc")
        bounding_southbc = self.root.find("idinfo/spdom/bounding/southbc") 
        tags =  self.root.findall("idinfo/keywords/theme/themekey") #common-use word or phrase used to describe the subject of the data set.
        uselimits = self.root.find("idinfo/useconst") #restrictions and legal prerequisites for using the data set after access is granted. These include any use constraints applied to assure the protection of privacy or intellectual property, and any special restrictions or limitations on using the data set.
        credits = self.root.find("idinfo/datacred") #recognition of those who contributed to the data set.

        data['title'] = self.__getXMLValue(title)
        data['geoform'] = self.__getXMLValue(geoform)
        data['abstract'] = self.__getXMLValue(abstract)
        data['purpose'] = self.__getXMLValue(purpose)
        data['bounding'] = {}
        data['bounding']['westbc'] = self.__getXMLValue(bounding_westbc)
        data['bounding']['eastbc'] = self.__getXMLValue(bounding_eastbc)
        data['bounding']['northbc'] = self.__getXMLValue(bounding_northbc)
        data['bounding']['southbc'] = self.__getXMLValue(bounding_southbc)
        data['identifiers'] = {identifier.text.split(":")[0].lower():identifier.text.split(":")[1].lower() for identifier in tags if len(identifier.text.split(":")) == 2}
        data['tags'] = [tag.text for tag in tags if ":" not in tag.text]
        data['use_limits'] = self.__getXMLValue(uselimits)
        data['credits'] = self.__getXMLValue(credits)

        self.meta_data['idinfo'] = data

    def __getSpdoInfo(self):
        data = {}
        direct = self.root.find('spdoinfo/direct')
        sdtstype = self.root.find('spdoinfo/ptvctinf/sdtsterm/sdtstype')
        ptvctcnt = self.root.find('spdoinfo/ptvctinf/sdtsterm/ptvctcnt')

        data['direct'] = self.__getXMLValue(direct)
        data['sdtstype'] = self.__getXMLValue(sdtstype)
        data['ptvctcnt'] = self.__getXMLValue(ptvctcnt)

        self.meta_data['spdoinfo'] = data

    def __getSprefInfo(self):
        data = {}
        mapprojn = self.root.find('spref/horizsys/planar/mapproj/mapprojn')
        #stdparll = self.root.findall('spref/horizsys/planar/mapproj/lambertc/stdparll')

        data['mapprojn'] = self.__getXMLValue(mapprojn)

        self.meta_data['sprefinfo'] = data

    def __getEaInfo(self):
        data = {}  

        enttypl = self.root.find('eainfo/detailed/enttyp/enttypl')
        self.root.findall('eainfo/detailed/attr')

        data['enttypl'] = self.__getXMLValue(enttypl)

        self.meta_data['eainfo'] = data

    def __getMetaInfo(self):
        data = {}
        metd = self.root.find("metainfo/metd")
        metstdn = self.root.find("metainfo/metstdn")
        metstdv = self.root.find("metainfo/metstdv")
        mettc = self.root.find("metainfo/mettc")
        try:
            data['metd'] = datetime.datetime.strptime(self.__getXMLValue(metd), '%Y%m%d')
        except:
            data['metd']=''
        data['metstdn'] = self.__getXMLValue(metstdn)
        data['metstdv'] = self.__getXMLValue(metstdv)
        data['mettc'] = self.__getXMLValue(mettc)

        self.meta_data['metainfo'] = data

    def __descItem(self):
        desc = arcpy.Describe(self.fcFile)
        p = self.fcFile.split("\\")
        p.remove(p[-1])
        p = '\\'.join(p)
        dsDesc = arcpy.Describe(p)
        try:
            dsDesc.datasetType
            ds_name = p.split("\\")[-1]
        except:
            ds_name = None
        data_type = desc.dataType
        base_name = None
        catalog_path = None
        data_elem_type = None
        name = None
        alias = None
        feature_type = None
        dataset = None
        shape_type = None
        if data_type == 'FeatureClass':
            shape_type = desc.shapeType
            feature_type = desc.featureType
            base_name = desc.baseName
            catalog_path = desc.catalogPath
            data_elem_type = desc.dataElementType
            name = desc.name
            alias = desc.aliasName
        else:
            base_name = desc.baseName
            catalog_path = desc.catalogPath
            data_elem_type = desc.dataElementType
            name = desc.name
            alias = desc.aliasName

        self.meta_data['info'] = {'catalog_path':catalog_path,'data_type':data_type,'base_name':base_name,'data_elem_type':data_elem_type,'name':name,'alias':alias,'shape_type':shape_type,'feature_type':feature_type,'ds_name':ds_name}

    def write(self, **kwargs):
        '''Use this method to overwrite any changes made to the metadata xml file.
        If this method isn't called the changes will not persist.  If you would like to save
        the file somewhere else and not overwrite the original use the variable "path" and give it
        the path where you would like to save the file.  If you would like to change the name but 
        not the path use the key word argument "file" without an extention since the .xml will be 
        appended to the end of the file name.  Use both key word arguments if you want to change 
        both the path and the file name.

        Use: object.write(path="path/folder", file="filename")'''
        file = kwargs.get('file',self.defaultName) + '.xml'
        path = kwargs.get('path',self.defaultPath)
        filePath = os.path.join(path,file)
        #print filePath
        fileOpen = open(filePath, "wb")
        #self.__updateEditDate()
        self.__removeTimestamp()
        self.tree.write(filePath, pretty_print=True)
        fileOpen.close()
        self.__getIdInfo()
        self.__getSpdoInfo()
        self.__getSprefInfo()
        self.__getEaInfo()
        self.__getMetaInfo()
        self.__descItem()
        #self.tree = ET.parse(self.xmlFileName, self.parser)
        #self.root = self.tree.getroot()
        #self.__getIdInfo()

    def __removeTimestamp(self):
        if hasattr(self.root.find('idinfo/descript/abstract'),'text'):
            abstract = self.root.find('idinfo/descript/abstract')
            datePattern = r" *?-? *?Last metadata edit \d{2}\D{1}\d{2}\D{1}\d{4}\D{1}\d{2}\D{1}\d{2}\D{1}\w{2}"
            pattern = re.compile(datePattern)
            search = pattern.search(abstract.text)
            newEditDateTime = time.strftime("Last metadata edit %m/%d/%Y %I:%M %p")
            try:
                if search.group(0):
                    oldEditDateTime = search.group(0)
                    abstractText = abstract.text.replace(oldEditDateTime,'')
                    abstract.text = abstractText.strip()
                    #abstract.text = abstractText.strip() + ' - ' + newEditDateTime
            except:
                abstract.text = abstract.text.strip()

    def __updateEditDate(self):
        import time
        '''This method is called by the write() method in order to give the metadata 
        a date and time stamp of when it was last edited'''
        if not hasattr(self.root.find('idinfo/descript/abstract'),'text'):
            pass
            self.__editElement('idinfo/descript/abstract', "-")
        abstract = self.root.find('idinfo/descript/abstract')
        datePattern = r" *?-? *?Last metadata edit \d{2}\D{1}\d{2}\D{1}\d{4}\D{1}\d{2}\D{1}\d{2}\D{1}\w{2}"
        pattern = re.compile(datePattern)
        search = pattern.search(abstract.text)
        newEditDateTime = time.strftime("Last metadata edit %m/%d/%Y %I:%M %p")
        try:
            if search.group(0):
                oldEditDateTime = search.group(0)
                abstractText = abstract.text.replace(oldEditDateTime,'')
                abstract.text = abstractText.strip() + ' - ' + newEditDateTime
        except:
            abstract.text = abstract.text.strip() + ' - ' + newEditDateTime

    def __editElement(self, node, elementText):
        '''This method will edit the text contained within any node or add the node if 
        it does not exist.  This is a private method and should be used carefully since 
        it can add tags that are not FGDC standard.  Any "edit" method should call this 
        to perform the edit.'''
        parentNode = node.split('/')
        childElem = parentNode.pop(-1)
        parentNode = '/'.join(parentNode)
        childNode = self.root.find(node)
        parentElem = self.root.find(parentNode)
        try:
            if childNode is None and parentElem is not None:
                print '"{0}" does not exist! Creating "{1}" node'.format(node,childElem)
                newElem = ET.Element(childElem)
                newElem.text = elementText
                parentElem.append(newElem)
            else:
                childNode.text = elementText
        except Exception, e:
            print e
        print "Added {0} to {1}".format(elementText,node)
        #ET.dump(parentElem)

    def lastEdit(self):
        '''This method is used to attempt to grab the last metadata date/time 
        stamp if there is one in the abstract and print it off. The timestamp 
        format should be "Last metadata edit mm/dd/yyyy hh:mm pm".

        Use: object.lastEdit()''' 
        meta_d = self.__getXMLValue(self.root.find("metainfo/metd"))
        if len(meta_d) == 8:
            meta_d = "{0}/{1}/{2}".format(meta_d[0:4],meta_d[4:6],meta_d[6:8])
            print meta_d
        abstract = self.root.find("idinfo/descript/abstract")
        datePattern = r" *?-? *?Last metadata edit \d{2}\D{1}\d{2}\D{1}\d{4}\D{1}\d{2}\D{1}\d{2}\D{1}\w{2}"
        pattern = re.compile(datePattern)
        search = pattern.search(abstract.text)
        try:
            if search.group(0):
                print search.group(0)   
        except Exception, e:
            print "The metadata does not contain a Last edit stamp in the abstract"

    def editTitle(self, newTitle):
        '''Title -- the name by which the data set is known.
        This method is used to modify the FGDC title tag text. 

        Use: object.editTitle("New Title").'''
        print "Current Title = {0}".format(self.title)
        node = 'idinfo/citation/citeinfo/title'
        self.__editElement(node, newTitle)
        self.title = newTitle
        print "New Title = {0}".format(self.title)
        self.write()


    def editAbstract(self, newAbstract):
        '''Abstract -- a brief narrative summary of the data set.

        This method is used to modify the FGDC abstract tag. Use: object.editAbstract("New Abstract Text")" 
        to commit any changes you will need to call the write() method to overwrite the old metadata

        Use: object.editAbstract("New Abstract")'''
        node = 'idinfo/descript/abstract'
        self.__editElement(node, newAbstract)
        self.write()

    def editPurpose(self, newPurpose):
        '''Purpose -- a summary of the intentions with which the data set was developed.

        this method is used to modify the FGDC purpose tag. Use: object.editAbstract("New Purpose Text")" 
        to commit any changes you will need to call the write() method to overwrite the old metadata

        Use: object.editPurpose("New Purpose")'''
        node = 'idinfo/descript/purpose'
        self.__editElement(node, newPurpose)
        self.write()

    def editAccessConstraints(self, newAccessConstraints):
        '''Access Constraints -- restrictions and legal prerequisites for accessing the data set. 
        These include any access constraints applied to assure the protection of privacy or 
        intellectual property, and any special restrictions or limitations on obtaining the data set.

        This method is used to modify the FGDC accconst tag text. 

        Use: object.editAccessConstraints("New Access Constraints")'''
        node = 'idinfo/accconst'
        self.__editElement(node, newAccessConstraints)
        self.write()

    def editUseConstraints(self, newUseConstraints):
        '''Use Constraints -- restrictions and legal prerequisites for using the data set after access 
        is granted.These include any use constraints applied to assure the protection of privacy or 
        intellectual property, and any special restrictions or limitations on using the data set.

        This method is used to modify the FGDC useconst tag text.

        Use: object.editUseConstraints("New Use Constraints")'''
        node = 'idinfo/useconst'
        self.__editElement(node, newUseConstraints)
        self.write()

    def editCredits(self, newCredits):
        '''Use Constraints -- restrictions and legal prerequisites for using the data set after access 
        is granted.These include any use constraints applied to assure the protection of privacy or 
        intellectual property, and any special restrictions or limitations on using the data set.

        This method is used to modify the FGDC useconst tag text.

        Use: object.editUseConstraints("New Use Constraints")'''
        node = 'idinfo/datacred'
        self.__editElement(node, newCredits)
        self.write()

    def addNode(self, node):
        import time
        node_list = node.split('/')
        nodes = node_list.pop(0)
        childNode = nodes
        previousNode = self.root.find(nodes)
        for i in range(len(node_list)):
            newNode = self.root.find(nodes)
            if newNode == None:
                newElem = ET.Element(childNode)
                newElem.text = ''
                previousNode.append(newElem)
                print "Created <{0}></{0}>".format(newElem.tag)
                previousNode = self.root.find(nodes)
            else:
                pass
                print "<{0}></{0}> exists".format(newNode.tag)
            childNode = node_list.pop(0)
            nodes = nodes + "/" + childNode
        #ET.dump(self.root.find('idinfo/test'))
        self.write()

    def addTag(self, newTag):
        '''This method will append a single tag contained within <themekey></themekey> elements

        Use: object.addTag("New Tag")'''
        rootIdinfoElem = self.root.find("idinfo")
        if self.root.find("idinfo/keywords") == None:
            kwElem = ET.Element("keywords")
            themeElem = ET.Element("theme")
            rootIdinfoElem.append(kwElem)
            idinfoKWelem = self.root.find("idinfo/keywords")
            idinfoKWelem.append(themeElem)
        tagsElement = self.root.find("idinfo/keywords/theme")
        tagText = [t.text for t in tagsElement.iterchildren() if t.tag == 'themekey']
        if tagsElement is not None:
            newElem = ET.Element("themekey")
            if newTag not in tagText:
                newElem.text = newTag
                tagsElement.append(newElem)
                print "Added the following node: <{0}>{1}</{0}>".format(newElem.tag,newElem.text)
            else:
                #pass
                print "The {0} tag already exists! Skipping........".format(newTag)
        else:
            #pass
            print "{0} is missing the <theme> node! Cannot perform the add operation!!!".format(self.xmlFileName)
        self.write()

    def addMultiTags(self,newTagsList):
        for tag in newTagsList:
            self.addTag(tag)
        tagsElement = self.root.find("idinfo/keywords/theme")
        print "Results for the theme node:"
        print ""
        ET.dump(tagsElement)
        self.write()

    def removeTags(self):
        '''This method will remove all themekey elements from the xml file

        Use: object.removeTags()'''
        tagsElement = self.root.find("idinfo/keywords/theme")
        if tagsElement is not None:
            for element in tagsElement.iterchildren():
                if element.tag == "themekey":
                    print "Removed the following node: <{0}>{1}</{0}>".format(element.tag,element.text)
                    tagsElement.remove(element)
        else:
            #pass
            print "{0} is missing the <theme> element! cannot perform the remove operation!!!".format(self.xmlFileName)
        print "Results for the theme node:"
        print ""
        #ET.dump(tagsElement)
        self.write()

    def removeLastTag(self):
        tagsElem = self.root.find("idinfo/keywords/theme")
        lastTagElem = [t for t in tagsElem.iterchildren()][-1]
        if lastTagElem.tag == "themekey":
            tagsElem.remove(lastTagElem)
            print "Removed the following node: <{0}>{1}</{0}>".format(lastTagElem.tag,lastTagElem.text)
        else:
            print "There are no more tags to remove!"
        self.write()

    def exportMetadata(self,fc, xmlOutputDir):
        self.xmlFileName = '{0}.xml' .format((os.path.join(xmlOutputDir,fc.split('\\')[-1])))
        dir = arcpy.GetInstallInfo("desktop")["InstallDir"]
        translator = os.path.join(dir,"Metadata\\Translator\\ARCGIS2FGDC.xml")
        if not os.path.exists(self.xmlFileName):
            arcpy.ExportMetadata_conversion (fc, translator, self.xmlFileName)
        elif os.path.exists(self.xmlFileName) and self.overwriteOutput:
            os.remove(self.xmlFileName)
            arcpy.ExportMetadata_conversion (fc, translator, self.xmlFileName)
        else:
            print "The following already exists and overwriteOutput is False: {0}".format(self.xmlFileName)

    def getCatalogPath(self):
        desc = arcpy.Describe(self.fcFile)
        catalog_path = desc.catalogPath
        return catalog_path

    def commitChanges(self):
        '''This method is ran to commit changes to the actual feature class.  If this 
        method is not ran no changes will be made to the feature class'''

        arcpy.MetadataImporter_conversion(self.xmlFileName, self.fcFile)
        arcpy.UpgradeMetadata_conversion(self.fcFile,"FGDC_TO_ARCGIS")
        #self.exportMetadata(self.fcFile, self.xmlOutputDir)

if __name__ == "__main__":
    m = MetaData(r"path\to\feature\class",r"path\to\xml\directory", overwriteOutput=True)
    m.addTag("Test")
    m.editPurpose("This is the Purpose")
    m.addMultiTags(["I","added","multiple","tags"])
    m.editTitle("This is the title")
    m.editAbstract("This is the abstract")
    m.editCredits("These are the credits")
    m.editUseConstraints("This is the useLimits")
    m.write() #writes changes to xml file and rereads the xml file
    m.commitChanges() #commits changes to the sde feature class
  • Thanks. Bottom line is that I needed to use MetadataImporter instead of ImportMetadata. – Darren Mar 3 '17 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.