1

We are currently trying to restructure some of the companies GIS data. It is located on an internal network drive (no SDEs). For each year, there are 1000+ MXDs with map layouts that still need to be working after the restucturing process. Therefore im trying to analyse our MXDs before we start changing the data links using the .findAndReplaceWorkspacePath() function.

What I've done so far: 1. For each year, I've listed all MXDs available (very fast, using os.walk) and 2. In a second loop, I now want to find all MXDs that contain data pointing to a certain directory. The search process involves arcpy.mapping, which I find terribly slow... Here is my function for the search process:

def MXDchecker(inList, searchString):
    internalList = []
    internalCount = 0
    counter = 0
    for mxdFile in inList:
        print(u"\tChecking MXD: {}".format(mxdFile))
        try:
            mxd = m.MapDocument(mxdFile)
            # create list for links to v-drive:
            shortList = []
            # start search, find all links to v-drive
            for df in m.ListDataFrames(mxd):
                for lyr in m.ListLayers(mxd, data_frame=df):
                    if lyr.supports("DATASOURCE"):
                        if searchString in lyr.dataSource:
                            shortList.append(lyr.dataSource)
                            print(u"\t\tFound v-link.")
            # Check if list has entries:
            if shortList:
                internalList.append([mxdFile, [shortList]])
                internalCount += 1
            # delete mxd
            del mxd
        except:
            print(u"\tError for processing MXD {}".format(mxdFile))
        if internalCount == 100:
            counter += internalCount
            internalCount = 0
            print(u"\tFound {} MXDs with v-links...".format(unicode(counter)))
    counter += internalCount
    print(u"Found a total of {} MXDs with v-links.".format(unicode(counter)))
    return internalList

The returned list contains all relevant connections and the mxd path. I tested this on only 5 MXDs, some of which contain multiple dataframes and numerous layers. The processing times are very long (> 10 min), for a run on 1000 + MXDs it would take a full day at least, without even changing any of the data links...

My questions:

How can I speed up this process in general? Is it better to first find all MXDs containing any connection to the v-drive and then iterate on them, straight with the processing?

Note that:

  • I cannot copy the MXDs to a local drive for processing, it has to be done in the given location
  • I tried using the new ArcGIS Pro arcpy.mp module, but I found the structure to be too different for the use with "old" MXD files.
  • I'm working with Python 2.7 for ArcGIS 10.6.1 with an Advanced License. I do also have access to ArcGIS Pro (Advanced).
  • How long does it take to open one of these mxd's in ArcMap? – Berend Dec 19 '18 at 15:45
  • @Berend: it takes a while too... worst case we are talking more than 1 min. But they still need to be processed too, unfortunately. There's not much I can change about the individual MXDs. – dru87 Dec 19 '18 at 15:59
  • 2
    I don't think you'll be able to find a much faster way. However, you can possibly split up the work between multiple machines or processes. For example, you can use the multiprocessing module to spawn up a different process for each year and write out text files to log what they find. So if there were 4 years of data to search through, you could run 4 processes simultaneously for each year, rather than having one process run through one year at a time. You could also do this with 4 different machines if you had resources available. – crmackey Dec 19 '18 at 16:43
  • There's a stack exchange site called Code Review, maybe they can help speed up your code. – csk Dec 19 '18 at 16:48
  • 1
    @dru87, If you don't pass a data_frame argument, then ListLayers() will search all data frames and return all layers. It would be nice if ESRI's documentation clarified how the data_frame parameter is used. – Tom Dec 20 '18 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.