Improving performance of looping over MXDs using ArcPy?

Question

I am attempting to find the fastest method for looping over multiple map documents while keeping memory resources down. I'm running this script outside of Arcmap 10.

I have 5 map documents which are about 1mb each.

I have used both return functions and generator functions. After some testing I found the following code below to be the fastest, running in about 20 seconds.

Memory usage balloons to about 220 000 k when running then back down to 150 000 k after the del statements. If I remove the the del statements run in 18 secs but memory after remains at 220 000 k. which is undesirable.

Note: Importing arcpy on its own uses 117 000 k

I am looking for any feedback you have or suggestions to get increase speed, and reduce memory usage for the code below.

import time
import glob
import arcpy

start = time.time()

path = "F:\\Jobs2012\\06292\\ESRI\\"

def getMxdPath(path):
    """ Returns list of mxd filepath
   as strings for a given path."""
    mxdPaths = list()
    for mxd in glob.glob(path + r"*.mxd"):
        mxdPaths.append(mxd)
    return mxdPaths

def getMxdObjects(mxdPaths):
    """ Returns list of mxd objects"""
    mxdObjects = list()
    for mxd in mxdPaths:
        mxdObjects.append(arcpy.mapping.MapDocument(mxd))
    return mxdObjects

def genr(mxdObjects):
    for mxd in mxdObjects:
        yield mxd

def saveMxd(mxdGen):
    for mxd in mxdGen:
        print mxd.filePath
        mxd.save()
        #del mxd

mxdPaths = getMxdPath(path)

mxdObject = getMxdObjects(mxdPaths)
#del mxdPaths

mxdGen = genr(mxdObject)
#del mxdObject

saveMxd(mxdGen)
#del mxdGen

del mxdPaths
del mxdObject
del mxdGen

elapsed = (time.time() - start)

print elapsed

Answer 1

I don't understand why you're doing this:

def getMxdObjects(mxdPaths):
    """ Returns list of mxd objects"""
    mxdObjects = list()
    for mxd in mxdPaths:
        mxdObjects.append(arcpy.mapping.MapDocument(mxd))
    return mxdObjects

def genr(mxdObjects):
    for mxd in mxdObjects:
        yield mxd

Consolidate the two so you don't have all the MXDs in memory at once (that list is unnecessary if you're using a generator):

def genr(mxdPaths):
    """ Returns list of mxd objects"""
    mxdObjects = list()
    for mxd in mxdPaths:
        yield arcpy.mapping.MapDocument(mxd)

saveMxd(genr(mxdPaths))

Another thing you can do is force a garbage collection:

def genr(mxdPaths):
    """ Returns list of mxd objects"""
    import gc
    for mxd in mxdPaths:
        yield arcpy.mapping.MapDocument(mxd)
        gc.collect()

Also, 220000 k is only about 215 megabytes, this is a very small amount of memory on modern hardware.