python.multiprocessing and “FATAL ERROR (INFADI) MISSING DIRECTORY”

Question

While trying to do multiprocessing with arcpy, I am occasionally running into this error:

FATAL ERROR (INFADI)
MISSING DIRECTORY

I have no clue what is triggering this error, and it crashes the python process, making it impossible to get a traceback on it. It occurs while writing final raster output from a lengthy sound model.

It is sometimes accompanied by an error

Unable to write BND file for %TEMP%\ras####

Where %Temp is parsed correclty and #### is some random 4 digit number. This is unusual because each process has its own workspace, which is where most files should be written.

The problem is not the input data... I can rerun the program on the failed inputs and it will run correctly.

Answer 1

Here are some things to check:

Are you using cursors? Are you releasing them? Are you trying to re-use any objects in different processes? Are you sharing the same temp location? Are you doing in memory processing?

In general, the arcpy is just a wrapper around the com objects and any type of multiprocessing will be tricky.

Answer 2

I must admit I am at this point, just a multithreading wannabee, but a blog at http://pythongisandstuff.wordpress.com/2011/07/11/using-arcpy-with-multiprocessing-%e2%80%93-part-3/ suggests that integrating the arcpy.Exists() function is key to making it happen.

Answer 3

I've encountered this as well and have yet to find a sound fix. My work around is 1) to make sure that the multiprocessing task is robust enough to check if tasks are complete or not then create a new job list. 2) schedule two scripts to launch every 10-15 minutes. One script contains a command to kill select running python processes and the second relaunches the desired multiprocessing script. Essentially, this refreshes the multiprocessing pool. The kill script is something like this:

def read_pid():
    inFile = open("E:/temp/pid.csv")
    for line in inFile:
        pid = str(line)
    inFile.close()
    return pid

def kill():
    if os.path.exists("E:/temp/pid.csv")==True:
        pid = read_pid()
        PROCESS_TERMINATE=1
        handle = ctypes.windll.kernel32.OpenProcess(PROCESS_TERMINATE,False,pid)
        ctypes.windll.kernel32.TerminateProcess(handle,-1)
        ctypes.windll.kernel32.CloseHandle(handle)
    else:
        return

Each launch of the desired script I have it write its PID to a csv.

Answer 4

I found that I was getting the INFADI error when trying have multiple threads/cores save and modify rasters in one folder. Assigning a subfolder to each task for outputs seems to solve the issue. I believe that the problem had to do with multiple read/writes to peripheral files associated with the raster (e.g. the "info" folder). I now also employ the following precautions :

import arcpy,multiprocessing,random

def run(foo,c):
    tempFolder = os.path.join("Z:/temp/",'temp_%s'%(str(c)))
    if not os.path.exists(tempFolder): os.mkdir(tempFolder)
    arcpy.env.scratchWorkspace = tempFolder
    arcpy.env.Workspace = tempFolder

    # create unique object in memory, run task, then delete unique object in memory
    tempMem = str(rnd)
    try:arcpy.Delete_management(tempMem)
    except:pass

    <tasks> #output to appropriate subfolder

    arcpy.Delete_management(tempMem)

if __name__ == '__main__':
    cores = 3
    pool = multiprocessing.Pool(cores)
    count = 0
    for foo in bar:
        pool.apply_async(run,(foo,c))
        count +=1
    pool.close()
    pool.join()