python.multiprocessing and "FATAL ERROR (INFADI) MISSING DIRECTORY"

Question

While trying to do multiprocessing with arcpy, I am occasionally running into this error:

FATAL ERROR (INFADI)
MISSING DIRECTORY

I have no clue what is triggering this error, and it crashes the python process, making it impossible to get a traceback on it. It occurs while writing final raster output from a lengthy sound model.

It is sometimes accompanied by an error

Unable to write BND file for %TEMP%\ras####

Where %Temp is parsed correclty and #### is some random 4 digit number. This is unusual because each process has its own workspace, which is where most files should be written.

The problem is not the input data... I can rerun the program on the failed inputs and it will run correctly.

Answer 1

Here are some things to check:

Are you using cursors? Are you releasing them? Are you trying to re-use any objects in different processes? Are you sharing the same temp location? Are you doing in memory processing?

In general, the arcpy is just a wrapper around the com objects and any type of multiprocessing will be tricky.

Answer 2

I found this issue arises when the arcpy.env.workspace and arcpy.env.scratchWorkspace are the same for two different processes. Arc writes almost all intermediate rasters to the workspace (or scratch workspace) in the ESRI GRID format. You can't write two ESRI GRID rasters into the same directory at the same time due to the pseudo-database structure of the format (the info folder holds unique keys for each raster).

I have avoided this error by assigning unique workspace and scratchWorkspace for each process using a temporary tempfile.mkdtemp folder.

Answer 3

I've encountered this as well and have yet to find a sound fix. My work around is 1) to make sure that the multiprocessing task is robust enough to check if tasks are complete or not then create a new job list. 2) schedule two scripts to launch every 10-15 minutes. One script contains a command to kill select running python processes and the second relaunches the desired multiprocessing script. Essentially, this refreshes the multiprocessing pool. The kill script is something like this:

def read_pid():
    inFile = open("E:/temp/pid.csv")
    for line in inFile:
        pid = str(line)
    inFile.close()
    return pid

def kill():
    if os.path.exists("E:/temp/pid.csv")==True:
        pid = read_pid()
        PROCESS_TERMINATE=1
        handle = ctypes.windll.kernel32.OpenProcess(PROCESS_TERMINATE,False,pid)
        ctypes.windll.kernel32.TerminateProcess(handle,-1)
        ctypes.windll.kernel32.CloseHandle(handle)
    else:
        return

Each launch of the desired script I have it write its PID to a csv.

Answer 4

I must admit I am at this point, just a multithreading wannabee, but a blog at https://pythongisandstuff.wordpress.com/2013/07/31/using-arcpy-with-multiprocessing-%E2%80%93-part-3/ suggests that integrating the arcpy.Exists() function is key to making it happen.

Answer 5

I found that I was getting the INFADI error when trying have multiple threads/cores save and modify rasters in one folder. Assigning a subfolder to each task for outputs seems to solve the issue. I believe that the problem had to do with multiple read/writes to peripheral files associated with the raster (e.g. the "info" folder). I now also employ the following precautions :

import arcpy,multiprocessing,random

def run(foo,c):
    tempFolder = os.path.join("Z:/temp/",'temp_%s'%(str(c)))
    if not os.path.exists(tempFolder): os.mkdir(tempFolder)
    arcpy.env.scratchWorkspace = tempFolder
    arcpy.env.Workspace = tempFolder

    # create unique object in memory, run task, then delete unique object in memory
    tempMem = str(rnd)
    try:arcpy.Delete_management(tempMem)
    except:pass

    <tasks> #output to appropriate subfolder

    arcpy.Delete_management(tempMem)

if __name__ == '__main__':
    cores = 3
    pool = multiprocessing.Pool(cores)
    count = 0
    for foo in bar:
        pool.apply_async(run,(foo,c))
        count +=1
    pool.close()
    pool.join()