Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

While trying to do multiprocessing with arcpy, I am occasionally running into this error:

FATAL ERROR (INFADI)
MISSING DIRECTORY

I have no clue what is triggering this error, and it crashes the python process, making it impossible to get a traceback on it. It occurs while writing final raster output from a lengthy sound model.

It is sometimes accompanied by an error

Unable to write BND file for %TEMP%\ras####

Where %Temp is parsed correclty and #### is some random 4 digit number. This is unusual because each process has its own workspace, which is where most files should be written.

The problem is not the input data... I can rerun the program on the failed inputs and it will run correctly.

share|improve this question
    
I will get back to this one soon, but having to work on a different model right now. –  blord-castillo Oct 5 '11 at 19:52

5 Answers 5

Here are some things to check:

Are you using cursors? Are you releasing them? Are you trying to re-use any objects in different processes? Are you sharing the same temp location? Are you doing in memory processing?

In general, the arcpy is just a wrapper around the com objects and any type of multiprocessing will be tricky.

share|improve this answer

I must admit I am at this point, just a multithreading wannabee, but a blog at http://pythongisandstuff.wordpress.com/2011/07/11/using-arcpy-with-multiprocessing-%e2%80%93-part-3/ suggests that integrating the arcpy.Exists() function is key to making it happen.

share|improve this answer

I've encountered this as well and have yet to find a sound fix. My work around is 1) to make sure that the multiprocessing task is robust enough to check if tasks are complete or not then create a new job list. 2) schedule two scripts to launch every 10-15 minutes. One script contains a command to kill select running python processes and the second relaunches the desired multiprocessing script. Essentially, this refreshes the multiprocessing pool. The kill script is something like this:

def read_pid():
    inFile = open("E:/temp/pid.csv")
    for line in inFile:
        pid = str(line)
    inFile.close()
    return pid

def kill():
    if os.path.exists("E:/temp/pid.csv")==True:
        pid = read_pid()
        PROCESS_TERMINATE=1
        handle = ctypes.windll.kernel32.OpenProcess(PROCESS_TERMINATE,False,pid)
        ctypes.windll.kernel32.TerminateProcess(handle,-1)
        ctypes.windll.kernel32.CloseHandle(handle)
    else:
        return

Each launch of the desired script I have it write its PID to a csv.

share|improve this answer

I found this issue arises when the arcpy.env.workspace and arcpy.env.scratchWorkspace are the same for two different processes. Arc writes almost all intermediate rasters to the workspace (or scratch workspace) in the ESRI GRID format. You can't write two ESRI GRID rasters into the same directory at the same time due to the pseudo-database structure of the format (the info folder holds unique keys for each raster).

I have avoided this error by assigning unique workspace and scratchWorkspace for each process using a temporary tempfile.mkdtemp folder.

share|improve this answer
    
I already use unique workspaces, but I will doublecheck that scratchWorkspace is unique too. I'm guessing not since it is writing to the %TEMP% diretory. –  blord-castillo Jun 24 '13 at 3:14
    
Jonah is right on. I'm processing thousands of rasters in a single directory across 5 simultaneous threads; setting a unique scratch workspace for each one is the only solution that has worked for me. Outputting to unique folders, as some have recommended, just creates more work for later...eventually I want them all in the same directory. –  Tom Sep 18 '13 at 15:48

I found that I was getting the INFADI error when trying have multiple threads/cores save and modify rasters in one folder. Assigning a subfolder to each task for outputs seems to solve the issue. I believe that the problem had to do with multiple read/writes to peripheral files associated with the raster (e.g. the "info" folder). I now also employ the following precautions :

import arcpy,multiprocessing,random

def run(foo,c):
    tempFolder = os.path.join("Z:/temp/",'temp_%s'%(str(c)))
    if not os.path.exists(tempFolder): os.mkdir(tempFolder)
    arcpy.env.scratchWorkspace = tempFolder
    arcpy.env.Workspace = tempFolder

    # create unique object in memory, run task, then delete unique object in memory
    tempMem = str(rnd)
    try:arcpy.Delete_management(tempMem)
    except:pass

    <tasks> #output to appropriate subfolder

    arcpy.Delete_management(tempMem)

if __name__ == '__main__':
    cores = 3
    pool = multiprocessing.Pool(cores)
    count = 0
    for foo in bar:
        pool.apply_async(run,(foo,c))
        count +=1
    pool.close()
    pool.join()
share|improve this answer
    
I never seem to get errors from writing multiple GRIDs to the same folder through multiple threads. The only problem seems to be that doing so slows down the processing and virtually invalidates the threading, as it only writes one raster at a time. –  Tom Aug 7 '13 at 18:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.