3

I have a question which is closely related to this old question here: python.multiprocessing and "FATAL ERROR (INFADI) MISSING DIRECTORY"

I have also looked at this, which seems to relate to something similar: Multiprocessing Errors - ArcGIS implementation

just like in the linked question, I run into the error: "FATAL ERROR (INFADI) MISSING DIRECTORY" at random times, but never the BND-error

I have multiple folders with thousands of shapefiles in each, where I want to convert two of the shapefile fields to two different rasterlayers. the shapefiles were converted from csvs via multiprocessing, with no issues whatsoever.

I have tried most of the workarounds from the previous question, such as creating temporary and unique scratchworkspaces and workspaces, check that the infile exists and create a string in memory and flush it, but to no avail. I have also tried splitting the workload, so that it will first write all rasters for one field, then do all the rasters for the second field.

so my question is, would anybody know of another potential workaround? I was thinking that it might be possible to somehow make sure that when a new raster is written to the folder, that no other processes try to access the space, but I'm unsure how to code it.

relevant code excerpts below:

import arcpy
import os
import sys
import arcgisscripting
import multiprocessing
from functools import partial

def multiproc(infile,path,savepath1,savepath2):

 try:    
        arcpy.Exists(infile)

        # Set local variables
        inLayer = path + "sample.xml"
        fieldList = arcpy.ListFields(infile[:-4]+".dbf", "field1")
        inData = infile + " field1"
        outLayer = infile[:-4]

        # Check out the ArcGIS Geostatistical Analyst extension license
        arcpy.CheckOutExtension("GeoStats")
        # Execute CreateGeostatisticalLayer
        arcpy.GACreateGeostatisticalLayer_ga(inLayer, inData, outLayer)   
        arcpy.GALayerToGrid_ga(outLayer, savepath1 + "/" + infile[:-4])
        arcpy.Delete_management(outLayer)

        #starting second raster creation process
        arcpy.Exists(infile)
        fieldList = arcpy.ListFields(infile[:-4]+".dbf", "field2")
        inData = infile + " field2"
        outLayer = infile[:-4]

        # Check out the ArcGIS Geostatistical Analyst extension license
        arcpy.CheckOutExtension("GeoStats")
        # Execute CreateGeostatisticalLayer
        arcpy.GACreateGeostatisticalLayer_ga(inLayer, inData, outLayer)
        arcpy.GALayerToGrid_ga(outLayer, savepath2 + "/" + infile[:-4])
        arcpy.Delete_management(outLayer)


    except:
        import logging
        LOG_FILENAME = 'c:/error.txt'
        logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG,)

        logging.debug(infile)

        logging.exception('Got exception on main handler')
        pass


if __name__=='__main__':
# Set environment settings
#setting paths for the directories 

##Finding paths and list of files      

#multiprocessing part
    Pool=multiprocessing.Pool
    workers=multiprocessing.cpu_count()
    pool=Pool(processes=workers)

    partial_multiproc=partial(multiproc,path=path,savepath1=savepath1,savepath2=savepath2)
    pool.map(partial_multiproc,filelist)
    pool.close()
    pool.join()

after having moved the data and tried on an internal harddrive (the previous attempts have been on an external HD), I also got the BND-error on rare occassions, whereas the INFADI is much more prevalent.

I have been unable to find a full solution so far, but a potential workaround lies in creating temp-folders that are deleted with arcpy at the end of the multiproc-function. even though the error persists, the faulty folders are - as far as I can tell - not deleted, meaning that at least it is possible to know which rasters were not completed and redo these later.

Improve this question
4
  • hey are you able to get a line number for the error? – bcollins Oct 7 '13 at 18:28
  • Hi, no, no line numbers. the error message is literally the caps lock msg (INFADI etc). however, on the BND error it gave a traceback on the arcpy.GALayerToGrid_ga-expressions – userNOR Oct 7 '13 at 18:34
  • 1
    Three suggestions - two could be related. First, I'd dump the import of arcgisscripting - it's included as part of arcpy. Second, the first line of your try block checks if the file exists, but doesn't do anything with that information, so the check doesn't help the program. You should try an "if not arcpy.Exists(infile): {do something here to handle that}. Lastly, try making your exception handling more robust with the traceback module. It might give you a better pointer to where this issue is coming from. – nicksan Oct 9 '13 at 3:39
  • Hi, thanks for your suggestions. I'll try the first and the third. as for the second one, I left it in because of a suggestion in the first blogpost here: pythongisandstuff.wordpress.com that said it might force a file refresh which would avoid randomly thrown errors. but might as well remove it again, since it didn't help me much. – userNOR Oct 9 '13 at 14:55