python.multiprocessing and “FATAL ERROR (INFADI) MISSING DIRECTORY” [closed]

Question

I have a question which is closely related to this old question here: python.multiprocessing and "FATAL ERROR (INFADI) MISSING DIRECTORY"

I have also looked at this, which seems to relate to something similar: Multiprocessing Errors - ArcGIS implementation

just like in the linked question, I run into the error: "FATAL ERROR (INFADI) MISSING DIRECTORY" at random times, but never the BND-error

I have multiple folders with thousands of shapefiles in each, where I want to convert two of the shapefile fields to two different rasterlayers. the shapefiles were converted from csvs via multiprocessing, with no issues whatsoever.

I have tried most of the workarounds from the previous question, such as creating temporary and unique scratchworkspaces and workspaces, check that the infile exists and create a string in memory and flush it, but to no avail. I have also tried splitting the workload, so that it will first write all rasters for one field, then do all the rasters for the second field.

so my question is, would anybody know of another potential workaround? I was thinking that it might be possible to somehow make sure that when a new raster is written to the folder, that no other processes try to access the space, but I'm unsure how to code it.

relevant code excerpts below:

import arcpy
import os
import sys
import arcgisscripting
import multiprocessing
from functools import partial

def multiproc(infile,path,savepath1,savepath2):

 try:    
        arcpy.Exists(infile)

        # Set local variables
        inLayer = path + "sample.xml"
        fieldList = arcpy.ListFields(infile[:-4]+".dbf", "field1")
        inData = infile + " field1"
        outLayer = infile[:-4]

        # Check out the ArcGIS Geostatistical Analyst extension license
        arcpy.CheckOutExtension("GeoStats")
        # Execute CreateGeostatisticalLayer
        arcpy.GACreateGeostatisticalLayer_ga(inLayer, inData, outLayer)   
        arcpy.GALayerToGrid_ga(outLayer, savepath1 + "/" + infile[:-4])
        arcpy.Delete_management(outLayer)

        #starting second raster creation process
        arcpy.Exists(infile)
        fieldList = arcpy.ListFields(infile[:-4]+".dbf", "field2")
        inData = infile + " field2"
        outLayer = infile[:-4]

        # Check out the ArcGIS Geostatistical Analyst extension license
        arcpy.CheckOutExtension("GeoStats")
        # Execute CreateGeostatisticalLayer
        arcpy.GACreateGeostatisticalLayer_ga(inLayer, inData, outLayer)
        arcpy.GALayerToGrid_ga(outLayer, savepath2 + "/" + infile[:-4])
        arcpy.Delete_management(outLayer)


    except:
        import logging
        LOG_FILENAME = 'c:/error.txt'
        logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG,)

        logging.debug(infile)

        logging.exception('Got exception on main handler')
        pass


if __name__=='__main__':
# Set environment settings
#setting paths for the directories 

##Finding paths and list of files      

#multiprocessing part
    Pool=multiprocessing.Pool
    workers=multiprocessing.cpu_count()
    pool=Pool(processes=workers)

    partial_multiproc=partial(multiproc,path=path,savepath1=savepath1,savepath2=savepath2)
    pool.map(partial_multiproc,filelist)
    pool.close()
    pool.join()

---

after having moved the data and tried on an internal harddrive (the previous attempts have been on an external HD), I also got the BND-error on rare occassions, whereas the INFADI is much more prevalent.

---

I have been unable to find a full solution so far, but a potential workaround lies in creating temp-folders that are deleted with arcpy at the end of the multiproc-function. even though the error persists, the faulty folders are - as far as I can tell - not deleted, meaning that at least it is possible to know which rasters were not completed and redo these later.