3

I am using ArcGIS 'watershed' routine in a script which makes use of multiprocessing. This works fine, but during execution I get the message:

Unable to remove directory.  Possible causes:
1- Not owner of the directory
2- Another person or application is accessing this directory

EDIT 27/02/14: I've incorporated the suggestions of using arcpy.Exists (with a very rudimentary way of checking if something didnt exist) and writing all results to disk:

import arcpy
from arcpy import sa
arcpy.CheckOutExtension("Spatial")

def watershed(pnts, flowdir, flowacc):
    direc = tempfile.mkdtemp(dir = "C:\\temp")   #Create a separate directory for files to be written to.
    arcpy.env.scratchWorkspace = direc
    arcpy.env.workspace = direc

    res = []
    for i, p in enumerate(pnts)
        pnt = arcpy.PointGeometry(arcpy.Point(p.x, p.y, ID = i)) # Convert the Shapely point to arcpy point
        pourpt = sa.SnapPourPoint(pnt, flowacc, 10000) #Snap point to high flow acc cell
        if arcpy.Exists(pourpt):
            ws = sa.Watershed(flowdir, pourpt) # Calculate watershed
            if arcpy.Exists(ws):
               out = os.path.join(direc, "poly_%i"%i)
               poly = arcpy.RasterToPolygon_conversion(ws, out) #Convert to polygon
               res.append(poly[0]) # Put the polygon in the results list for this set of points
            else polylist = "NoWS"
        else polylist = "NoPourPt"

    return res

The parallel function remains the same:

def watershed_pll(mod, proc=6):
    """
    Calculate the watershed for each station point using parallel processing 
    All relevant data is held in the mod object
    """
    pool = Pool(processes = proc)

    # Iterate over each feature of the drainage path and submit the list of pour points to the worker function
    jobs = []
    for key, val in mod.stations.geometry.iteritems():
        jobs.append((key, pool.apply_async(watershed, (val.points, key, mod.flowdir,
                                           mod.flowacc, rdict))))

    pool.close()
    pool.join()
    return jobs

Sometimes it runs perfectly, other times I get one of various errors such as the initial one above, or also:

FATAL ERROR (INFADI)
MISSING DIRECTORY

and:

ERROR 010088: Invalid input geodataset (Layer, Tin, etc.).]

and:

 ERROR 010050: Cell size is not set.

Which is odd as I explicitly set the cell size within the function...

When it works and when it doesnt seems to be hit and miss/random. Such errors lead me to believe that arcpy is still trying to delete stuff behind the scenes. Is there any way to explicitly prevent it from doing so?


I wonder if my problem could be related to one of two things:

  1. Environments/folders getting mixed up. The input data is passed as complete filepaths to some raster datasets which are outside of the scratchWorkspace (which is set locally for each process) where intermediate/output data is created. However, I have noticed that Arc may make folders (typically and 'info' folder) in the directories of the input data. Why is that? Can it be prevented, or can I prevent Arc from then trying to delete it?

  2. Are there any potential problems with accessing the input raster data sets at the same time? I.e each process will be attempting to open and read from the flow direction and flow accumulation rasters which are passed into the function.

  • Have you read the ESRI posts about Multiprocessing? blogs.esri.com/esri/arcgis/2012/09/26/… – klewis Feb 25 '14 at 15:11
  • Absolutely, and none of it addresses my problem. ESRI's blogs are for very simple implementations of multiprocessing that are not really applicable to larger programs. If I remove ESRI-related functions in the above functions and run a 'dummy' function, it works fine - the problem is definitely something arcpy is doing behind the scenes. – James Feb 25 '14 at 15:29
  • 1
    You might try using something like Process Monitor to see what directories are being modified: technet.microsoft.com/en-us/sysinternals/bb896645.aspx – blah238 Feb 25 '14 at 18:06
  • i would start creating temp directory to C:\temp\process\randomnumber , where randomnumber is generated for each process\thread. Then at end when all processing is done i would read all data from process subdirectories to one big result set and save it. – simplexio Feb 26 '14 at 8:41
  • A unique temp directory is already created for each process using tempfile. There are no 'final' results written to file - the final result is a Polygon, so nothing to read. The temp directories are for intermeadiate data, which at the moment I do nothing with (I dont delete it) while I am debugging. – James Feb 26 '14 at 8:41
2

Some general steps for debugging this kind of problem...

  • Set the current workspace, and the scratch workspace to something random and specific to the process.
  • Use actual output files instead of variables. You will be able to see what has been created, and avoid the use of automatically generated file names.
  • Some spatial analyst functions (RasterToPolygon) need very short file names to work properly.
  • Some good help, but I dont see how using files instead of variables will make a difference...in fact, would that not increase the possibility of disk conflicts when writing? Surely using variables defined in RAM is safer for multiprocessing? – James Feb 26 '14 at 14:37
  • Behind the scene ArcGIS may be saving your variables to disk which could cause issues. By explicitly specifying this information you reduce this risk. It may make things slower, but you could re-enable once you know what the problem is. – Matthew Snape Feb 26 '14 at 15:45
  • I incorporated your advice, please see my edited post above – James Feb 27 '14 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.