Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I was wondering if anyone else in the community here has attempted to use multi-processing for spatial analyses. Namely I am trying to iterate through a series of rasters, create a multiprocessing job for each and run them through a number of geoprocessing steps within one def function. Something along the lines of

def net(RasterImage, OutFolderDir):
    arcpy.env.overwriteOutput = True  
    arcpy.env.workspace = OutFolderDir 
    DEM_Prj = DEM_Prj.tif

    try:
        arcpy.ProjectRaster_management(RasterImage, DEM_Prj....
        FocalStatistics(DEM_prj....)
        ...

if __name__ == '__main__':  
    InputFolder = r'C:\test\somepath'  
    Output = r'C:\test\somepath2'  
    arcpy.env.workspace = InputFolder  
    arcpy.env.scratchWorkspace = r'C:\test.gdb'    

    fcs = arcpy.ListRasters('*')
    pool = multiprocessing.Pool(4)   
    jobs = []                 
    for fc in fcs:
        rIn = os.path.join(InputFolder,fc)
        rOut = os.path.join(Output,fc[:-4])
        jobs.append(pool.apply_async(net,(rIn, rOut)))    

Now the multiprocessing does run, usually for the first batch! However, I keep running into several different errors when attempting several datasets(more than 4 files - i.e. 4 core multiprocessing) including:

ERROR 010302: Unable to create the output raster: C:\somepath\sr6f8~1\FocalSt_srtm1
ERROR 010067: Error in executing grid expression.
Failed to execute (FocalStatistics).

and

ERROR 999999: Error executing function.
Failed to copy raster dataset
Failed to execute (ProjectRaster)

Notice in the first error the strange folder that is created (in the OutFolderDir location) associated with the focal statistics that nearly creates an exact replica of the final output.

My question is based off your experience is it impossible to create several step geoprocessing within one multiprocessing function? Or do I need to tile these steps into their individual geoprocessing steps?

Kind Regards, Bjorn

UPDATE

Still encoutering similar errors - moving the import functions to the def function has shown that

import arcpy 
from arcpy.sa import *

cannot create an output with an added syntax warning that of import * is not allowed.

UPDATE #2

I know this is a late reply but I thought it might benefit someone else for future reference to my workaround that allows multiprocessing to work with arcpy. The main problem I found after returning to this problem is not the competition of the arcpy modules but rather competition over the scratchWorkspace that the ArcObjects utilize to save the temporary files. Therefore consider running a counter into the multiprocessing parsing argument to make a unique scratchWorkspace for each process i.e.

Counter = 0 
for fc in fcs:              
    rIn = os.path.join(InputFolder,fc)              
    rOut = os.path.join(Output,fc[:-4])                    
    jobs.append(pool.apply_async(net,(rIn, rOut,Counter)))            
    Counter += 1

Then in the main function make a specific temporary directory and assign an unique scratchWorkspace to each multiprocessing task.

def main(RasterImage,OutFolderDir,Counter)      
    TempFolder = os.path.join(os.path.dirname(OutFolderDir),'Temp_%s'%  (Counter))      
    os.mkdir(TempFolder)      
    arcpy.scratchWorkspace = TempFolder      
    ... 

Hope that helps and thanks to Ragi for the inital suggestion to use separate temp workspaces - still baffled by why it originally did not work.

Additional Resources

ESRI Multiprocessing Blog

Python,Gis and Stuff Blog

share|improve this question
    
This suggestion is so crude I don't want to formalize it in a reply, but have you considered running ArcGIS in multiple virtual machines simultaneously? (You might need a separate installation in each VM, each with its own directory structure.) Another radical thought is to farm out some of the processing: for instance, focalstats could be done in R. These are not good suggestions for general-purpose work, because they may be more trouble than they're worth, but when you can save hours at a time, repeatedly, the effort could pay off. –  whuber Dec 14 '11 at 16:37
add comment

2 Answers

up vote 3 down vote accepted

Each IWorkspace connection (i.e each database connection) has thread affinity. Two threads cannot share the same workspace. You can have one thread own the resource and then sync the access, but if you are going to be using straight gp functions, then that is even not an option.

The easiest (lame) way is to create separate processes and then do multi process synchronization (as opposed to multithread synchronization). Even then you should be aware of the underlying workspace type. if you are not using arcsde (a multi-user datasource) you will probably use a single user datasource (like personal or filegdb). Then remember that means only one process can write at a time! The typical (lame) synchronization for these scenarios is that each parallel process writes to a different temp workspace and then you merge it all in to your destination workspace in a single process.

share|improve this answer
    
Good suggestions... Actually, although I have not added it to this post, I am creating a new folder based on the raster image name and setting the workspace for each process to that specific directory. These are seperate file directories for each raster image and not seperate geodatabases (do I need that?). I had then planned to use a simple os.walk function to find all those files i needed move them to my desired file geodatabase. –  BJEBN Dec 15 '11 at 13:41
    
Are you only doing raster operations? Are there any threads or processes reading/writing to the same geodatabase at the same time? –  Ragi Yaser Burhum Dec 15 '11 at 16:26
    
Hi, sorry I might have been a bit unclear with the previous statement. Only raster operations (reproject, focalstats, reclassify etc...) and all those geoprocessing steps are done in a sequential order (or need to be) for each raster image. These raster images are saved to a unique folder workspace. All the original rasters are reading from the same directory (not the same image though) as that creates the individuals jobs to be sent. –  BJEBN Dec 16 '11 at 7:55
    
After rethinking somewhat I tried to specify a specific scratch workspace as well for each image. The DEM are being correctly projected however this has produced a new error at the focalstats stage -- "type <Raster> is not supported". I've tried to specify the entire directory address but with no luck. I've loaded the projected rasters into arcgis with no problem. –  BJEBN Dec 16 '11 at 17:36
    
Well, that means you are moving forward. For the focalstats, it depends how it is implemented internally. If it is a new implementation, it can take a scratchworkspace (i.e. a Geodatabase). However, if it is one of those functions that has not been upgraded yet (!?!?!) the workspace that it allows may only be a folder. For that particular GP function, specify only a folder (keep the scratchworkspace for the rest) and see what happens. –  Ragi Yaser Burhum Dec 16 '11 at 19:02
show 3 more comments

You have several threads competing for the same resource.

Try moving your 'import arcpy' statement into the target of the multiprocessing. You'll ensure arcpy is working with it's own set of environment variables and memory.

It sounds absurd, but even though you are setting environment variables in the Multiprocess target method, python is still using a shared memory space to manage the arcpy module and therefore any variables you set.

Arcpy isn't thread safe. It was always intended to be used within a single process. But there are workarounds.

UPDATE

Sorry for the ambiguity. My suggestion was to import arcpy within the target for new process.

def _multiprocessing_target(args):
    import arcpy
    ...code
share|improve this answer
    
Hi, thanks for your advice... although I appear to still have problems. When you refer to'import arcpy into the target of the multiprocessing' do you imply under the if__name... statement or actually within the def function. As I thought importing in the def function was invalid. –  BJEBN Dec 14 '11 at 9:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.