Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have seen some difficult ways to do parallel processing, but I wonder if it is possible to simply execute multiple process of the same ArcPy script at the same time.

My script makes some changes to the default geodatabase, so I thought of making a geodatabase copy for each process.

I have updated the script to copy the shared resources between the processes, so it copies the geodatabase and the mxd's related to it.

I have made a test to parallelize, using this script:

pool = multiprocessing.Pool(2), [1, 2] , 1)


I noticed when I browse RAM and CPU use, every process consumes 200Mb. So, if I have 6 Gb of RAM, I think I have to exploit 5Gb RAM of it by enlarging the pool size to:

5000 / 200 = 25

So, to exploit the whole power of the machine, I think I should use 25 as pool size.

I need to know if this is the best manner, or how I could measure the efficiency of this parallelization.

This an example of the code that I'm trying to parallelize. The whole script contains 1500 lines of code almost like this one:

def dora_layer_goned():
    arcpy.Select_analysis( "layer_goned" , "layer_goned22" )
    arcpy.MakeFeatureLayer_management("layer_goned", "layer_goned_lyr")
    arcpy.SelectLayerByLocation_management("layer_goned_lyr" ,"WITHIN", "current_parcel"  ,  "" , "NEW_SELECTION")
    arcpy.SelectLayerByAttribute_management("layer_goned_lyr" , "SWITCH_SELECTION" )
    arcpy.Select_analysis("layer_goned_lyr" , "layer_goned_2_dora2" )
    arcpy.Clip_analysis("layer_goned_lyr" , "current_parcel_5m_2" , "layer_goned_2_dora" )
    arcpy.SelectLayerByAttribute_management("layer_goned_lyr" , "CLEAR_SELECTION" )
    arcpy.MakeFeatureLayer_management("layer_goned_2_dora2", "layer_goned_2_dora_lyr")
    arcpy.SelectLayerByLocation_management("layer_goned_2_dora_lyr" ,"INTERSECT", "layer_goned_2_dora_point"  ,  "" , "NEW_SELECTION")
    arcpy.FeatureVerticesToPoints_management("current_parcel","current_parcel__point", "ALL")
    arcpy.FeatureVerticesToPoints_management("carre_line","carre_line__point", "ALL")
    arcpy.SpatialJoin_analysis("carre_line__point"  , "current_parcel__point" , "carre_line__point_sj","JOIN_ONE_TO_ONE" , "KEEP_COMMON" , "" , "CLOSEST")
    arcpy.Append_management("current_parcel__point" , "carre_line__point_sj" , "NO_TEST") #
    arcpy.PointsToLine_management("carre_line__point_sj", "carre_line__point_sj_line", "id")
    arcpy.Buffer_analysis("carre_line__point_sj_line" , "carre_line__point_sj_line_buf" , 0.2)
    arcpy.Erase_analysis("layer_goned_2_dora2" , "carre_line__point_sj_line_buf"  , "layer_goned_2_dora_erz")
    arcpy.MultipartToSinglepart_management("layer_goned_2_dora_erz" , "layer_goned_2_dora_erz_mono")
    arcpy.MakeFeatureLayer_management("layer_goned_2_dora_erz_mono", "layer_goned_2_dora_erz_lyr")
    arcpy.SelectLayerByLocation_management("layer_goned_2_dora_erz_lyr" ,"SHARE_A_LINE_SEGMENT_WITH", "current_parcel"  ,  "" , "NEW_SELECTION")
    arcpy.SelectLayerByLocation_management("layer_goned_lyr" ,"CONTAINS", "layer_goned_2_dora_erz_lyr"  ,  "" , "NEW_SELECTION")
    arcpy.Append_management("layer_goned_2_dora_erz_lyr" , "layer_goned" , "NO_TEST") #
    arcpy.SelectLayerByAttribute_management("layer_goned_2_dora_erz_lyr" , "CLEAR_SELECTION" )
share|improve this question
What exactly are you trying to parallelize? Do you have any idea whether you are CPU or IO-bound? – blah238 Mar 21 '13 at 1:14
This isn't a duplicate of the suggested question. This one deals with arcpy, and the other one has a 1 barely useful answer concerned with ArcObjects. – Devdatta Tengshe Mar 21 '13 at 16:27
I'm trying to get you to describe the actual process you're trying to speed up. Not everything can be easily parallelized. – blah238 Mar 21 '13 at 17:37
I think the point I'm trying to make is that if you are CPU-bound that you should only use as many processes as you have CPUs (subtract one for background activity). Are you CPU or IO-bound? This is why I keep asking what your process is. If it's mostly just shifting data around on disk then there is likely to be no benefit to be gained from parallelization. – blah238 Mar 22 '13 at 0:03
up vote 4 down vote accepted

See this blog post, it should cover it

share|improve this answer
Here's another blog post about multiprocessing. – dmahr Mar 21 '13 at 19:45
@dmahr, weird, looks like (almost) the same thing. – blah238 Mar 21 '13 at 22:58

Just use the following function

def run_MultiPros(function, variables):
    """<function, variables> Execute a process on multiple processors.
    function(required) Name of the function to be executed.
    variables(required) Variable to be passed to function.
    Description: This function will run the given fuction on to multiprocesser. Total number of jobs is equal to number of variables.        
    pool = multiprocessing.Pool(), variables)
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.