I have several shapefiles of the entire Amazon rainforest each with millions of polygons (they are fairly detailed and take some time to draw on ArcMap). I need each of these shapefiles to be spatially joined with another shapefile called landuseA (which, itself, is a very big file). Given how computer intensive this task is, I would like to use the multiprocessing module on python to improve performance. How do I do this?

I have read the blog post by Esri on the multiprocessing module, but I cannot figure out how I can use their example to write a script that will use multiple processors to execute a spatial join. Blog post is here: http://blogs.esri.com/esri/arcgis/2012/09/26/distributed-processing-with-arcgis-part-1/

If you have other suggestions for how this task might be better accomplished, feel free to share that too.

closed as unclear what you're asking by PolyGeo Mar 13 '16 at 22:29

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    If the shapefiles are large, you should be using file geodatabase to prevent hitting file size limits. You should edit the question to contain a link to the referenced blog. – Vince Mar 23 '15 at 11:37
  • 1
    So do you have "several shapefiles" or "several FGDB feature classes"? It changes the problem significantly. – Vince Mar 23 '15 at 16:35
  • 1
    multiprocessing or multithreading? take some time to familiarize yourself with the difference between those two and I believe the answer will come – nickves Mar 23 '15 at 16:47
  • 4
    You can't use several processes to execute a spatial join, since the implication on a join is that it's to be used by a single process. You can use one process to partition the dataset into working subfolders, then use multiprocessing to run UNION operations in the subfolders, then once the jobs are complete merge the result. 90% of the work of using multiprocessing is balancing the load so that the effort of organization gains you more time than just letting a job run overnight or over a weekend. – Vince Mar 23 '15 at 17:03
  • 1
    With no answer in nearly a year I think you should edit your question to incorporate your learnings from comments and re-clarify what it is that you want to ask now. – PolyGeo Mar 13 '16 at 22:29