Arcpy and multiprocessing not working

Question

I want to utilize multiprocessing within my arcpy scripts but my first test has run into a problem which I don't understand. I'm running the script in the python window:

import arcpy
from multiprocessing import Pool

def f(x):
    return x*x
pool = Pool(processes=4)
print pool.map(f,[1,2,3,4])

As soon as I assign the processes, ArcMap open four new instances and the script does not print the values of x as expected (the above code is from an example I found). The new instances of ArcMap initialize but then close with the following error:

What am I doing wrong? Is there a setting/patch I've ignored? I'm using ArcMap Desktop 10.0 SP5 build 4400 and I have also tried the code on 10.3.1.4959.

Answer 1

Multiprocessing is trying to use ArcMap.exe instead of python.exe to run the child processes.

Either disable running the script 'in process' or use multiprocessing.set_executable(os.path.join(sys.exec_prefix, 'python.exe'))

Note: don't use sys.executable to get the path to python.exe if running in-process as it will be pointing to ArcMap.exe, use sys.exec_prefix instead:

>>> import os, sys
>>> print sys.executable
C:\Program Files (x86)\ArcGIS\Desktop10.2\bin\arcmap.exe
>>> print os.path.join(sys.exec_prefix, 'python.exe')
C:\Python27\ArcGIS10.2\python.exe