I am trying a script that makes use of multiprocessing and a number of tools within arcpy. The script works only if I write all input, intermediate, and output from child multiprocessing executions to the harddisk. Using the "in_memory" workspace as I had intended produces the feature class "does not exist or is not supported" error.

I've read over what I could find on problems with arcpy and multiprocessing. I've read several pages that specify that writing to the harddisk is necessary.

Is this true?

This seems like a horrible limitation of arcpy where combined with multiprocessing and I'm in disbelief that ESRI would let such a problem persist.

Can someone suggest a workaround?

  • 1
    I don't have an answer, but want to point out that shared memory is sort of antithetical to using multiprocessing (see docs.python.org/2/library/…) so don't beat up Esri too much about it. Synchronization probably comes cheap with on-disk workspaces and is more fraught with in-memory ones. – sgillies Dec 5 '14 at 15:36
  • Thanks for taking the time to help. Can you explain a bit more? If I pass, for example, a feature class saved in the in_memory workspace to a child process, that argument is supposed to be copied in the background by the multiprocessing object, no? So the argument isn't actually shared. Or at least, that was my understanding. Is my understanding incorrect? – gcolumbus Dec 5 '14 at 16:09
  • If I wanted to share a feature class between child processes then I'd need to use one of the multiprocessing module's built-in sharing mechanisms (e.g. the multiprocessing.Queue() object). I'm not trying to share, however. I simply want to pass a feature class to a child process that has be defined as existing within the in_memory workspace. – gcolumbus Dec 5 '14 at 16:12
  • 1
    Would you be able to edit your question to include a code snippet that demonstrates your problem, please? I think it would also be worth moving some of your comments into a revised version of your original question. – PolyGeo Apr 11 '15 at 10:16