I am trying a script that makes use of multiprocessing and a number of tools within arcpy. The script works only if I write all input, intermediate, and output from child multiprocessing executions to the harddisk. Using the "in_memory" workspace as I had intended produces the feature class "does not exist or is not supported" error.

I've read over what I could find on problems with arcpy and multiprocessing. I've read several pages that specify that writing to the harddisk is necessary.

Is this true?

This seems like a horrible limitation of arcpy where combined with multiprocessing and I'm in disbelief that ESRI would let such a problem persist.

Can someone suggest a workaround?

closed as off-topic by PolyGeo Nov 18 '16 at 7:02

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking help to debug/write/improve code must include the desired behavior, a specific problem or error and the shortest code necessary to reproduce it in the question itself. Providing a clear problem statement and evidence of a code attempt will help others to help you. See: How to create a Minimal, Complete, and Verifiable example." – PolyGeo
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    I don't have an answer, but want to point out that shared memory is sort of antithetical to using multiprocessing (see docs.python.org/2/library/…) so don't beat up Esri too much about it. Synchronization probably comes cheap with on-disk workspaces and is more fraught with in-memory ones. – sgillies Dec 5 '14 at 15:36
  • Thanks for taking the time to help. Can you explain a bit more? If I pass, for example, a feature class saved in the in_memory workspace to a child process, that argument is supposed to be copied in the background by the multiprocessing object, no? So the argument isn't actually shared. Or at least, that was my understanding. Is my understanding incorrect? – gcolumbus Dec 5 '14 at 16:09
  • If I wanted to share a feature class between child processes then I'd need to use one of the multiprocessing module's built-in sharing mechanisms (e.g. the multiprocessing.Queue() object). I'm not trying to share, however. I simply want to pass a feature class to a child process that has be defined as existing within the in_memory workspace. – gcolumbus Dec 5 '14 at 16:12
  • 1
    Would you be able to edit your question to include a code snippet that demonstrates your problem, please? I think it would also be worth moving some of your comments into a revised version of your original question. – PolyGeo Apr 11 '15 at 10:16