Multiprocessing ERROR 000210 Cannot create output 'name' Failed to execute Intersect

Question

I am trying to use multiprocessing to process intersects between polygons and points for a list of states and territories (the data sets for the points are large, so my hope was that the multiprocessing would speed the process up). But there seems to be an issue. Suppose I was just doing DC and PR. The intersect first produce an empty table output for PR and then throws the following error for PR:

Traceback (most recent call last):
  File "C:\Projects\hh_intersects\just_hh_intersects.py", line 68, in <module>
    pool.map(multi_proc, state_list)
    ....
arcgisscripting.ExecuteError: ERROR 000210: Cannot create output 
T:\...\civis_hh_0520.gdb\PR_hh_intersects
Failed to execute (Intersect).

I'm not sure what I am doing incorrectly here. Any ideas on how to fix this?

import arcpy
import os
import time
import multiprocessing as mp
from multiprocessing import Pool, cpu_count

# Paths for hh_pts_gdb, input_polys_dir, and output_gdb

state_list = ['DC', 'PR']
start = time.time()
p_count = cpu_count()

# Functions
def multi_proc(state):
    state_start = time.time()
    arcpy.env.workspace = output_gdb

    # Make output based on state
    output = os.path.join(output_gdb, "{}_intersect".format(state))

    # Pts
    pt_name = "{}_hh_pts".format(state)
    hh_pts = os.path.join(hh_pts_gdb, pt_name)

    # Polys
    st_poly = os.path.join(input_polys_dir, "{}/{}_UnionOutput.gdb/{}_Union".format(state, state, state))

    # Intersect selected pts with polys
    arcpy.Intersect_analysis([hh_pts, st_poly], output, "ALL")
    print("Output row count for {}".format(state))
    counter = arcpy.GetCount_management(output)
    print(counter)
    state_end = time.time()
    print("{} Process duration: ".format(state))
    print(state_end - state_start)

if __name__ == '__main__':
    pool = mp.Pool(p_count)
    pool.map(multi_proc, state_list)
    end = time.time()
    print("Total Process duration: ")
    print(end - start)

*Note- I have run the following by itself (outside of the multiprocessing) and it worked with state = "DC": arcpy.Intersect_analysis([hh_pts, st_poly], output, "ALL").

Answer 1

It looks like you are working with data in a geodatabase. Multiprocessing doesn't work on feature classes in a geodatabase because each update acquires a lock.

According to https://www.esri.com/arcgis-blog/products/arcgis-desktop/analytics/multiprocessing-with-arcgis-approaches-and-considerations-part-1/:

...will not work with feature classes in a file geodatabase because each update must acquire a schema lock on the workspace. A schema lock effectively prevents any other process from simultaneously updating the FGDB.