1

I am trying to use multiprocessing to process intersects between polygons and points for a list of states and territories (the data sets for the points are large, so my hope was that the multiprocessing would speed the process up). But there seems to be an issue. Suppose I was just doing DC and PR. The intersect first produce an empty table output for PR and then throws the following error for PR:

Traceback (most recent call last):
  File "C:\Projects\hh_intersects\just_hh_intersects.py", line 68, in <module>
    pool.map(multi_proc, state_list)
    ....
arcgisscripting.ExecuteError: ERROR 000210: Cannot create output 
T:\...\civis_hh_0520.gdb\PR_hh_intersects
Failed to execute (Intersect).

I'm not sure what I am doing incorrectly here. Any ideas on how to fix this?

import arcpy
import os
import time
import multiprocessing as mp
from multiprocessing import Pool, cpu_count

# Paths for hh_pts_gdb, input_polys_dir, and output_gdb

state_list = ['DC', 'PR']
start = time.time()
p_count = cpu_count()

# Functions
def multi_proc(state):
    state_start = time.time()
    arcpy.env.workspace = output_gdb

    # Make output based on state
    output = os.path.join(output_gdb, "{}_intersect".format(state))

    # Pts
    pt_name = "{}_hh_pts".format(state)
    hh_pts = os.path.join(hh_pts_gdb, pt_name)

    # Polys
    st_poly = os.path.join(input_polys_dir, "{}/{}_UnionOutput.gdb/{}_Union".format(state, state, state))

    # Intersect selected pts with polys
    arcpy.Intersect_analysis([hh_pts, st_poly], output, "ALL")
    print("Output row count for {}".format(state))
    counter = arcpy.GetCount_management(output)
    print(counter)
    state_end = time.time()
    print("{} Process duration: ".format(state))
    print(state_end - state_start)

if __name__ == '__main__':
    pool = mp.Pool(p_count)
    pool.map(multi_proc, state_list)
    end = time.time()
    print("Total Process duration: ")
    print(end - start)

*Note- I have run the following by itself (outside of the multiprocessing) and it worked with state = "DC": arcpy.Intersect_analysis([hh_pts, st_poly], output, "ALL").

0
1

It looks like you are working with data in a geodatabase. Multiprocessing doesn't work on feature classes in a geodatabase because each update acquires a lock.

According to https://www.esri.com/arcgis-blog/products/arcgis-desktop/analytics/multiprocessing-with-arcgis-approaches-and-considerations-part-1/:

...will not work with feature classes in a file geodatabase because each update must acquire a schema lock on the workspace. A schema lock effectively prevents any other process from simultaneously updating the FGDB.

2
  • Argghhh! Is that specifically the output? What if I used arcpy.MakeFeatureLayer_management() on the inputs first? Would that work?
    – gwydion93
    May 21 '20 at 14:16
  • That would be the output I would expect because that is what it throws when it tries to access data that has a lock on it. I think that should work. If not, you could try to get to the same end-point using CopyFeatures to "in_memory".
    – Pdavis327
    May 21 '20 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.