I use the Python multiprocessing module to speed up a function (get_craters_for_buffering_BCC) in my script. In this function, I want to identify points (passed as a list: crater_features_list) within a certain geodesic distance from a polygon (passed as a polygon: geom) using OGR and Shapely.
The idea is to split the original list (crater_features_list) into n parts (crater_features_list_splitted[n]), to pass the respective parts to individual processes (processes[n]) and to pass the results from the function using multiprocessing Queues (out_q, out_q_2, out_q_3).
The procedure works fine when I execute the script in Linux OS. However, when I execute the script in Windows OS, I get the following error message:
Traceback (most recent call last): File "C:\Christian\Projects\2016-07 Geodesic Polygon Buffer\jo\Geodesic_Buffering_Areas18_2-Multicore.py", line 879, in <module> processes[process_count2].start() File "C:\Python27\ArcGIS10.4\lib\multiprocessing\process.py", line 130, in start self._popen = Popen(self) File "C:\Python27\ArcGIS10.4\lib\multiprocessing\forking.py", line 280, in __init__ to_child.close() IOError: [Errno 22] Invalid argument
This is my script:
def get_craters_for_buffering_BCC(out_q, out_q_2, out_q_3, crater_features_list_part, geom): """ ... determination of points within a certain distance from geom Polygon: The results are saved in craters_inside_area and craters_within_range lists ... """ craters_for_counting_list = craters_inside_area + craters_within_range # passing information to main script using Queues out_q.put(craters_for_counting_list) out_q_2.put(len(craters_within_range)) out_q_3.put(len(craters_inside_area)) if __name__ == '__main__': multiprocessing.freeze_support() # get geometry from file path_to_area_file = "path/to/shapefile.shp" driver = ogr.GetDriverByName('ESRI Shapefile') area_file = driver.Open(path_to_area_file) layer = area_file.GetLayer() area_feature = layer.GetFeature(1) geom = area_feature.GetGeometryRef() # crater_features_list was created at a different point of the script. # Here, the list is splitted in 20 parts for multiprocessing # (or 21 parts - depending on the number of features) crater_features_list_splitted = [crater_features_list[v:v + (len(crater_features_list)/20)] for v in xrange(0, len(crater_features_list), (len(crater_features_list)/20))] out_q = multiprocessing.Queue() out_q_2 = multiprocessing.Queue() out_q_3 = multiprocessing.Queue() processes = dict() process_count = 0 # definition of process tasks (processes[n]) for the individual lists for crater_features_list_part in crater_features_list_splitted: processes[process_count] = multiprocessing.Process(target = get_craters_for_buffering_BCC, args=(out_q, out_q_2, out_q_3, crater_features_list_splitted[process_count], geom)) process_count += 1 # execution of multiprocessing processes process_count2 = 0 for crater_features_list_part in crater_features_list_splitted: processes[process_count2].start() # This is where the error occures process_count2 += 1
The error occures only on Windows OS when I try to start the multiprocessing process:
I have only little experience using multiprocessing. I tried different workarounds such as using multiprocessing.Pool, passing only string arguments to the function and calling the script from the Windows command line. They all resulted in the same error message. How can I use the multiprocessing script on Windows OS? I use Python 2.7 (32 bit on Windows, 64 bit on Linux).