2

I use the Python multiprocessing module to speed up a function (get_craters_for_buffering_BCC) in my script. In this function, I want to identify points (passed as a list: crater_features_list) within a certain geodesic distance from a polygon (passed as a polygon: geom) using OGR and Shapely.

The idea is to split the original list (crater_features_list) into n parts (crater_features_list_splitted[n]), to pass the respective parts to individual processes (processes[n]) and to pass the results from the function using multiprocessing Queues (out_q, out_q_2, out_q_3).

The procedure works fine when I execute the script in Linux OS. However, when I execute the script in Windows OS, I get the following error message:

Traceback (most recent call last):
  File "C:\Christian\Projects\2016-07 Geodesic Polygon Buffer\jo\Geodesic_Buffering_Areas18_2-Multicore.py", line 879, in <module>
    processes[process_count2].start()
  File "C:\Python27\ArcGIS10.4\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\ArcGIS10.4\lib\multiprocessing\forking.py", line 280, in __init__
    to_child.close()
IOError: [Errno 22] Invalid argument

This is my script:

def get_craters_for_buffering_BCC(out_q, out_q_2, out_q_3, crater_features_list_part, geom): 

    """ ... determination of points within a certain distance from geom Polygon: The results 
    are saved in craters_inside_area and craters_within_range lists ... """

    craters_for_counting_list = craters_inside_area + craters_within_range

    # passing information to main script using Queues

    out_q.put(craters_for_counting_list)
    out_q_2.put(len(craters_within_range))
    out_q_3.put(len(craters_inside_area))

if __name__ == '__main__':

    multiprocessing.freeze_support()

    # get geometry from file

    path_to_area_file = "path/to/shapefile.shp"
    driver = ogr.GetDriverByName('ESRI Shapefile')
    area_file = driver.Open(path_to_area_file)
    layer = area_file.GetLayer()
    area_feature = layer.GetFeature(1)
    geom = area_feature.GetGeometryRef()

    # crater_features_list was created at a different point of the script. 
    # Here, the list is splitted in 20 parts for multiprocessing 
    # (or 21 parts - depending on the number of features)

    crater_features_list_splitted = [crater_features_list[v:v + (len(crater_features_list)/20)] for v in xrange(0, len(crater_features_list), (len(crater_features_list)/20))]

    out_q = multiprocessing.Queue()
    out_q_2 = multiprocessing.Queue()
    out_q_3 = multiprocessing.Queue()
    processes = dict()
    process_count = 0

    # definition of process tasks (processes[n]) for the individual lists 

    for crater_features_list_part in crater_features_list_splitted:
        processes[process_count] = multiprocessing.Process(target = get_craters_for_buffering_BCC, args=(out_q, out_q_2, out_q_3, crater_features_list_splitted[process_count], geom))
        process_count += 1

    # execution of multiprocessing processes

    process_count2 = 0
    for crater_features_list_part in crater_features_list_splitted:
        processes[process_count2].start() # This is where the error occures
        process_count2 += 1

The error occures only on Windows OS when I try to start the multiprocessing process:

processes[process_count2].start()

I have only little experience using multiprocessing. I tried different workarounds such as using multiprocessing.Pool, passing only string arguments to the function and calling the script from the Windows command line. They all resulted in the same error message. How can I use the multiprocessing script on Windows OS? I use Python 2.7 (32 bit on Windows, 64 bit on Linux).

closed as off-topic by PolyGeo Aug 26 '16 at 5:36

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions relating to general Information Technology, with no clear GIS component, are off-topic here, but can be researched/asked at Stack Overflow (software development), Super User (computing hardware and software) and Database Administrators (relational databases)" – PolyGeo
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    You might get a faster response if you post your question in StackOverflow since this is focused on Python multiprocessing. – Joseph Aug 25 '16 at 13:25