I am using ArcGIS 10.4.1 and Network Analyst extension. As an IDE i use Spyder and Python version 2.7. I created a script where there is a method with the following arguments.

def ra(EndPoint, RoadRules, data, StationName, Store_shp, SortedStationStops_shp, StationDeliveries_shp, stationRoutes_Dic, BranchId, groupid)

This method is based on the given example in http://pro.arcgis.com/en/pro-app/tool-reference/network-analyst/make-route-analysis-layer.htm

A simplified part of the code where I am trying to use multiprocessing follows;

if __name__ == '__main__':
   groupids = list(stationRoutes_Dic.keys())
   print groupids

   func = partial(ra,EndPoint, RoadRules, routeDeliv, StationName, Store_shp, SortedStationStops_shp, StationDeliveries_shp, stationRoutes_Dic, branch_id)

   # declare number of cores to use, use 1 less than the max  
   cpuNum = multiprocessing.cpu_count() - 1 

   # Create the pool object  
   pool = multiprocessing.Pool(processes=cpuNum)  


I cannot make it work. I've tried many ways. When i run it i get "TypeError: can't pickle pyodbc.Row objects". Whilst when i debug it the logger shows that infinite workers are created and closed respectively. Through trial and error, i noticed that the more functions has my script the more workers are created.

The iterable variable is groupids and i use it to take the stops for each separate route.


It appears that the code stucks in def map, at the return:

    def map(self, func, iterable, chunksize=None):
       Equivalent of `map()` builtin
       assert self._state == RUN
       return self.map_async(func, iterable, chunksize).get()

i suspect that my problem has to do with my method's arguments. Their types are as following;

  1. EndPoint is a string
  2. RoadRules is a list
  3. data is a list
  4. StationName is a unicode
  5. Store_shp is a string (datasource of a shapefile)
  6. SortedStationStops_shp is a string (datasource of a shapefile)
  7. StationDeliveries_shp is a string (datasource of a shapefile)
  8. stationRoutes_Dic is a dictionary
  9. BranchId is a string


Below is the method that I am calling

def ra(EndPoint, RoadRules, data, StationName, Store_shp, SortedStationStops_shp, StationDeliveries_shp, stationRoutes_Dic, BranchId, groupid):
        print " "
        print "------>>>>ra"    


        #Set LOCAL variables
        matchesStops = []
        matchesRoutes = []

        #Check out the Network Analyst extension license

        #Set environment settings
        env.overwriteOutput = True
        env.outputCoordinateSystem = arcpy.SpatialReference("WGS 1984")
        arcpy.env.geographicTransformations = "GGRS_1987_To_WGS_1984"

        # Make a feature layer
        arcpy.MakeFeatureLayer_management(SortedStationStops_shp, "SortedStationStops")

        # Selection
        RouteStopsLayer=arcpy.SelectLayerByAttribute_management("SortedStationStops", "NEW_SELECTION", "Group_ID LIKE '"+GroupName+"%'")

        # Write the selected features to a new featureclass
        RouteStops=os.path.join(path, "04_InStationRouteStops_"+StationName+"_"+GroupName+".shp")
        arcpy.CopyFeatures_management(RouteStopsLayer, RouteStops)

        outNALayerName = "RA_StationRoute_"+StationName+"_"+GroupName+"_"+BranchId
        RouteStops_shp =os.path.join(path, "04_StationRouteStops_"+StationName+"_"+GroupName+"_"+BranchId+".shp")
        SingleRoute_shp =os.path.join(path, "04_StationRoute_"+StationName+"_"+GroupName+"_"+BranchId+".shp")

        #Set RA variables
        inNetworkDataset = Network_Dataset
        impedanceAttribute = "Minutes"
        timeUnits = "Minutes"
        distanceUnits = "Meters"
        distanceAttribute = "Length"
        inDepots = Store_shp
        inOrders = RouteStops
        outStopsFC =RouteStops_shp
        outRoutesFC =SingleRoute_shp
        outLayerFile = os.path.join(path, "RA_StationRoute_"+StationName+"_"+GroupName+".lyr")
        searchQuery = [["RoadNetwork", '"Access" <> -1'],["Greece_J18_ND_Junctions", ""]]

        ## Set Barriers
        if StationName =='ΒΣ' and GroupName.startswith('105'):
            Barriers_shp=os.path.join("C:\\Users\\tria\\Documents\\ArcGIS\\ACS_VRP\\BS_Cluster", "105barrier6.shp")
        if StationName =='ΒΣ' and GroupName.startswith('101'):
            Barriers_shp=os.path.join("C:\\Users\\tria\\Documents\\ArcGIS\\ACS_VRP\\BS_Cluster", "101barrier2.shp")
        if StationName =='ΣΛ':
            Barriers_shp=os.path.join("C:\\Users\\tria\\Documents\\ArcGIS\\ACS_VRP\\Salamina", "salaminaBarrier2.shp")

        ## Create a new Route layer. Optimize on Minutes, but compute the distance travelled by accumulating the Length attribute.
        #SYNTAX MakeRouteLayer_na (in_network_dataset, out_network_analysis_layer, impedance_attribute, {find_best_order}, {ordering_type}, {time_windows}, {accumulate_attribute_name}, {UTurn_policy}, {restriction_attribute_name}, {hierarchy}, {hierarchy_settings}, {output_path_shape}, {start_date_time})
        outRouteResultObject = arcpy.na.MakeRouteLayer(inNetworkDataset, outNALayerName, impedanceAttribute, "FIND_BEST_ORDER", ""+EndPoint+"", "", ["Minutes", "Length"], "ALLOW_DEAD_ENDS_AND_INTERSECTIONS_ONLY", RoadRules, "USE_HIERARCHY", "", "TRUE_LINES_WITH_MEASURES","")
        #Get the layer object from the result object. The route layer can now be referenced using the layer object.
        outNALayer = outRouteResultObject.getOutput(0)
        #print outNALayer

        #Get the names of all the sublayers within the route layer.(used in field mapping to parametized)
        subLayerNames = arcpy.na.GetNAClassNames(outNALayer)
        #print subLayerNames

        #Store the layer names that we will use later
        stopsLayerName = subLayerNames["Stops"]
        routesLayerName = subLayerNames["Routes"]
        pointBarriersLayerName = subLayerNames["PolylineBarriers"]# {u'Barriers': u'Point Barriers', u'Routes': u'Routes', u'PolylineBarriers': u'Line Barriers', u'Stops': u'Stops', u'PolygonBarriers': u'Polygon Barriers'} etc

        #Before loading locations as route stops, set up field mapping. Map the "Group_ID" field from the input data to the RouteName property in the Stops sublayer,
        # which ensures that each unique Group_ID will be placed in a separate route. Matching Group_ID from different stop files will end up in the same route.
        fieldMappingsStore = arcpy.na.NAClassFieldMappings(outNALayer, stopsLayerName)
        fieldMappingsStore["RouteName"].mappedFieldName = "Group_ID" #correlates fields ie naStops attribute table with inStores attribute table
        fieldMappingsStops = arcpy.na.NAClassFieldMappings(outNALayer, stopsLayerName)
        fieldMappingsStops["RouteName"].mappedFieldName = "Group_ID" #correlates fields ie naStops attribute table with inStores attribute table
        fieldMappingsBR = arcpy.na.NAClassFieldMappings(outNALayer, pointBarriersLayerName)
        fieldMappingsBR["BarrierType"].defaultValue = 0 #if it is restriction otherwise 1

        if StationName =='ΒΣ' and GroupName.startswith('105') or StationName =='ΒΣ' and GroupName.startswith('101') or StationName =='ΣΛ':
            #Figure out the layer's impedance attribute
            solver_props = arcpy.na.GetSolverProperties(outNALayer)
            impedance = solver_props.impedance
            fieldMappingsBR["Attr_" + impedance].defaultValue = 1.4
            #Load restrictions features
            arcpy.na.AddLocations(outNALayer, pointBarriersLayerName, Barriers_shp, fieldMappingsBR)

        ##Add locations as Stops. The same field mapping works for both input feature classes because they both have a field called "Group_ID"
        #AddLocations_na (in_network_analysis_layer, sub_layer, in_table, field_mappings, search_tolerance, {sort_field}, {search_criteria}, {match_type}, {append}, {snap_to_position_along_network}, {snap_offset}, {exclude_restricted_elements}, {search_query})
        arcpy.na.AddLocations(outNALayer, stopsLayerName, inDepots, fieldMappingsStore, "300 meters", "FID", [["J18_ND", "MIDDLE"], ["J18_ND_Junctions", "NONE"]], "MATCH_TO_CLOSEST", "CLEAR", "NO_SNAP", "", "", "")
        arcpy.na.AddLocations(outNALayer, stopsLayerName, inOrders, fieldMappingsStops, "300 meters", "FID", [["J18_ND", "SHAPE"], ["J18_ND_Junctions", "NONE"]], "MATCH_TO_CLOSEST", "APPEND", "SNAP", "1", "EXCLUDE", searchQuery)

        #Solve the route layer.

        # Get the output Routes sublayer and save it to a feature class
        RoutesSubLayer = arcpy.mapping.ListLayers(outNALayer, routesLayerName)[0]
        StopsSubLayer = arcpy.mapping.ListLayers(outNALayer, stopsLayerName)[0]

        #Save the solved networkAnalyst layer as a layer file on disk (ok)
        arcpy.management.CopyFeatures(RoutesSubLayer, outRoutesFC)
        arcpy.management.CopyFeatures(StopsSubLayer, outStopsFC)
        matchesRoutes.append (outRoutesFC)
        matchesStops.append (outStopsFC)

        print "Routing completed successfully"

            ##JOIN TABLES (resolved Stops (Stops_shp) with points inforrmation (StationDeliveries_shp))

        ExpStops_shp =os.path.join(path, "042_StationExpStops_"+StationName+"_"+BranchId+".shp")
        Stops_shp=os.path.join(path, "040_StationALLStops__"+StationName+"_"+BranchId+".shp")
        Route_shp =os.path.join(path, "041_StationALLRoutes_"+StationName+"_"+BranchId+".shp")
        arcpy.Merge_management(matchesStops, Stops_shp)
        arcpy.Merge_management(matchesRoutes, Route_shp)
        del matchesStops[:]
        del matchesRoutes[:]

        #Call method to eliminate redStops

        out_feature_class = ExpStops_shp

        # Replace a layer/table view name with a path to a dataset (which can be a layer file) or create the layer/table view within the script
        arcpy.SpatialJoin_analysis(target_features, join_features, out_feature_class, join_operation="JOIN_ONE_TO_ONE", join_type="KEEP_ALL", match_option="CLOSEST", search_radius="500 Meters", distance_field_name="")

        ##Sorting by route name
        in_dataset = ExpStops_shp
        sortedExpStops_shp =os.path.join(path, "043_StationExpStopsSorted_"+StationName+"_"+BranchId+".shp")
        out_dataset = sortedExpStops_shp
        # Order features first by DISTANCE (NEAR_RANK)
        sort_fields = [["RouteName", "ASCENDING"]]
        # execute the function
        env.overwriteOutput = True
        arcpy.Sort_management(in_dataset, out_dataset, sort_fields)

        ##Create groupRouteidsDic (needed in update routingpaths)
        listGroupIDs=unique_values(sortedExpStops_shp, ['Group_ID','Route_ID']) #Group_ID because RouteName has now more than one  bins
        RouteId= ','.join(str(e) for e in list(groupRouteidsDic.values()))
        print "RouteId", RouteId

        listGroupBins=unique_values(sortedExpStops_shp, ['Group_ID','Bin_No'])
        print "groupBinDic", groupBinDic

        fields = ['Group_ID', 'RouteName', 'Route_ID', 'Bin_No']
        with arcpy.da.UpdateCursor(sortedExpStops_shp, fields) as cursor:
              for row in cursor:
                    row[2] =groupRouteidsDic[int(row[1])]
                    row[3] =groupBinDic[int(row[1])]

  except Exception as e:
        # If an error occurred, print line number and error message
        import traceback, sys
        tb = sys.exc_info()[2]
        print "An error occured on line %i" % tb.tb_lineno
        print str(e)

  #return ExpStops_shp, groupRouteidsDic, RouteId, Route_shp, sortedExpStops_shp, Stops_shp


I followed Hornbydd instructions and it worked. Also i had to remove from method's inputs an argument that i was getting with cur.fetchall() because it was producing "TypeError: can't pickle pyodbc.Row objects". The main changes are shown below:

        # Make a feature layer
        arcpy.MakeFeatureLayer_management(SortedStationStops_shp, "SortedStationStops"+str(groupid))

        # Selection
        RouteStopsLayer=arcpy.SelectLayerByAttribute_management("SortedStationStops"+str(groupid), "NEW_SELECTION", "Group_ID LIKE '"+GroupName+"%'")
  • What exactly is the error? Can you elaborate on "I cannot make it work"
    – KHibma
    Jan 8, 2019 at 13:38
  • The method works perfectly when i dont use multiprocessing. When i use multiprocessing it doesnt execute the method. I believe that it has to do with the fact that it has more than one arguments
    – sotokan80
    Jan 8, 2019 at 15:07
  • I would suspect the issue is with your ra() function. As you don't actually show us what is happening in that function I suspect its failing, possibly thread locking and silently passing up and out to the main() function. Have a look at my article I posted on geonet. Until you edit your question and show what is being done in the worker function no one has a chance in solving this for you...
    – Hornbydd
    Jan 8, 2019 at 16:49
  • Very nice post.One of the errors that i faced was "pyodbc.raw objects cannot be pickled" so i am guessing that it has to do with your clarification that the full path of the shapefile must be given. So suppose by using .dataSource do i get the full path? i.e Store_shp.dataSource
    – sotokan80
    Jan 8, 2019 at 19:31
  • I'll say it again... Until you edit your question and show what is being done in the worker function (ra()) no one has a chance in solving this for you because it may have nothing to do with your input parameters...
    – Hornbydd
    Jan 9, 2019 at 15:20

1 Answer 1


Now I can see your worker function, one immediate issue that leaps out is that your layer names are not unique and this will upset things, so this line needs to be adjusted

arcpy.MakeFeatureLayer_management(SortedStationStops_shp, "SortedStationStops")

Refer to my document on geonet and you will see that I create unique layers by attaching the id that is being fed in into the worker function.

I have not checked your code fully but outputs must have unique full file path names. So if for example you are creating a temporary dataset that ultimately feeds into the final output which is what you actually want then the temporary dataset must have unique names. You cannot keep writing to c:\temp\mytempdata.shp for example.

So double check your code and anything that is currently common or being reused must all be unique. That's the issue I have had when trying to use multiprocessing.

How that affects the output of network analyst which creates predefined layer names I am unsure?

Also you don't check in the extension, I would do that in the finally part of your try/except.

  • SortedStationStops_shp is a shapefile that has all the stops of a station. Each station has a number of routes. those routes are in the iterable variable named groupids(i.e [101, 102, 103, 104]). So i am using arcpy.MakeFeatureLayer_management in order to make a selection and create new shapefiles of stops one for each route. the new shapefiles will be unique since they look like "RouteStops=os.path.join(path, "04_InStationRouteStops_"+StationName+"_"+GroupName+".shp")" in the beggining of the method i assign GroupName=str(groupid) where groupid is the iterable variable in the method ra.
    – sotokan80
    Jan 9, 2019 at 20:25
  • Do you believe that somehow it cannot make 4 selections simultaneously by using .MakeFeatureLayer_management and then creating 4 times a RouteStopsLayer=arcpy.SelectLayerByAttribute_management("SortedStationStops", "NEW_SELECTION", "Group_ID LIKE '"+GroupName+"%'") (one for each route)? Furthermore i check in the network analyst extension at the beginning of the code.
    – sotokan80
    Jan 9, 2019 at 20:28
  • Easy one first. you check OUT the Extension but you never check IN the extension. Not sure this really affects things but makes your code cleaner.
    – Hornbydd
    Jan 9, 2019 at 22:30
  • 1
    When you run Make FeatureLayer your layer is always called "SortedStationStops". I don't believe this will work (well it never did for me) they need to be uniquely named, hence I attach the ID to it to make them unique names. Your subsequent call to SelectLayerByAttribute is selecting features by group_id so I believe you need unique names to ensure unique selections.
    – Hornbydd
    Jan 9, 2019 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.