2

I am using ArcGIS 10.4.1 and Network Analyst extension. As an IDE i use Spyder and Python version 2.7. I created a script where there is a method with the following arguments.

def ra(EndPoint, RoadRules, data, StationName, Store_shp, SortedStationStops_shp, StationDeliveries_shp, stationRoutes_Dic, BranchId, groupid)

This method is based on the given example in http://pro.arcgis.com/en/pro-app/tool-reference/network-analyst/make-route-analysis-layer.htm

A simplified part of the code where I am trying to use multiprocessing follows;

if __name__ == '__main__':
   groupids = list(stationRoutes_Dic.keys())
   print groupids

   func = partial(ra,EndPoint, RoadRules, routeDeliv, StationName, Store_shp, SortedStationStops_shp, StationDeliveries_shp, stationRoutes_Dic, branch_id)

   # declare number of cores to use, use 1 less than the max  
   cpuNum = multiprocessing.cpu_count() - 1 

   # Create the pool object  
   pool = multiprocessing.Pool(processes=cpuNum)  

   pool.map(func,groupids)
   pool.close()  
   pool.join()

I cannot make it work. I've tried many ways. When i run it i get "TypeError: can't pickle pyodbc.Row objects". Whilst when i debug it the logger shows that infinite workers are created and closed respectively. Through trial and error, i noticed that the more functions has my script the more workers are created.

The iterable variable is groupids and i use it to take the stops for each separate route.

UPDATE 1

It appears that the code stucks in def map, at the return:

    def map(self, func, iterable, chunksize=None):
       '''
       Equivalent of `map()` builtin
       '''
       assert self._state == RUN
       return self.map_async(func, iterable, chunksize).get()

i suspect that my problem has to do with my method's arguments. Their types are as following;

  1. EndPoint is a string
  2. RoadRules is a list
  3. data is a list
  4. StationName is a unicode
  5. Store_shp is a string (datasource of a shapefile)
  6. SortedStationStops_shp is a string (datasource of a shapefile)
  7. StationDeliveries_shp is a string (datasource of a shapefile)
  8. stationRoutes_Dic is a dictionary
  9. BranchId is a string

UPDATE 2

Below is the method that I am calling

def ra(EndPoint, RoadRules, data, StationName, Store_shp, SortedStationStops_shp, StationDeliveries_shp, stationRoutes_Dic, BranchId, groupid):
  try:
        print " "
        print "------>>>>ra"    

        GroupName=str(groupid)

        #Set LOCAL variables
        StationName=StationName.encode('utf-8')
        matchesStops = []
        matchesRoutes = []

        #Check out the Network Analyst extension license
        arcpy.CheckOutExtension("Network")

        #Set environment settings
        env.overwriteOutput = True
        env.outputCoordinateSystem = arcpy.SpatialReference("WGS 1984")
        arcpy.env.geographicTransformations = "GGRS_1987_To_WGS_1984"

        # Make a feature layer
        arcpy.MakeFeatureLayer_management(SortedStationStops_shp, "SortedStationStops")

        # Selection
        RouteStopsLayer=arcpy.SelectLayerByAttribute_management("SortedStationStops", "NEW_SELECTION", "Group_ID LIKE '"+GroupName+"%'")

        # Write the selected features to a new featureclass
        RouteStops=os.path.join(path, "04_InStationRouteStops_"+StationName+"_"+GroupName+".shp")
        arcpy.CopyFeatures_management(RouteStopsLayer, RouteStops)

        outNALayerName = "RA_StationRoute_"+StationName+"_"+GroupName+"_"+BranchId
        RouteStops_shp =os.path.join(path, "04_StationRouteStops_"+StationName+"_"+GroupName+"_"+BranchId+".shp")
        SingleRoute_shp =os.path.join(path, "04_StationRoute_"+StationName+"_"+GroupName+"_"+BranchId+".shp")

        #Set RA variables
        inNetworkDataset = Network_Dataset
        impedanceAttribute = "Minutes"
        timeUnits = "Minutes"
        distanceUnits = "Meters"
        distanceAttribute = "Length"
        inDepots = Store_shp
        inOrders = RouteStops
        outStopsFC =RouteStops_shp
        outRoutesFC =SingleRoute_shp
        outLayerFile = os.path.join(path, "RA_StationRoute_"+StationName+"_"+GroupName+".lyr")
        searchQuery = [["RoadNetwork", '"Access" <> -1'],["Greece_J18_ND_Junctions", ""]]

        ## Set Barriers
        if StationName =='ΒΣ' and GroupName.startswith('105'):
            Barriers_shp=os.path.join("C:\\Users\\tria\\Documents\\ArcGIS\\ACS_VRP\\BS_Cluster", "105barrier6.shp")
        if StationName =='ΒΣ' and GroupName.startswith('101'):
            Barriers_shp=os.path.join("C:\\Users\\tria\\Documents\\ArcGIS\\ACS_VRP\\BS_Cluster", "101barrier2.shp")
        if StationName =='ΣΛ':
            Barriers_shp=os.path.join("C:\\Users\\tria\\Documents\\ArcGIS\\ACS_VRP\\Salamina", "salaminaBarrier2.shp")

        ## Create a new Route layer. Optimize on Minutes, but compute the distance travelled by accumulating the Length attribute.
        #SYNTAX MakeRouteLayer_na (in_network_dataset, out_network_analysis_layer, impedance_attribute, {find_best_order}, {ordering_type}, {time_windows}, {accumulate_attribute_name}, {UTurn_policy}, {restriction_attribute_name}, {hierarchy}, {hierarchy_settings}, {output_path_shape}, {start_date_time})
        outRouteResultObject = arcpy.na.MakeRouteLayer(inNetworkDataset, outNALayerName, impedanceAttribute, "FIND_BEST_ORDER", ""+EndPoint+"", "", ["Minutes", "Length"], "ALLOW_DEAD_ENDS_AND_INTERSECTIONS_ONLY", RoadRules, "USE_HIERARCHY", "", "TRUE_LINES_WITH_MEASURES","")
        #Get the layer object from the result object. The route layer can now be referenced using the layer object.
        outNALayer = outRouteResultObject.getOutput(0)
        #print outNALayer

        #Get the names of all the sublayers within the route layer.(used in field mapping to parametized)
        subLayerNames = arcpy.na.GetNAClassNames(outNALayer)
        #print subLayerNames

        #Store the layer names that we will use later
        stopsLayerName = subLayerNames["Stops"]
        routesLayerName = subLayerNames["Routes"]
        pointBarriersLayerName = subLayerNames["PolylineBarriers"]# {u'Barriers': u'Point Barriers', u'Routes': u'Routes', u'PolylineBarriers': u'Line Barriers', u'Stops': u'Stops', u'PolygonBarriers': u'Polygon Barriers'} etc

        #Before loading locations as route stops, set up field mapping. Map the "Group_ID" field from the input data to the RouteName property in the Stops sublayer,
        # which ensures that each unique Group_ID will be placed in a separate route. Matching Group_ID from different stop files will end up in the same route.
        fieldMappingsStore = arcpy.na.NAClassFieldMappings(outNALayer, stopsLayerName)
        fieldMappingsStore["RouteName"].mappedFieldName = "Group_ID" #correlates fields ie naStops attribute table with inStores attribute table
        fieldMappingsStops = arcpy.na.NAClassFieldMappings(outNALayer, stopsLayerName)
        fieldMappingsStops["RouteName"].mappedFieldName = "Group_ID" #correlates fields ie naStops attribute table with inStores attribute table
        fieldMappingsBR = arcpy.na.NAClassFieldMappings(outNALayer, pointBarriersLayerName)
        fieldMappingsBR["BarrierType"].defaultValue = 0 #if it is restriction otherwise 1

        if StationName =='ΒΣ' and GroupName.startswith('105') or StationName =='ΒΣ' and GroupName.startswith('101') or StationName =='ΣΛ':
            #Figure out the layer's impedance attribute
            solver_props = arcpy.na.GetSolverProperties(outNALayer)
            impedance = solver_props.impedance
            fieldMappingsBR["Attr_" + impedance].defaultValue = 1.4
            #Load restrictions features
            arcpy.na.AddLocations(outNALayer, pointBarriersLayerName, Barriers_shp, fieldMappingsBR)

        ##Add locations as Stops. The same field mapping works for both input feature classes because they both have a field called "Group_ID"
        #AddLocations_na (in_network_analysis_layer, sub_layer, in_table, field_mappings, search_tolerance, {sort_field}, {search_criteria}, {match_type}, {append}, {snap_to_position_along_network}, {snap_offset}, {exclude_restricted_elements}, {search_query})
        arcpy.na.AddLocations(outNALayer, stopsLayerName, inDepots, fieldMappingsStore, "300 meters", "FID", [["J18_ND", "MIDDLE"], ["J18_ND_Junctions", "NONE"]], "MATCH_TO_CLOSEST", "CLEAR", "NO_SNAP", "", "", "")
        arcpy.na.AddLocations(outNALayer, stopsLayerName, inOrders, fieldMappingsStops, "300 meters", "FID", [["J18_ND", "SHAPE"], ["J18_ND_Junctions", "NONE"]], "MATCH_TO_CLOSEST", "APPEND", "SNAP", "1", "EXCLUDE", searchQuery)

        #Solve the route layer.
        arcpy.na.Solve(outNALayer)

        # Get the output Routes sublayer and save it to a feature class
        RoutesSubLayer = arcpy.mapping.ListLayers(outNALayer, routesLayerName)[0]
        StopsSubLayer = arcpy.mapping.ListLayers(outNALayer, stopsLayerName)[0]

        #Save the solved networkAnalyst layer as a layer file on disk (ok)
        outNALayer.saveACopy(outRoutesFC)
        arcpy.management.CopyFeatures(RoutesSubLayer, outRoutesFC)
        arcpy.management.CopyFeatures(StopsSubLayer, outStopsFC)
        matchesRoutes.append (outRoutesFC)
        matchesStops.append (outStopsFC)

        print "Routing completed successfully"

            ##JOIN TABLES (resolved Stops (Stops_shp) with points inforrmation (StationDeliveries_shp))

        ExpStops_shp =os.path.join(path, "042_StationExpStops_"+StationName+"_"+BranchId+".shp")
        Stops_shp=os.path.join(path, "040_StationALLStops__"+StationName+"_"+BranchId+".shp")
        Route_shp =os.path.join(path, "041_StationALLRoutes_"+StationName+"_"+BranchId+".shp")
        arcpy.Merge_management(matchesStops, Stops_shp)
        arcpy.Merge_management(matchesRoutes, Route_shp)
        del matchesStops[:]
        del matchesRoutes[:]

        #Call method to eliminate redStops
        dealStatus5(Stops_shp)

        target_features=StationDeliveries_shp
        join_features=Stops_shp
        out_feature_class = ExpStops_shp
        join_operation="JOIN_ONE_TO_ONE"
        join_type="KEEP_ALL"

        # Replace a layer/table view name with a path to a dataset (which can be a layer file) or create the layer/table view within the script
        arcpy.SpatialJoin_analysis(target_features, join_features, out_feature_class, join_operation="JOIN_ONE_TO_ONE", join_type="KEEP_ALL", match_option="CLOSEST", search_radius="500 Meters", distance_field_name="")

        ##Sorting by route name
        in_dataset = ExpStops_shp
        sortedExpStops_shp =os.path.join(path, "043_StationExpStopsSorted_"+StationName+"_"+BranchId+".shp")
        out_dataset = sortedExpStops_shp
        # Order features first by DISTANCE (NEAR_RANK)
        sort_fields = [["RouteName", "ASCENDING"]]
        # execute the function
        env.overwriteOutput = True
        arcpy.Sort_management(in_dataset, out_dataset, sort_fields)

        ##Create groupRouteidsDic (needed in update routingpaths)
        listGroupIDs=unique_values(sortedExpStops_shp, ['Group_ID','Route_ID']) #Group_ID because RouteName has now more than one  bins
        groupRouteidsDic=dict(listGroupIDs)
        RouteId= ','.join(str(e) for e in list(groupRouteidsDic.values()))
        print "RouteId", RouteId

        listGroupBins=unique_values(sortedExpStops_shp, ['Group_ID','Bin_No'])
        groupBinDic=dict(listGroupBins)
        print "groupBinDic", groupBinDic

        fields = ['Group_ID', 'RouteName', 'Route_ID', 'Bin_No']
        with arcpy.da.UpdateCursor(sortedExpStops_shp, fields) as cursor:
              for row in cursor:
                    row[2] =groupRouteidsDic[int(row[1])]
                    row[3] =groupBinDic[int(row[1])]
                    cursor.updateRow(row)

  except Exception as e:
        # If an error occurred, print line number and error message
        import traceback, sys
        tb = sys.exc_info()[2]
        print "An error occured on line %i" % tb.tb_lineno
        print str(e)
        cn.close

  #return ExpStops_shp, groupRouteidsDic, RouteId, Route_shp, sortedExpStops_shp, Stops_shp

UPDATE 3

I followed Hornbydd instructions and it worked. Also i had to remove from method's inputs an argument that i was getting with cur.fetchall() because it was producing "TypeError: can't pickle pyodbc.Row objects". The main changes are shown below:

        # Make a feature layer
        arcpy.MakeFeatureLayer_management(SortedStationStops_shp, "SortedStationStops"+str(groupid))

        # Selection
        RouteStopsLayer=arcpy.SelectLayerByAttribute_management("SortedStationStops"+str(groupid), "NEW_SELECTION", "Group_ID LIKE '"+GroupName+"%'")
  • What exactly is the error? Can you elaborate on "I cannot make it work" – KHibma Jan 8 at 13:38
  • The method works perfectly when i dont use multiprocessing. When i use multiprocessing it doesnt execute the method. I believe that it has to do with the fact that it has more than one arguments – sotokan80 Jan 8 at 15:07
  • I would suspect the issue is with your ra() function. As you don't actually show us what is happening in that function I suspect its failing, possibly thread locking and silently passing up and out to the main() function. Have a look at my article I posted on geonet. Until you edit your question and show what is being done in the worker function no one has a chance in solving this for you... – Hornbydd Jan 8 at 16:49
  • Very nice post.One of the errors that i faced was "pyodbc.raw objects cannot be pickled" so i am guessing that it has to do with your clarification that the full path of the shapefile must be given. So suppose by using .dataSource do i get the full path? i.e Store_shp.dataSource – sotokan80 Jan 8 at 19:31
  • I'll say it again... Until you edit your question and show what is being done in the worker function (ra()) no one has a chance in solving this for you because it may have nothing to do with your input parameters... – Hornbydd Jan 9 at 15:20
1

Now I can see your worker function, one immediate issue that leaps out is that your layer names are not unique and this will upset things, so this line needs to be adjusted

arcpy.MakeFeatureLayer_management(SortedStationStops_shp, "SortedStationStops")

Refer to my document on geonet and you will see that I create unique layers by attaching the id that is being fed in into the worker function.

I have not checked your code fully but outputs must have unique full file path names. So if for example you are creating a temporary dataset that ultimately feeds into the final output which is what you actually want then the temporary dataset must have unique names. You cannot keep writing to c:\temp\mytempdata.shp for example.

So double check your code and anything that is currently common or being reused must all be unique. That's the issue I have had when trying to use multiprocessing.

How that affects the output of network analyst which creates predefined layer names I am unsure?

Also you don't check in the extension, I would do that in the finally part of your try/except.

  • SortedStationStops_shp is a shapefile that has all the stops of a station. Each station has a number of routes. those routes are in the iterable variable named groupids(i.e [101, 102, 103, 104]). So i am using arcpy.MakeFeatureLayer_management in order to make a selection and create new shapefiles of stops one for each route. the new shapefiles will be unique since they look like "RouteStops=os.path.join(path, "04_InStationRouteStops_"+StationName+"_"+GroupName+".shp")" in the beggining of the method i assign GroupName=str(groupid) where groupid is the iterable variable in the method ra. – sotokan80 Jan 9 at 20:25
  • Do you believe that somehow it cannot make 4 selections simultaneously by using .MakeFeatureLayer_management and then creating 4 times a RouteStopsLayer=arcpy.SelectLayerByAttribute_management("SortedStationStops", "NEW_SELECTION", "Group_ID LIKE '"+GroupName+"%'") (one for each route)? Furthermore i check in the network analyst extension at the beginning of the code. – sotokan80 Jan 9 at 20:28
  • Easy one first. you check OUT the Extension but you never check IN the extension. Not sure this really affects things but makes your code cleaner. – Hornbydd Jan 9 at 22:30
  • 1
    When you run Make FeatureLayer your layer is always called "SortedStationStops". I don't believe this will work (well it never did for me) they need to be uniquely named, hence I attach the ID to it to make them unique names. Your subsequent call to SelectLayerByAttribute is selecting features by group_id so I believe you need unique names to ensure unique selections. – Hornbydd Jan 9 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.