2

I'm trying to cut down on arcpy processing time by using the python multiprocessing toolbox. Using the example from http://wltrimbl.github.io/2014-06-10-spelman/intermediate/python/04-multiprocessing.html as a guideline.

Basically, I'm trying to select all tracks (point features) with a specific trackID (in field 'track'), give it a buffer and extract all road features within that buffer as preprocessing for another script. But the data set is so large it would take me about a week to run it.

I now have the following code, but I'm having a problem with the program executing normally, but not outputting the files in Tracks.gdb. Does anyone have advice?

import arcpy
from arcpy import env
import multiprocessing
def MyFunction(i):
    TrackFeatureOut = "C:\tryingOut\Tracks.gdb\Track_" + str(i)
    RoadFeatureOut = "C:\tryingOut\Tracks.gdb\Road_" + str(i)
    if not (RoadFeatureOut in featureclasses):
        BufferFeatureOut = "C:\tryingOut\Temporary.gdb\Buffer_" + str(i)
        Query = "track =" + str(i)

        arcpy.Select_analysis("C:\tryingOut\Input.gdb\Tracks", TrackFeatureOut, Query)

        arcpy.Buffer_analysis(TrackFeatureOut, BufferFeatureOut, "100 Meters")

        arcpy.Clip_analysis("C:\tryingOut\Input.gdb\Roads", BufferFeatureOut, RoadFeatureOut)

if __name__ == '__main__':
    global featureclasses
    env.workspace = "C:\tryingOut\Tracks.gdb"
    featureclasses = arcpy.ListFeatureClasses()

    # Getting all unique track numbers
    Track = set([])
    cursor = arcpy.da.SearchCursor("C:\tryingOut\Input.gdb\Tracks",["track"],sql_clause=(None, 'ORDER BY track' ))
    for row in cursor:
        Track.add(row[0])

    # Trying to apply multiprocessing
    pool = multiprocessing.Pool(multiprocessing.cpu_count())
    for i in Track:
        if Counter % 1000 == 0:
            print "Counter = ", Counter, "Current Track = ",i
        Counter += 1    
        pool.apply_async(MyFunction, i)
  • Your featureclasses does not exist inside my_function. Remove global and fill list after import line – FelixIP Feb 28 '17 at 19:55
  • You might be interested in this document I wrote on geonet? – Hornbydd Feb 28 '17 at 23:45
  • Also put an r in front of your paths as \t is a tab character, for example you want to be writing TrackFeatureOut = r"C:\tryingOut\Tracks.gdb\Track_" + str(i). – Hornbydd Feb 28 '17 at 23:53
  • I'm currently looking at the suggestions, and all code-advice is appreciated. But the script runs well without the implementation of multiprocessing, and it does not return any errors even with multiprocessing. However, it does not seem to enter the MyFunction for some reason – Martin13131 Mar 1 '17 at 9:48
0

There's two improvements I should've made here:

Firstly, there's the arcpy function arcpy.SplitByAttributes_analysis(), which (in my experience) works more efficiently than the select function.

Secondly, my form of multiprocessing was off, or at least I found a way that did work. I managed to write a short function which should take any function and iterable list as input arguments and multiprocess it to the number of cores available:

def multiprocessingfunc(Func, List):
    Count = 0
    while Count < len(List):
        if len(multiprocessing.active_children()) < multiprocessing.cpu_count():
            p = multiprocessing.Process(target=Func, args=(List[Count],)
            p.start()
            Count += 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.